← All benchmarks

ProgramBench (Anthropic 166-task, 5-episode hidden-test pass rate)

Code · 2026-07-24

Anthropic's ProgramBench protocol from the Claude Opus 5 system card. It drops the 34 tasks whose reference binary scores below 0.9 on the hidden tests, leaving 166, and scores only the hidden tests the reference binary passes. The reported value is the hidden-test pass rate (partial credit) after five cumulative episodes, each continuing from the previous codebase with a fresh budget of up to 1M tokens. Not comparable to ProgramBench's 200-task all-tests-pass resolve rate.

Top models (higher is better)