ProgramBench Average Hidden-Test Pass Rate
Code
Mean hidden-test pass percentage across the 200 ProgramBench tasks, including partial credit. Vals calls this Raw Pass Rate. Distinct from fully resolved tasks, which require passing every hidden test and remain the ProgramBench primary observation.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-6 Astra | 85.4 |
| GPT-6.1 Sol | 85.3 |
| Gemini 4 Argon | 83.7 |
| GPT-6 Sol | 81.9 |
| Claude Haiku 5.5 | 78.7 |
| Claude Sonnet 5.5 | 76.3 |
| Grok 4.7 | 75.7 |
| Grok 4.6 | 74.0 |
| GPT-5.6 Terra | 72.3 |
| Sonnet 5 | 72.1 |
| DeepSeek V4.1 Flash | 72.0 |
| Gemini 3.8 Flash | 71.9 |
| Muse Spark 1.3 | 71.7 |
| MiMo-V2.6-Flash | 71.7 |
| DeepSeek V4 Pro 0813 | 70.1 |
| MiMo-V2.6-Pro | 69.8 |
| Hy4 Preview | 69.2 |
| Gemini 3.7 Flash | 68.7 |
| GPT-5.4 | 68.6 |
| GPT-5.6 Luna | 68.3 |
| GLM-5.3 | 66.3 |
| GPT-6 Luna | 66.2 |
| Grok 4.5 | 65.2 |
| Opus 4.7 | 64.2 |
| Kimi K3 | 62.8 |
| GLM-5.2 | 62.6 |
| Muse Spark 1.2 | 61.9 |
| Gemini 3.6 Flash | 59.9 |
| Sonnet 4.6 | 59.1 |
| DeepSeek-V4-Flash-0731 | 57.3 |
| Gemini 3.5 Flash | 56.4 |
| GPT-5.4 Mini | 54.4 |
| GLM-5.1 | 50.9 |
| Kimi K2.7 Code | 49.4 |
| GLM-5.3 Flash | 48.6 |
| Kimi K2.6 | 48.0 |
| DeepSeek-V4-Pro | 47.8 |
| Inkling-Small | 42.5 |
| Qwen3.8-Max | 40.0 |
| Gemini 3.1 Pro Preview | 39.5 |