ZeroBench Main Mean pass@1 (five samples)
Multimodal
ZeroBench main-question accuracy estimated from five samplings per question: average the fraction correct for each of the 100 questions, then average over questions. The official filled-circle mean pass@1 is distinct from its one-sample greedy pass@1. Current leaderboard scores include corrections after evaluation red teaming.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-5.6 Sol | 22.0 |
| Claude Opus 5 | 17.2 |
| GPT-5.5 | 17.2 |
| GPT-5.4 | 16.4 |
| Claude Fable 5 | 15.8 |
| GPT-5.6 Terra | 14.2 |
| GPT-5.6 Luna | 13.4 |
| Gemini 3.5 Flash | 12.8 |
| Gemini 3.6 Flash | 12.2 |
| Gemini 3.1 Pro Preview | 11.6 |
| Grok 4.5 | 11.4 |
| GPT-5.2 | 11.2 |
| Grok 4.6 | 11.0 |
| Opus 4.8 | 10.4 |
| Gemini 3 Pro Preview | 10.0 |
| Sonnet 5 | 9.4 |
| Gemini 3 Flash Preview | 9.4 |
| GPT-5.4 Mini | 9.0 |
| Opus 4.7 | 8.8 |
| Sonnet 4.6 | 7.8 |
| Grok 4.3 | 6.4 |
| Opus 4.6 | 6.4 |
| GPT-5 Mini | 6.2 |
| Gemini 3.5 Flash-Lite | 6.0 |
| O4 Mini | 4.6 |
| Opus 4.5 | 3.6 |
| Gemini 3.1 Flash-Lite Preview | 3.4 |
| Opus 4.1 | 3.4 |
| Sonnet 4 | 2.8 |
| Gemini 2.5 Pro Experimental 03-25 | 2.8 |
| Opus 4 | 2.4 |
| o3 | 2.4 |
| Claude 3.7 Sonnet | 2.4 |
| GPT-5 | 2.2 |
| Sonnet 4.5 | 2.0 |
| GPT-4.5 | 2.0 |
| GPT-5.1 | 1.8 |
| Llama 4 Scout Instruct | 1.6 |
| Grok 4 | 1.4 |
| Gemini 2.5 Flash Preview 04-17 | 1.4 |