SciCode (main problems)
Science · 2024-07-12
Main-problem resolve rate on SciCode's 65-problem test split: a problem counts only when every one of its subproblems passes. The standard SciCode row instead pools the 288 subproblems. It is a diagnostic: the SciCode family falls back to a single component in the fit, and a second component would drop the whole family.
Top models (higher is better)
| Model | Score |
|---|---|
| Claude Opus 5.5 | 27.3 |
| Claude Fable 5.1 | 24.2 |
| Claude Fable 5 | 21.5 |
| GPT-6 Astra | 19.6 |
| Opus 4.7 | 17.7 |
| Claude Opus 5 | 17.3 |
| Opus 4.8 | 16.9 |
| GPT-5.6 Sol | 16.2 |
| GPT-5.4 | 15.8 |
| GPT-5.6 Terra | 15.8 |
| Kimi K3 | 15.8 |
| Gemini 3.1 Pro Preview | 15.0 |
| GPT-6 Luna | 15.0 |
| GPT-6.1 Sol | 14.6 |
| Grok 4.6 | 14.6 |
| Gemini 3.6 Flash | 14.2 |
| Gemini 3.8 Flash | 14.2 |
| GPT-6 Sol | 12.7 |
| Grok 4.5 | 12.7 |
| GPT-5.5 | 12.3 |
| Sonnet 4.6 | 11.9 |
| Gemini 3.5 Flash | 11.9 |
| GLM-5.2 | 11.9 |
| MiniMax M3 | 11.9 |
| DeepSeek V4.1 Flash | 11.2 |
| GPT-5.6 Luna | 11.2 |
| Inkling | 11.2 |
| o3-mini | 10.8 |
| Kimi K2.7 Code | 9.2 |
| DeepSeek-V3.2 | 8.1 |
| o1 Preview | 7.7 |
| Kimi K2 Thinking | 7.7 |
| MiniMax M2.7 | 6.9 |
| gpt-oss-120b | 5.8 |
| Claude 3.5 Sonnet (June 2024) | 4.6 |
| Claude 3.5 Sonnet (Oct 2024) | 4.6 |
| DeepSeek-R1 | 4.6 |
| DeepSeek-Coder-V2-Instruct | 3.1 |
| DeepSeek-V3 | 3.1 |
| Opus 3 | 1.5 |