SciCode (Artificial Analysis)
Science · 2024-07-12
Artificial Analysis evaluation of SciCode, a scientist-curated research-coding benchmark. SciCode contains 338 subproblems from 80 main problems across 16 subfields; the standard test split has 288 subproblems from 65 main problems. The reported score is pass@1/accuracy over solved subproblems.
Top models (higher is better)
| Model | Score |
|---|---|
| Claude Fable 5 | 60.2 |
| Gemini 3.1 Pro Preview | 59.0 |
| Kimi K3 | 58.7 |
| Muse Spark 1.1 | 58.2 |
| GPT-5.6 Sol | 56.9 |
| GPT-5.4 | 56.6 |
| GPT-5.5 | 56.1 |
| Claude Opus 5 | 55.7 |
| GPT-5.2-Codex | 54.6 |
| Opus 4.7 | 54.5 |
| Grok 4.5 | 54.1 |
| GPT-5.6 Terra | 53.9 |
| Sonnet 5 | 53.6 |
| Opus 4.8 | 53.5 |
| Kimi K2.6 | 53.5 |
| GPT-5.3-Codex | 53.2 |
| Gemini 3.5 Flash | 53.1 |
| Gemini 3.6 Flash | 52.7 |
| GPT-5.6 Luna | 52.5 |
| Muse Spark | 51.5 |
| Gemini 3 Flash Preview | 50.6 |
| GLM-5.2 | 50.5 |
| GPT-5.5 Instant | 50.3 |
| Grok Build 0.1 | 50.2 |
| MiMo-V2.5-Pro | 50.2 |
| DeepSeek-V4-Pro | 50.0 |
| DeepSeek-V4-Flash-0731 | 49.9 |
| Gemini 3 Pro Preview | 49.9 |
| GPT-5.4 Mini | 49.9 |
| Opus 4.5 | 49.5 |
| Kimi K2.5 | 49.0 |
| Qwen3.7-Max | 48.8 |
| Inkling-Small | 48.7 |
| GPT-5.5 Instant (2026-06-25 hosted snapshot) | 48.6 |
| Hy3 | 47.6 |
| Kimi K2.7 Code | 47.5 |
| Grok 4.3 | 47.3 |
| MiniMax M2.7 | 47.0 |
| Sonnet 4.6 | 46.9 |
| GPT-5.4 Nano | 46.9 |