SWE-Bench Pro (Private Dataset)
Code · 2025-09-19
This private SWE-bench Pro split evaluates repository-level software-engineering tasks withheld from the public set. Scores report the percentage resolved.
Top models (higher is better)
| Model | Score |
|---|---|
| Muse Spark 1.1 | 51.5 |
| Opus 4.6 | 47.1 |
| Muse Spark | 44.7 |
| GPT-5.4 | 43.4 |
| Gemini 3.1 Pro Preview | 32.2 |
| GPT-5.2-Codex | 27.7 |
| GPT-5.2 | 23.8 |
| Opus 4.5 | 23.4 |
| Gemini 3 Pro Preview | 17.9 |
| Opus 4.1 | 17.8 |
| GPT-5 | 14.9 |
| Sonnet 4 | 9.1 |
| GPT-4o | 3.6 |