Real-SWE
Code
Specific Labs' Real-SWE evaluates coding agents in their native harnesses on ten tasks inspired by or lifted verbatim from private production codebases licensed from real companies, such as billing, tax, customer-migration and infrastructure changes. The median reference solution edits 11 files; verifiers derived from each codebase's test suites are injected at grading time. Resolution rate is pass@1 averaged over eight independent runs per task. The tasks are not public, and the report is dated only to September 2026. Real-SWE 1.1 (October 2026) replaced this ten-task set with 25 tasks; these September results are preserved as a separate historical benchmark.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-6 Astra | 46.3 |
| Claude Fable 5.1 | 45.0 |
| Gemini 3.8 Flash | 38.8 |
| GLM-5.3 | 37.5 |
| Muse Spark 1.3 | 36.3 |
| Grok 4.6 | 32.5 |
| GPT-5.6 Sol | 26.3 |
| Kimi K3 | 20.0 |