← All benchmarks

Real-SWE 1.1

Code

Specific Labs' Real-SWE 1.1 evaluates coding agents in their native harnesses on 25 tasks inspired by or lifted verbatim from private production codebases licensed from real companies, across TypeScript, Go, Python, Ruby and Java, such as billing, tax and customer-migration changes. Reference solutions edit 10.2 files on average; verifiers derived from each codebase's test suites are injected at grading time. Resolution rate is pass@1 averaged over five independent runs per task, with every run at the highest effort each model offers in its harness. It is a distinct, larger task set from the ten-task September 2026 Real-SWE report, so the two are not pooled. The tasks are not public, and the report is dated only to October 2026.

Top models (higher is better)