SWE-sweep
Code · 2026-10-01
Meta's benchmark of open-ended bug finding: each of 100 real repositories in 22 languages is checked out at a commit where many reported bugs coexist, 4,068 in total, and the agent must discover and fix as many as it can with no hint of their type or location. Each bug counts as resolved when the fix passes the hidden tests from its original pull request without regressing the existing suite. The score is the share of all bugs resolved by a mini-SWE-agent run on each repository.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-6 Astra | 9.5 |
| GPT-5.6 Sol | 4.7 |
| Claude Opus 5.5 | 3.6 |
| Claude Fable 5.1 | 2.9 |
| GPT-5.6 Luna | 2.5 |
| GPT-5.6 Terra | 1.5 |
| Claude Opus 5 | 1.3 |
| Kimi K3 | 0.6 |
| GPT-5.4 Mini | 0.5 |
| Gemini 3.5 Flash-Lite | 0.1 |