SWEeper-Bench
Code · 2026-10-07
Princeton's benchmark of finding and fixing unreported bugs in interactive software: each of 200 tasks is a real user-facing bug fix in a different open-source web application, and the agent gets only the codebase, the running app with a browser, and an open-ended request naming a product area, never the bug itself. An agentic verifier applies the patch to a fresh copy of the app and replays a hidden target behaviour test and a preservation test in the browser. The score is the percentage of the 200 tasks where both tests pass.
Top models (higher is better)
| Model | Score |
|---|---|
| Grok 4.6 | 59.0 |
| Qwen3.8 Flash | 57.5 |
| Claude Fable 5.1 | 56.5 |
| Claude Opus 5 | 56.5 |
| GPT-6 Astra | 56.5 |
| DeepSeek V4.1 Flash | 51.5 |
| Kimi K3 | 48.5 |
| GPT-5.6 Sol | 47.0 |
| GLM-5.3 | 43.5 |
| GPT-5.6 Luna | 42.0 |
| GLM-5.3 Flash | 39.5 |
| GPT-5.6 Terra | 35.0 |