SWE Atlas - Codebase QnA (pass@3)
Code · 2026-03-04
This aggregation of the SWE Atlas Codebase QnA track reports pass@3 on repository-level questions requiring runtime analysis and multi-file reasoning. It is kept separate from single-run resolve-rate observations on the parent track.
Top models (higher is better)
| Model | Score |
|---|---|
| Opus 4.8 | 57.3 |
| Macaron-V1-Venti | 49.5 |
| GLM-5.2 | 48.9 |
| GPT-5.5 | 45.4 |
| MiniMax M3 | 37.9 |
| Qwen3.7-Max | 22.6 |
| Gemini 3.1 Pro Preview | 13.5 |