SWE-Shift (Shifted)
Code · 2026-10-23
Coding agents resume a repository task from a saved history and workspace and receive a task-relevant mid-execution change (code, dependency, runtime, requirement, tool or evidence updates across 12 change families) at one of nine scheduled positions. Strict success requires completing the current task, preserving still-valid work and passing every applicable state and evidence check. 120 scenarios drawn from SWE-bench Verified, Multilingual and other repository tasks; the success rate uses the paper's family-cluster-scenario weighting over all positions, in percent.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-6 Astra | 24.8 |
| Claude Fable 5.1 | 18.3 |
| Claude Opus 5 | 16.1 |
| GPT-5.6 Sol | 12.6 |
| Kimi K3 | 12.1 |
| DeepSeek-V4-Pro | 10.3 |
| DeepSeek-V4-Flash | 8.6 |
| Kimi K2.6 | 8.0 |
| GLM-5.3 Flash | 6.8 |
| MiniMax M3 | 6.0 |
| MiniMax M2.7 | 5.9 |
| Qwen3.8 27B | 5.4 |
| Devstral 2 | 5.3 |
| Qwen3-Coder-Next | 4.1 |
| MiMo-V2.5-Pro | 3.6 |
| Qwen3 30B A3B Instruct 2507 | 1.2 |