CyberGym-E2E-AA (Artificial Analysis)
Agents · 2026-09-25
Artificial Analysis' implementation of Berkeley RDI's CyberGym-E2E on a filtered set of 131 memory-safety vulnerabilities, one per OSS-Fuzz project, excluding tasks with oracle leaks or sandbox constraints. A task passes when the agent's proof of concept crashes the unpatched build, its patch fixes that crash and the project still passes its functionality tests (stages 1-3); ground-truth repair is diagnostic only. Runs use the Stirrup harness with 90 minutes per task, scoring pass@1.
Top models (higher is better)
| Model | Score |
|---|---|
| Mistral Large 4 Preview | 81.7 |
| MiMo-V2.6-Pro | 78.6 |
| GPT-6 Luna | 77.9 |
| Step 5 Preview | 74.8 |
| GLM-5.3 Flash | 74.0 |
| Grok 4.7 | 74.0 |
| Muse Spark 1.3 | 59.5 |
| Kimi K3 | 58.0 |
| Beam 1.0 501B A23B | 45.0 |
| GLM-5.3 | 29.0 |
| MiniMax M3 | 24.4 |
| DeepSeek V4.1 Flash | 22.9 |
| Nemotron 3 Ultra 550B A55B | 10.7 |
| Gemini 3.8 Flash | 8.4 |
| GPT-6 Astra | 0.0 |
| GPT-6 Sol | 0.0 |
| Qwen3.8 2.4T A95B | 0.0 |
| Qwen3.8 27B | 0.0 |