CWE-Bench-AA (Artificial Analysis)
Code · 2026-09-25
Artificial Analysis' implementation of Collinear AI's CWE-bench on 120 held-out tasks private to Collinear AI and Artificial Analysis, covering all ten OWASP Top 10 (2025) categories across six languages. The agent audits a repository from a described area of concern and must patch the vulnerability without breaking legitimate behavior; a deterministic exploit and behavior verifier grades all-or-nothing. Runs use the Stirrup harness, no internet access and a two-hour limit, scoring pass@1 on a single attempt. This protocol is separate from Collinear's own four-rollout CWE-bench v1 leaderboard.
Top models (higher is better)
| Model | Score |
|---|---|
| DeepSeek V4.1 Flash | 68.3 |
| Grok 4.7 | 68.3 |
| GPT-6 Sol | 64.2 |
| GPT-6 Astra | 63.3 |
| MiMo-V2.6-Pro | 63.3 |
| Muse Spark 1.3 | 60.8 |
| GPT-6 Luna | 56.7 |
| GLM-5.3 Flash | 55.8 |
| GLM-5.3 | 55.8 |
| Kimi K3 | 53.3 |
| Gemini 3.8 Flash | 52.5 |
| Step 5 Preview | 52.5 |
| Mistral Large 4 Preview | 50.8 |
| Beam 1.0 501B A23B | 49.2 |
| MiniMax M3 | 41.7 |
| Qwen3.8 2.4T A95B | 28.3 |
| Nemotron 3 Ultra 550B A55B | 27.5 |
| Qwen3.8 27B | 26.7 |