CWE-bench v1 Programmatic Pass@1
Code · 2026-09-25
Collinear AI's private 120-task defensive security audit-and-patch evaluation across 73 CWEs and six languages. Models receive four high-reasoning rollouts per task, each limited to one hour; a deterministic verifier requires the exploit to fail and all regression tests to pass. Pass@1 averages successful rollouts per task.
Top models (higher is better)
| Model | Score |
|---|---|
| Gemini 4 Argon | 68.0 |
| GPT-6 Astra | 68.0 |
| Grok 4.7 | 68.0 |
| Claude Opus 5.5 | 67.0 |
| Claude Fable 5.1 | 58.0 |
| Grok 4.6 | 57.0 |
| DeepSeek V4.1 Flash | 55.0 |
| Muse Spark 1.3 | 55.0 |
| Hy4 Preview | 53.0 |
| GPT-6 Sol | 52.0 |
| Inkling | 37.0 |