DeepsecBench-AA (Artificial Analysis)
Code · 2026-09-25
Artificial Analysis' implementation of Vercel's DeepsecBench. The agent reviews real open-source application source code pinned before major vulnerability fixes and reports findings, which a judge model with source access classifies and matches against a private expert-verified golden set. The score is F2 (recall weighted above precision), the median of three runs, using the Stirrup harness with up to 500 turns per batch and no internet access. The golden-set size is not published, and this protocol is separate from Vercel's AI Gateway leaderboard.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-6 Sol | 46.3 |
| GPT-6 Astra | 36.9 |
| DeepSeek V4.1 Flash | 31.2 |
| Grok 4.7 | 26.9 |
| MiMo-V2.6-Pro | 26.2 |
| Claude Opus 5 | 25.9 |
| GPT-5.6 Terra | 25.1 |
| GLM-5.3 | 24.3 |
| GPT-6 Luna | 23.5 |
| GLM-5.3 Flash | 19.9 |
| Mistral Large 4 Preview | 16.0 |
| Step 5 Preview | 15.8 |
| Gemini 3.8 Flash | 14.5 |
| Muse Spark 1.3 | 12.5 |
| Kimi K3 | 12.5 |
| Qwen3.8 27B | 11.5 |
| Qwen3.8 2.4T A95B | 9.5 |
| Beam 1.0 501B A23B | 6.9 |
| MiniMax M3 | 1.1 |
| Nemotron 3 Ultra 550B A55B | 1.0 |