PWNBench v0.1
Agents · 2026-08-20
Novee's PWNBench v0.1 tests whether an agent can find and validate real vulnerabilities in running forks of large open-source web applications under greybox access, against 400+ expert-built ground-truth labels spanning the OWASP Top 10. Findings from k repeated runs of a configuration are merged before scoring, and the headline is all-severity F0.5 computed per application and averaged equally across applications. Novee runs every model in one fixed, simplified internal harness and supplied the exact values to Fireworks' Specialized Intelligence Index.
Top models (higher is better)
| Model | Score |
|---|---|
| Grok 4.6 | 63.3 |
| Claude Opus 5 | 60.0 |
| Opus 4.8 | 55.6 |
| GPT-5.6 Sol | 54.0 |
| DeepSeek V4.1 Flash | 49.2 |
| Grok 4.5 | 49.1 |
| GLM-5.3 | 48.8 |
| Kimi K3 | 48.2 |
| Opus 4.6 | 47.9 |
| DeepSeek-V4-Flash-0731 | 47.4 |
| GLM-5.2 | 44.3 |
| GLM-5.3 Flash | 38.0 |
| Muse Glimmer 30B | 25.8 |