P2NPCBench (NP-complete, pass@1)
Code · 2026-10-21
P2NPCBench's 1,003 NP-complete programming statements across twelve problem families, each rewritten by an adversarial multi-agent pipeline to hide textbook cues and paired with a polynomial-time twin. A model writes one Python program per statement, which passes only if every test case does; the score is pass@1 macro-averaged over the families, in percent.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-5.6 Sol | 68.4 |
| Gemini 3.8 Flash | 62.6 |
| Gemini 3.7 Flash | 57.8 |
| DeepSeek V4.1 Flash | 39.3 |
| Qwen3.8-Omni-Flash | 32.3 |
| Gemma 3 12B IT | 26.4 |
| Gemma 3 27B IT | 25.0 |
| Mistral Medium 3.5 | 23.1 |
| Llama 4 Scout Instruct | 21.4 |
| Qwen3.8 27B | 21.2 |
| Gemma 3 4B IT | 7.3 |