Tampered ML
Science · 2026-10-23
1,858 pairs of machine-learning experiment proposals, each an original reconstructed from a published ICLR, ICML or NeurIPS paper and a minimally edited counterfactual with a targeted methodological flaw. The model must pick the more methodologically sound proposal in both presentation orders; both-orders accuracy, with a 25% chance level, is the share of pairs answered correctly in both orders, in percent. A human validation study estimates 86% of items valid.
Top models (higher is better)
| Model | Score |
|---|---|
| Claude Opus 5 | 77.9 |
| Opus 4.8 | 76.5 |
| Claude Fable 5 | 73.5 |
| Gemini 3.1 Pro Preview | 73.3 |
| GPT-5.5 | 71.8 |
| GPT-5.4 | 68.2 |
| Gemini 3 Flash Preview | 64.4 |
| Opus 4.7 | 63.2 |
| Kimi K2.6 | 59.8 |
| GPT-5.2 | 59.6 |
| GLM-5.1 | 56.6 |
| DeepSeek-V4-Pro | 56.2 |
| Sonnet 4.6 | 55.3 |
| Grok 4.3 | 54.0 |
| Sonnet 4.5 | 52.0 |
| Opus 4.5 | 48.9 |
| Opus 4.6 | 42.9 |
| DeepSeek-V3.1 | 42.6 |
| Qwen3 235B A22B Thinking 2507 | 40.4 |
| Haiku 4.5 | 39.1 |
| MiniMax M2.5 | 38.6 |
| Qwen3 30B A3B Thinking 2507 | 32.7 |