AutoformBench
Math · 2026-10-12
FAIR at Meta's 341 audited Lean 4 proof-completion tasks drawn from 24 advanced mathematical sources, each embedded in a wider development where the agent must build missing definitions, lemmas and supporting infrastructure. Agents run in the authors' offline common tool loop (files, Lean REPL, Loogle, build) with the informal source excerpt and a $200 per-task provider-cost cap. A task counts only when clean-room Comparator grading and Nanoda kernel replay accept every graded obligation; scores are from release 1.0.0-rc1.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-6 Astra | 82.4 |
| Claude Opus 5 | 61.9 |
| GPT-5.6 Sol | 58.9 |
| Claude Fable 5 | 53.1 |
| DeepSeek V4.1 Flash | 52.8 |
| Gemini 3.7 Flash | 52.5 |
| Kimi K3 | 48.4 |
| Sonnet 5 | 44.6 |
| Opus 4.8 | 44.0 |