CritPt v2 (agentic, avg@5)
Science
CritPt v2 is the CritPt authors' revision of the dataset: its 70 research-level physics challenges, reviewed with physics expert collaborators to correct underspecified statements, errors and typos. Artificial Analysis runs it agentically in its Stirrup harness, in an offline Linux sandbox with Python 3.13 (numpy, scipy, sympy, matplotlib) and at most 50 turns, using the authors' one-step prompt for models with code tools; the authors' grading server scores each repeat, and the reported pass@1 is the mean over five repeats per challenge. Scores are not comparable with the tool-free CritPt v1.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-6 Astra | 83.1 |
| GPT-6.1 Sol | 82.0 |
| Claude Opus 5.5 | 77.7 |
| GPT-5.6 Sol | 74.9 |
| Muse Wonder Full | 73.1 |
| Gemini 4 Argon | 72.9 |
| Claude Fable 5.1 | 72.0 |
| Muse Wonder | 69.7 |
| Grok 4.7 | 50.6 |
| DeepSeek V4.1 Flash 1017 | 50.0 |
| GPT-5 | 28.9 |