GSM8K
Math · 2021-10-27
A grade-school math word problem benchmark focused on multi-step arithmetic reasoning and exact-match solutions.
Top models (higher is better)
| Model | Score |
|---|---|
| Claude 3.5 Sonnet (June 2024) | 96.4 |
| Opus 3 | 95.0 |
| DeepSeek-Coder-V2-Instruct | 94.5 |
| Qwen2.5 Coder 32B Instruct | 93.0 |
| Claude 3 Sonnet | 92.3 |
| GPT-4 | 92.0 |
| GPT-4o Mini | 91.3 |
| Gemini 1.5 Pro | 90.8 |
| GPT-4 0613 | 90.0 |
| Claude 3 Haiku | 88.9 |
| Phi-3.5-MoE-instruct | 88.7 |
| Qwen2.5-Coder-14B | 88.7 |
| DeepSeek-Coder-V2-Lite-Instruct | 87.6 |
| Claude Instant 1.2 | 86.7 |
| Qwen2.5-Coder-7B-Instruct | 86.7 |
| Phi-3.5 Mini Instruct | 86.2 |
| Gemma 2 9B | 84.9 |
| Mistral NeMo Base 2407 | 84.2 |
| Qwen2.5 Coder 7B | 83.9 |
| Llama 3.1 8B Instruct | 82.4 |
| Claude Instant | 80.9 |
| PaLM 2-L | 80.0 |
| Qwen2.5-Coder-3B | 75.7 |
| Mixtral 8x7B | 74.4 |
| Stable Beluga 2 | 69.6 |
| Yi-34B | 67.2 |
| DeepSeek-Coder-V2-Lite-Base | 67.1 |
| Qwen2.5 Coder 1.5B | 65.8 |
| Grok 1 | 62.9 |
| StarCoder2-15B | 57.7 |
| PaLM 540B | 56.5 |
| Falcon 180B | 54.4 |
| Falcon2-11B | 53.8 |
| GPT-3.5 Turbo 0301 | 53.1 |
| Qwen 7B | 51.7 |
| Gemma 7B | 46.4 |
| Nemotron-4 15B | 46.0 |
| Llama 2 34B | 42.2 |
| CodeQwen1.5 7B | 37.7 |
| DeepSeek-Coder-Base 33B | 35.4 |