ARC AI2
Classic NLP · 2018-03-14
A multiple-choice science benchmark assessing grade-school science knowledge and reasoning on questions from the AI2 ARC dataset.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-4 | 96.3 |
| DeepSeek-V3 | 95.3 |
| Llama 3.1 405B | 95.3 |
| Qwen2.5 72B | 94.5 |
| DeepSeek-V2 | 92.2 |
| Phi-3 Medium 128K Instruct | 91.6 |
| Phi-3-Small-8K-Instruct | 90.7 |
| GPT-3.5 Turbo 1106 | 87.4 |
| Claude Instant 1.2 | 86.3 |
| Stable Beluga 2 | 86.1 |
| Claude Instant | 85.7 |
| Phi-3-mini-4k-instruct | 84.9 |
| Qwen 14B | 84.4 |
| Llama 3 8B Instruct | 82.8 |
| InternLM 20B | 81.7 |
| Phi-2 | 75.9 |
| Qwen 7B | 75.3 |
| InternLM 7B | 69.5 |
| PaLM 2-L | 69.2 |
| Qwen2.5-Coder-14B | 66.0 |
| PaLM 2-M | 64.9 |
| ChatGLM2 6B (chat) | 61.0 |
| Qwen2.5 Coder 7B | 60.9 |
| PaLM 2-S | 59.6 |
| DeepSeek-Coder-V2-Lite-Base | 57.3 |
| Yi-9B | 55.6 |
| Nemotron-4 15B | 55.5 |
| Llama 2 34B | 54.5 |
| Qwen 1.8B | 53.2 |
| Qwen2.5-Coder-3B | 52.9 |
| LLaMA-13B | 52.7 |
| PaLM 62B | 52.5 |
| MPT-30B | 50.6 |
| Yi-6B | 50.3 |
| StarCoder2-15B | 47.2 |
| Qwen2.5 Coder 1.5B | 45.2 |
| Phi-1.5 | 44.4 |
| Vicuna 13B v1.1 | 43.2 |
| DeepSeek-Coder-Base 33B | 42.2 |
| Gemma 2B | 42.1 |