MMLU
General QA · 2020-09-07
MMLU (Measuring Massive Multitask Language Understanding) is a multiple-choice benchmark spanning 57 subjects across STEM, the humanities, the social sciences, and more. It measures broad world knowledge and problem-solving, typically reported as average accuracy. Higher scores indicate better performance.
Top models (higher is better)
| Model | Score |
|---|---|
| Claude 3.5 Sonnet (June 2024) | 88.7 |
| Claude 3.5 Sonnet (Oct 2024) | 88.7 |
| Opus 3 | 88.2 |
| GPT-4o (2024-11-20) | 88.1 |
| Gemini 1.5 Pro 002 | 86.9 |
| GPT-4 | 86.4 |
| Gemini 1.5 Pro 001 | 85.9 |
| Qwen2.5 72B | 85.0 |
| Phi-4 | 84.8 |
| Llama 3.1 405B Instruct | 84.5 |
| Llama 3.1 405B | 84.4 |
| GPT-4o (2024-08-06) | 84.3 |
| GPT-4o | 84.2 |
| GPT-4 0613 | 82.4 |
| Qwen2 72B Instruct | 82.4 |
| Nova Pro | 82.0 |
| Claude 3 Sonnet | 81.5 |
| Llama 3.2 90B Vision Instruct | 80.3 |
| Llama 3.1 70B Instruct | 80.1 |
| Qwen2.5 14B Instruct | 79.9 |
| Gemini 2.0 Flash Experimental | 79.7 |
| Llama 3 70B Instruct | 79.3 |
| Yi-Large | 79.3 |
| Claude 2 | 78.5 |
| DeepSeek-V2 | 78.4 |
| PaLM 2-L | 78.4 |
| Jamba 1.5 Large | 78.2 |
| Mixtral 8x22B | 77.8 |
| Haiku 3.5 | 77.6 |
| Claude 1.3 | 77.0 |
| Nova Lite | 77.0 |
| Qwen1.5 110B Chat | 76.8 |
| Claude 3 Haiku | 76.7 |
| Gemma 2 27B IT | 75.7 |
| Phi-3-Small-8K-Instruct | 75.7 |
| Qwen2.5-Coder-14B | 75.2 |
| Qwen1.5 32B | 74.4 |
| DBRX Instruct | 74.1 |
| Gemini 1.5 Flash 002 | 73.9 |
| Claude 2.1 | 73.5 |