MMLU Anatomy (Open Medical-LLM)
Science · 2024-04-19
The anatomy subset of MMLU, 135 questions, scored by the Open Medical-LLM Leaderboard under lm-evaluation-harness. Carried as its own family rather than pooled into a general MMLU score, because the leaderboard reports the medical subsets separately and pooling them would hide which clinical area a model is weak in.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-4 | 80.0 |
| Llama 3 8B | 68.9 |
| GPT-3.5 Turbo 1106 | 65.9 |
| Llama 3 8B Instruct | 62.2 |
| Gemma 7B | 58.5 |
| Solar 10.7B Instruct V1.0 | 57.8 |
| Zephyr 7B Beta | 57.0 |
| Mistral 7B v0.1 Base | 55.6 |
| Qwen1.5 7B Chat | 54.8 |
| Gemma 1.1 7B IT | 53.3 |
| Qwen1.5 7B | 49.6 |
| Gemma 7B IT | 48.1 |
| Mistral 7B Instruct v0.1 | 45.2 |
| Phi-1.5 | 39.3 |
| Gemma 2B | 37.0 |
| Falcon 7B | 25.2 |