Name: ZeroEval MMLU-Redux
Published: 2024-08-01

ZeroEval MMLU-Redux

General QA · 2024-08-01

MMLU-Redux evaluated with the ZeroEval zero-shot prompt format, greedy decoding, and answer parser.

Model	Score
o1 Preview	92.8
Claude 3.5 Sonnet (Oct 2024)	88.9
ChatGPT-4o Latest (September 2024 benchmark entry)	88.9
GPT-4o (2024-08-06)	88.3
GPT-4o	88.0
Grok 2 1212	87.4
o1-mini	86.7
Llama 3.1 405B Instruct	86.2
Gemini 1.5 Pro Experimental 0827	86.1
Claude 3.5 Sonnet (June 2024)	86.0
Qwen2.5 72B Instruct	85.6
Gemini 1.5 Pro Experimental 0801	85.5
GPT-4 Turbo	85.3
Qwen2.5 Instruct 32B	83.7
Llama 3.1 70B Instruct	83.0
Mistral Large 2 (Instruct 2407)	83.0
Gemini 1.5 Pro	82.8
Opus 3	82.5
Yi-Large	82.2
Gemini 1.5 Flash Experimental 0827	82.1
GPT-4	81.6
Qwen2 72B Instruct	81.6
GPT-4o Mini	81.5
DeepSeek-V2-Chat-0628	80.8
DeepSeek-V2.5	80.3
DeepSeek-Coder-V2-Instruct-0724	80.2
DeepSeek-Coder-V2-Instruct	79.6
Haiku 3.5	79.6
Llama 3 70B Instruct	78.0
Gemini 1.5 Flash	77.4
Athene-70B	76.6
Reka Core 20240501 (ZeroEval label)	76.4
Gemma 2 27B IT	75.7
Qwen2.5 7B Instruct	75.1
Claude 3 Sonnet	74.9
Gemma 2 9B IT	72.8
Yi 1.5 34B Chat	72.8
Claude 3 Haiku	72.3
Phi-3-mini-4k-instruct	70.3
C4AI Command R+	68.6