Atlas

Benchmarks

← All benchmarks

MMMU

Multimodal · 2023-11-27

MMMU evaluates expert-level multimodal understanding and reasoning across college-level subject areas using image-question pairs. Scores in this dataset are reported as accuracy/pass@1, commonly on the validation split when model cards label the metric as MMMU-Val.

Top models (higher is better)

ModelScore
o382.9
Gemini 2.5 Pro82.0
Gemini 2.5 Pro Experimental 03-2581.7
O4 Mini81.6
Opus 4.580.7
Gemini 2.5 Flash Preview (09-2025)80.3
Gemini 2.5 Flash79.7
Gemini 2.5 Pro Preview 05-0679.6
Gemini 2.5 Flash Preview 04-1776.7
Opus 476.5
Grok 376.0
Claude 3.7 Sonnet75.0
Sonnet 474.4
Gemini 2.5 Flash-Lite Preview (09-2025)74.0
Haiku 4.573.2
Gemini 2.5 Flash-Lite72.9
Claude 3.5 Sonnet (Oct 2024)70.4
Claude 3.5 Sonnet (June 2024)68.3
Gemini 2.0 Flash-Lite68.0
Opus 359.4
GPT-4 Turbo with Vision56.8
Claude 3 Sonnet53.1
Gemini 1.5 Flash-8B50.3
Claude 3 Haiku50.2
Gemini 1.0 Pro47.9
Gemini Nano 232.6
Gemini Nano 126.3
Loading Atlas data…