Atlas

Benchmarks

← All benchmarks

ZeroEval CRUX

Code · 2024-08-01

CRUX code reasoning evaluated with the ZeroEval zero-shot prompt format, greedy decoding, and answer parser.

Top models (higher is better)

ModelScore
o1 Preview95.9
o1-mini93.8
GPT-4o (2024-08-06)87.0
ChatGPT-4o Latest (September 2024 benchmark entry)86.5
GPT-4o86.1
Claude 3.5 Sonnet (Oct 2024)83.9
Claude 3.5 Sonnet (June 2024)80.8
Gemini 1.5 Pro Experimental 082779.6
GPT-4 Turbo78.9
GPT-4o Mini75.9
Grok 2 121275.3
Mistral Large 2 (Instruct 2407)75.1
Gemini 1.5 Pro Experimental 080174.9
GPT-474.5
Gemini 1.5 Flash Experimental 082774.5
Llama 3.1 405B Instruct74.1
Qwen2.5 72B Instruct73.9
DeepSeek-V2-Chat-062870.5
Opus 370.4
DeepSeek-V2.570.0
Qwen2.5 Instruct 32B69.9
DeepSeek-Coder-V2-Instruct-072469.5
Haiku 3.568.8
Gemini 1.5 Pro68.0
Claude 3 Sonnet66.6
Llama 3.1 70B Instruct64.3
Gemini 1.5 Flash63.8
Yi-Large60.4
Qwen2 72B Instruct59.1
Llama 3 70B Instruct58.9
Gemma 2 27B IT57.3
Claude 3 Haiku54.8
GPT-3.5 Turbo 012554.8
Qwen2.5 7B Instruct52.8
Athene-70B50.6
Reka Core 20240501 (ZeroEval label)46.3
Gemma 2 9B IT46.0
Mixtral 8x7B Instruct44.9
Phi-3-mini-4k-instruct44.8
Yi 1.5 9B Chat44.8
Loading Atlas data…