HumanEval
Code · 2021-07-07
HumanEval is a code-generation benchmark of 164 hand-written Python programming problems, each with hidden unit tests. Models are scored by pass@k, the fraction of problems solved by generated code, most often reported as pass@1. Higher is better.
Top models (higher is better)
| Model | Score |
|---|---|
| Claude 3.5 Sonnet (Oct 2024) | 93.7 |
| Claude 3.5 Sonnet (June 2024) | 92.0 |
| Haiku 3.5 | 88.1 |
| Opus 3 | 84.9 |
| Gemini 1.5 Pro | 84.1 |
| Claude 3 Haiku | 75.9 |
| Gemini 1.0 Ultra | 74.4 |
| Gemini 1.5 Flash | 74.3 |
| Claude 3 Sonnet | 73.0 |
| Gemini 1.0 Pro | 67.7 |
| GPT-4 | 67.0 |
| Grok 1 | 63.2 |