BBH (BIG-Bench Hard)
General QA · 2022-10-17
A curated subset of BIG-bench tasks that remain difficult for large language models, probing compositional, symbolic, and multi-step reasoning.
Top models (higher is better)
| Model | Score |
|---|---|
| Claude 3.5 Sonnet (Oct 2024) | 93.2 |
| Claude 3.5 Sonnet (June 2024) | 93.1 |
| Gemini 1.5 Pro 001 | 89.2 |
| Opus 3 | 86.8 |
| Haiku 3.5 | 86.6 |
| Gemini 1.5 Flash | 85.5 |
| Gemini 1.0 Ultra | 83.6 |
| GPT-4 | 83.1 |
| Llama 3.1 405B | 82.9 |
| Claude 3 Sonnet | 82.9 |
| Phi-3 Medium 128K Instruct | 81.4 |
| Qwen2.5 72B | 79.8 |
| Phi-3-Small-8K-Instruct | 79.1 |
| DeepSeek-V2 | 78.8 |
| PaLM 2-L | 77.7 |
| GPT-4 0613 | 75.1 |
| Gemini 1.0 Pro | 75.0 |
| Claude 3 Haiku | 73.7 |
| Phi-3-mini-4k-instruct | 71.7 |
| Gemini 1.5 Flash-8B | 69.5 |
| Stable Beluga 2 | 69.3 |
| GPT-3.5 Turbo 0613 | 61.6 |
| Phi-2 | 59.4 |
| Nemotron-4 15B | 58.7 |
| Gemma 7B | 55.1 |
| Yi-34B | 54.3 |
| Qwen 14B | 53.4 |
| InternLM 20B | 52.5 |
| Qwen 7B | 45.0 |
| Llama 2 34B | 44.1 |
| Vicuna 13B v1.1 | 43.0 |
| Yi-6B | 42.8 |
| Gemini Nano 2 | 42.4 |
| MPT-30B | 38.0 |
| Falcon 40B | 37.1 |
| InternLM 7B | 37.0 |
| Gemma 2B | 35.2 |
| Gemini Nano 1 | 34.8 |
| Baichuan-7B | 32.5 |
| Qwen 1.8B | 28.2 |