BBH (BIG-Bench Hard)
General QA · 2022-10-17
A curated subset of BIG-bench tasks that remain difficult for large language models, probing compositional, symbolic, and multi-step reasoning.
Top models (higher is better)
| Model | Score |
|---|---|
| Claude 3.5 Sonnet (Oct 2024) | 93.2 |
| Claude 3.5 Sonnet (June 2024) | 93.1 |
| Gemini 1.5 Pro | 89.2 |
| Opus 3 | 86.8 |
| Haiku 3.5 | 86.6 |
| Gemini 1.5 Flash | 85.5 |
| Gemini 1.0 Ultra | 83.6 |
| GPT-4 | 83.1 |
| Claude 3 Sonnet | 82.9 |
| PaLM 2-L | 77.7 |
| Gemini 1.0 Pro | 75.0 |
| Claude 3 Haiku | 73.7 |
| Gemini 1.5 Flash-8B | 69.5 |
| Gemini Nano 2 | 42.4 |
| Gemini Nano 1 | 34.8 |