Confabulations Leaderboard Confabulation Rate
Safety · 2024-10-10
This metric is the percentage of leaderboard answers judged to be confabulated. Lower is better, and it contributes 50% of the published weighted score.
Top models (lower is better)
| Model | Score |
|---|---|
| Sonnet 4 | 2.5 |
| Opus 4 | 2.5 |
| Opus 4.1 | 2.5 |
| Gemini 2.5 Pro Preview 03-25 | 4.0 |
| Gemini 2.5 Pro | 4.0 |
| Grok 4 | 4.0 |
| Gemini 2.5 Flash Preview 04-17 | 4.5 |
| Gemini 2.5 Pro Preview 05-06 | 5.9 |
| Grok 3 Mini | 6.9 |
| GLM-4.5 | 7.9 |
| Claude 3.7 Sonnet | 7.9 |
| GPT-5 | 10.9 |
| O1 | 10.9 |
| GPT-4.5 | 11.9 |
| Qwen3-30B-A3B | 12.9 |
| DeepSeek-R1-0528 | 12.9 |
| Claude 3.5 Sonnet (Oct 2024) | 12.9 |
| Qwen3 235B A22B Thinking 2507 | 13.9 |
| Llama 3.1 405B | 14.4 |
| Gemini 2.0 Flash Thinking Experimental 01-21 | 14.9 |
| Gemini 2.0 Pro Experimental 02-05 | 15.8 |
| Gemini 1.5 Pro 002 | 16.8 |
| DeepSeek-R1 | 17.3 |
| Grok 3 | 17.8 |
| Llama 3.3 70B Instruct | 17.8 |
| o1 Preview | 18.3 |
| GPT-5 Mini | 21.8 |
| GPT-4o (2024-08-06) | 22.3 |
| Qwen3-235B-A22B | 23.3 |
| gpt-oss-120b | 23.3 |
| o3-pro | 23.4 |
| Gemini 2.0 Flash | 24.3 |
| o3 | 24.8 |
| QwQ-32B | 25.2 |
| ERNIE 4.5 300B A47B | 25.2 |
| Grok 2 | 25.7 |
| GPT-4o (2024-11-20) | 26.2 |
| o1-mini | 26.2 |
| O4 Mini | 26.7 |
| ChatGPT-4o Latest observed 2025-01-29 | 26.7 |