Vectara Hallucination Leaderboard

Name: Vectara Hallucination Leaderboard
Published: 2025-11-18

Safety · 2025-11-18

Vectara's hallucination leaderboard: the share of a model's summaries that Vectara's commercial HHEM-2.3 judge finds factually inconsistent with the source article; lower is better. Models summarize over 7,700 private articles (news, science, medicine, legal, business and more, 50 to 24K words) using only facts in the article, at temperature 0 where the API allows it. The rate is computed over answered prompts; answer rate is reported separately. This is the dataset v2 leaderboard Vectara launched on November 18, 2025; the earlier dataset is not imported.

Top models (lower is better)

Model	Score
GPT-5.4 Nano	3.1
Gemini 2.5 Flash-Lite	3.3
Phi-4	3.7
Llama 3.3 70B Instruct	4.1
Arctic Instruct	4.3
Gemma 3 12B IT	4.4
Mistral Large 2.1 (Instruct 2411)	4.5
Qwen3 8B	4.8
Mistral Small 3 24B Instruct 2501	5.1
Nova 2 Lite	5.1
Nova Pro	5.1
Gemma 4 26B A4B IT	5.2
Granite 4.0 H Small	5.2
DeepSeek-V3.2-Exp	5.3
Qwen3 14B	5.4
DeepSeek-V3.1	5.5
GPT-5.4 Mini	5.5
Nova Micro	5.5
GPT-4.1	5.6
Qwen3 4B	5.7
Grok 3 (API)	5.8
Qwen3 32B	5.9
DeepSeek-V3	6.1
Nova Lite	6.1
DeepSeek-V3.2	6.3
Gemma 3 4B IT	6.4
GPT-6 Sol	6.5
C4AI Command R+ 08-2024	6.9
Trinity Large Preview	6.9
Gemini 2.5 Pro	7.0
GPT-5.4	7.0
Ministral 3B	7.3
Gemma 3 27B IT	7.4
Gemma 4 31B IT	7.4
Ministral 8B Instruct 2410	7.4
Llama 4 Scout Instruct	7.7
Gemini 2.5 Flash	7.8
Gemini 3.1 Flash-Lite Preview	8.2
Llama 4 Maverick Instruct FP8	8.2
GPT-5.4 Pro	8.3