Vectara HHEM-2.3

82 models with a score. Leader: GPT-5.4 nano with 3.1 %.

Benchmark profile
Task	Zusammenfassung - Treue zur Vorlage
Metric	Anteil der Zusammenfassungen mit mindestens einer Aussage, die im Quelldokument nicht steht
Direction	↓ lower is better
Limitation	Nur Englisch, und die Antwortrate gehoert zur Zahl: wer ein Drittel der Dokumente verweigert, halluziniert auf dem Rest seltener. Gemessen wird Treue zur Vorlage, nicht Nuetzlichkeit - ein Modell, das abschreibt, gewinnt.
As of	11 May 2026
Source	github.com
Licence	Apache-2.0 — Vectara Hallucination Leaderboard

Ranking (82)

Model	Rank	Score	Answer rate	Developer	EU processing: InputEU in	EU processing: OutputEU out	all providers: Inputall in	all providers: Outputall out	Open weights	GPU memory (4-bit, derived)GPU memory
GPT-5.4 nano	1	3.1 %	100 %	OpenAI	–	–	€0.09	€0.56	– no	–
Gemini 2.5 Flash-Lite	2	3.3 %	99.5 %	Google	–	–	€0.045	€0.18	– no	–
Phi-4	3	3.7 %	80.7 %	Microsoft	–	–	€0.063	€0.13	✓ yes	from 13 GB
Llama 3.3 70B	4	4.1 %	99.5 %	Meta	€0.45	€0.65	€0.09	€0.29	✓ yes	from 43 GB
Gemma 3 12B	5	4.4 %	97.4 %	Google	–	–	€0.045	€0.13	✓ yes	from 11 GB
Mistral Large 2.1	6	4.5 %	99.9 %	Mistral AI	–	–	–	–	✓ yes	from 72 GB
Qwen3 8B	7	4.8 %	99.9 %	Alibaba Cloud	–	–	–	–	✓ yes	from 9 GB
Mistral Small 3 24B	8	5.1 %	97.9 %	Mistral AI	–	–	€0.045	€0.072	✓ yes	from 17 GB
Nova 2 Lite	8	5.1 %	99.6 %	Amazon	€0.34	€2.82	€0.15	€1.23	– no	–
Nova Pro	8	5.1 %	99.3 %	Amazon	€0.82	€3.29	€0.36	€1.43	– no	–
Gemma 4 26B A4B	11	5.2 %	99.8 %	Google	€0.11	€0.27	€0.063	€0.30	–	–
Granite 4.0 H Small	11	5.2 %	100 %	IBM	–	–	–	–	✓ yes	from 22 GB
Qwen3-14B	13	5.4 %	99.9 %	Alibaba Cloud	€0.18	€0.90	€0.11	€0.21	✓ yes	from 13 GB
DeepSeek V3.1	14	5.5 %	94.5 %	DeepSeek	–	–	€0.22	€0.85	✓ yes	from 374 GB
GPT-5.4 mini	14	5.5 %	100 %	OpenAI	€0.74	€4.43	€0.34	€2.02	– no	–
Nova Micro	14	5.5 %	100 %	Amazon	€0.036	€0.14	€0.031	€0.13	– no	–
GPT-4.1	17	5.6 %	99.9 %	OpenAI	€1.97	€7.87	€0.90	€3.58	– no	–
GPT-4.1-2025-04-14	17	5.6 %	99.9 %	OpenAI	€0.99	€3.94	€0.90	€3.58	– no	–
Qwen3-32B	19	5.9 %	99.9 %	qwen-alibaba	€0.28	€0.28	€0.072	€0.25	✓ yes	from 23 GB
DeepSeek-V3	20	6.1 %	97.5 %	DeepSeek	–	–	€0.29	€0.80	✓ yes	from 381 GB
Nova Lite	20	6.1 %	99.9 %	Amazon	€0.062	€0.25	€0.054	€0.21	– no	–
deepseek-v3.2	22	6.3 %	92.6 %	DeepSeek	€0.35	€0.58	€0.23	€0.34	✓ yes	from 381 GB
Gemma 3 4B	23	6.4 %	67.3 %	Google	–	–	€0.018	€0.036	✓ yes	from 7 GB
GPT-6 sol	24	6.5 %	100 %	OpenAI	€2.15	€10.75	€0.90	€4.48	– no	–
Gemini 2.5 Pro	25	7 %	99.1 %	Google	€1.34	€8.94	€0.56	€4.48	– no	–
GPT-5.4	25	7 %	99.9 %	OpenAI	€2.60	€13.87	€1.12	€6.72	– no	–
Gemma 3 27B	27	7.4 %	98.8 %	Google	€0.089	€0.27	€0.072	€0.14	✓ yes	from 20 GB
Gemma 4 31B	27	7.4 %	92.6 %	Google	€0.09	€0.27	€0.09	€0.27	✓ yes	from 22 GB
Llama 4 Scout 17B-16E	29	7.7 %	99 %	Meta	–	–	€0.09	€0.27	✓ yes	from 64 GB
Gemini 2.5 Flash	30	7.8 %	99 %	Google	€0.27	€2.24	€0.13	€1.12	– no	–
Gemini 3.1 Flash-Lite	31	8.2 %	99.6 %	Google	€0.24	€1.46	€0.11	€0.67	– no	–
Llama 4 Maverick	31	8.2 %	100 %	Meta	–	–	€0.16	€0.52	✓ yes	from 225 GB
GPT-5.4 Pro	33	8.3 %	100 %	OpenAI	–	–	€13.43	€80.61	– no	–
DeepSeek V4 Pro	34	8.6 %	97.2 %	DeepSeek	€0.90	€1.79	€0.90	€1.79	✓ yes	from 884 GB
MiniMax M2.5	35	9.1 %	98.2 %	MiniMax	€0.31	€1.25	€0.13	€0.54	✓ yes	from 130 GB
GLM 4.7 Flash	36	9.3 %	91.6 %	Z.ai	€0.072	€0.43	€0.031	€0.18	✓ yes	from 21 GB
GPT-5.5	36	9.3 %	100 %	OpenAI	€4.93	€29.56	€2.24	€13.43	– no	–
Aya Expanse 8B	38	9.5 %	77.5 %	–	–	–	€0.45	€1.34	–	–
GLM 4.6	38	9.5 %	94.5 %	Z.ai	–	–	€0.39	€1.57	✓ yes	from 201 GB
GPT-4o	40	9.6 %	93.8 %	OpenAI	€2.39	€9.55	€1.12	€4.48	– no	–
GPT-4o-2024-08-06	40	9.6 %	93.8 %	OpenAI	€1.23	€4.93	€1.12	€4.48	– no	–
NVIDIA Nemotron 3 Nano 30B	40	9.6 %	99.6 %	NVIDIA	€0.054	€0.21	€0.027	€0.11	✓ yes	from 22 GB
Claude Haiku 4.5	43	9.8 %	99.5 %	Anthropic	€0.89	€4.47	€0.45	€2.24	– no	–
GLM-5	44	10.1 %	99.7 %	Z.ai	€1.15	€3.68	€0.45	€1.43	–	–
Claude Sonnet 4 (retired, except on Claude Models on Google Cloud (Google managed) and Bedrock)	45	10.3 %	98.6 %	Anthropic	–	–	€1.34	€6.72	– no	–
Gemini 3.1 Pro (Vorschau)	46	10.4 %	99.4 %	Google	–	–	€0.90	€5.37	– no	–
GPT-5 nano	47	10.5 %	100 %	OpenAI	€0.049	€0.39	€0.022	€0.18	– no	–
Qwen3.5-35B-A3B	47	10.5 %	99.8 %	Alibaba Cloud	–	–	€0.13	€0.90	✓ yes	from 24 GB
Claude Sonnet 4.6	49	10.6 %	99.9 %	Anthropic	€2.87	€14.31	€1.34	€6.72	– no	–
GPT-5.2	50	10.8 %	100 %	OpenAI	€1.72	€13.79	€0.78	€6.27	– no	–
Kimi K2.6	50	10.8 %	99.7 %	Moonshot AI	€0.43	€2.67	€0.35	€1.61	✓ yes	from 569 GB
Aya Expanse 32B	52	10.9 %	99.8 %	–	–	–	€0.45	€1.34	–	–
Opus 4.5	52	10.9 %	98.7 %	Anthropic	€4.77	€23.85	€2.24	€11.20	– no	–
Qwen3.5-122B-A10B	54	11.2 %	99.8 %	Alibaba Cloud	€0.22	€1.57	€0.22	€1.57	✓ yes	from 73 GB
GLM 4.7	55	11.7 %	99.8 %	Z.ai	–	–	€0.27	€0.99	✓ yes	from 201 GB
Claude Opus 4.1 (retired, except on Claude Models on Google Cloud (Google managed) and Bedrock)	56	11.8 %	92.4 %	Anthropic	–	–	€6.72	€33.59	– no	–
Minimax M2.1	56	11.8 %	98.5 %	MiniMax	€0.32	€1.29	€0.13	€0.54	✓ yes	from 130 GB
Claude Opus 4 (retired, except on Claude Models on Google Cloud (Google managed))	58	12 %	91 %	Anthropic	–	–	€6.72	€33.59	– no	–
Opus 4.7	58	12 %	98 %	Anthropic	€4.88	€24.40	€2.24	€11.20	– no	–
Sonnet 4.5	58	12 %	95.6 %	Anthropic	€2.68	€13.42	€1.34	€6.72	– no	–
Gemini 4 Argon	61	12.1 %	99.1 %	Google	€2.02	€10.10	€0.90	€4.48	– no	–
gpt-5.1	61	12.1 %	100 %	OpenAI	€1.23	€9.84	€0.56	€4.48	– no	–
Qwen3.5-27B	61	12.1 %	99.8 %	Alibaba Cloud	–	–	€0.23	€2.33	✓ yes	from 20 GB
Opus 4.6	64	12.2 %	99.8 %	Anthropic	€4.77	€23.84	€2.24	€11.20	– no	–
Mercury 2	65	12.3 %	100 %	inception	–	–	€0.28	€0.84	– no	–
GPT-5.6 sol	66	12.4 %	99.1 %	OpenAI	€3.94	€19.70	€1.79	€8.96	– no	–
GPT-5 mini	67	12.9 %	99.9 %	OpenAI	€0.25	€1.97	€0.11	€0.90	– no	–
MiniMax M2.7	67	12.9 %	99.4 %	MiniMax	–	–	€0.27	€1.07	✓ yes	from 130 GB
Gemini 3 Flash Preview	69	13.5 %	99.8 %	Google	–	–	€0.22	€1.34	– no	–
gpt-oss-120b	70	14.2 %	99.9 %	OpenAI	€0.026	€0.12	€0.026	€0.12	✓ yes	from 69 GB
Kimi K2.5 (Retiring)	70	14.2 %	92.2 %	Moonshot AI	€0.58	€3.22	€0.27	€1.34	✓ yes	from 554 GB
Mistral Large 3	72	14.5 %	98.8 %	Mistral AI	€0.55	€1.65	€0.22	€0.67	✓ yes	from 376 GB
gpt-5	73	15.1 %	99.9 %	OpenAI	€0.62	€4.93	€0.56	€4.48	– no	–
GPT-5 2025-08-07	73	15.1 %	99.9 %	OpenAI	–	–	€0.56	€4.48	– no	–
Grok 4.1 Fast Non-Reasoning	75	17.8 %	98.5 %	xAI	–	–	€0.18	€0.45	– no	–
Kimi K2 Instruct 0905	76	17.9 %	98.6 %	Moonshot AI	–	–	€0.51	€2.06	✓ yes	from 554 GB
o4-mini 2025-04-16	77	18.6 %	99.2 %	OpenAI	€0.54	€2.17	€0.49	€1.97	– no	–
Grok 4.1 Fast Reasoning	78	19.2 %	99.7 %	xAI	–	–	€0.18	€0.45	– no	–
Ministral 3 14B	79	19.4 %	99.6 %	Mistral AI	€0.21	€0.21	€0.09	€0.09	✓ yes	from 12 GB
Ministral 3 8B	80	21.7 %	99.1 %	Mistral AI	€0.16	€0.16	€0.063	€0.063	✓ yes	from 9 GB
o3-pro	81	23.3 %	100 %	OpenAI	–	–	€8.96	€35.83	– no	–
Ministral 3 3B	82	24.2 %	74.3 %	Mistral AI	€0.11	€0.11	€0.045	€0.045	✓ yes	from 7 GB

Equal scores share a rank. When filtering, the rank stays the position in the whole list. GPU memory “from …” is a lower bound (parameters × 0.55 GB + 4 GB). Prices: € per 1M tokens. Source: providers' pricing pages, read and checked by searchit. As of 7 Nov 2026.

Price and performance

79 models with a price: vertical the score (top = better), horizontal the blended price of the cheapest offer. Filled dots are also available with processing in the EU.

Scores: Vectara HHEM-2.3, Apache-2.0 — Vectara Hallucination Leaderboard. Taken unchanged from the source.