Name: Gemma 2 27B
Author: Google DeepMind

Gemma 2 27B

Google DeepMind · 2024-06-27 · 27.2B parameters

Gemma 2 27B is Google DeepMind's open-weight, text-to-text, decoder-only pre-trained checkpoint in the Gemma 2 family, released on June 27, 2024. Trained from scratch on 13 trillion primarily English tokens, it alternates 4,096-token local and 8,192-token global attention layers with grouped-query attention; Google designed full-precision inference for a single A100 80GB or H100 GPU or a Google Cloud TPU host.

Benchmark scores

Benchmark	Score
Arabic	661
BBH (Open LLM Leaderboard v2)	56.4
Capability	119.7
Chinese	966
Confabulations Leaderboard Confabulation Rate	47.0
Confabulations Leaderboard Non-Response Rate	7.2
Confabulations Leaderboard Weighted Score	27.1
Epoch Capabilities Index (ECI)	122
Global PIQA - Non-Parallel (Log-Likelihood Accuracy)	63.4
Global PIQA - Parallel (Log-Likelihood Accuracy)	30.7
GPQA (Open LLM Leaderboard v2)	35.1
IFEval (Open LLM Leaderboard v2)	24.8
Japanese	892
LLM Deception Disinformation Effectiveness	0.5
LLM Deception Vulnerability Score	1.4
LLM Divergent Thinking Repeat Rate	1.1
LLM Divergent Thinking Score	4.4
MATH Level 5 (Open LLM Leaderboard v2)	16.6
MMLU-Pro (Open LLM Leaderboard v2)	43.7
MuSR (Open LLM Leaderboard v2)	44.0
Public Goods Game Average Contribution	31.2
Spanish	951
Step Game TrueSkill Mu	0.7