Name: Gemini 1.5 Pro
Author: Google DeepMind

Gemini 1.5 Pro

Google DeepMind · 2024-02-15

Gemini 1.5 Pro is Google's long-context multimodal model, first announced for early testing on February 15, 2024 and later served with a 2,097,152-token input limit. It is a sparse mixture-of-experts Transformer that accepts text, images, audio, and video and produces text.

Benchmark scores

Benchmark	Score
A-Fantasia	62.3
A-Fantasia Backwards Spelling	88.0
A-Fantasia Chess	35.0
A-Fantasia Cube Rotation	64.0
AI2D (OpenVLM)	79.1
AidanBench	1487
Artificial Analysis Coding Index	19.8
Artificial Analysis Intelligence Index	6.3
BuzzBench	54.9
Capability	123.4
Cybench	7.5
Epoch Capabilities Index (ECI)	127
ForecastBench Baseline Dataset Brier Index	57.8
ForecastBench Baseline Market Brier Index	59.6
ForecastBench Baseline Overall Brier Index	58.7
ForecastBench Preliminary Dataset Brier Index	57.8
ForecastBench Tournament Dataset Brier Index	57.8
ForecastBench Tournament Market Brier Index	63.6
ForecastBench Tournament Overall Brier Index	60.0
FORTRESS Average Risk Score (ARS)	53.9
Global MMLU Lite	80.8
GSM8K	90.8
HallusionBench (OpenVLM)	45.6
HellaSwag (Unspecified Scoring Protocol)	93.3
HumanEval	84.1
Humanity's Last Exam (Text-Only)	3.9
LisanBench Difficulty-Weighted	120
LisanBench Path Length	492
LiveCodeBench	34.2
LiveCodeBench (Artificial Analysis)	24.4