Gemini 1.5 Flash (May 2024 report)
This benchmark entry represents the Gemini 1.5 Flash checkpoint evaluated in Google's May 2024 Gemini 1.5 technical report. It is a multimodal transformer-decoder model distilled from Gemini 1.5 Pro for efficient serving. The report does not establish a machine identifier that unambiguously maps its evaluated checkpoint to the preview-0514 or 001 API versions, so its reported results remain separate from those released artifacts and from source-unspecified aliases.
Benchmark scores
| Benchmark | Score |
|---|---|
| BBH (BIG-Bench Hard) | 85.5 |
| GSM8K | 86.2 |
| HellaSwag (Unspecified Scoring Protocol) | 86.5 |
| HumanEval | 74.3 |
| MATH | 54.9 |
| MathVista (mini) | 58.4 |
| MGSM | 82.6 |
| MMLU | 78.9 |
| MMMU | 56.1 |