Models

← All models

Gopher (280B)

Google DeepMind · 2021-12-08

Gopher is DeepMind's dense autoregressive Transformer language model, announced in December 2021 and reported at a rounded 280 billion parameters. It was trained on 300 billion tokens from MassiveText with a 2,048-token training context; relative-position clamping let the paper evaluate longer sequences, but no exact maximum supported context was stated. Evaluated across 152 tasks, it set state of the art on 100 of the 124 tasks with comparable prior language-model results, with its largest scaling gains in reading comprehension, fact-checking, and toxic-language identification.

Benchmark scores