Models

← All models

PaLM 62B

Google DeepMind · 2022-04-04

PaLM 62B is the middle scale of the original Google Pathways Language Model family, introduced in April 2022 as a dense decoder-only Transformer for studying scaling and few-shot capabilities. The paper evaluates a checkpoint reported at 795B pretraining tokens due to a checkpoint-selection oversight; both that exposure and the reported 62.50B parameter total are rounded, while 2,048 is documented only as the training sequence length rather than a supported maximum context. Its architecture uses parallel attention/MLP layers, multi-query attention, RoPE, and a 256k SentencePiece vocabulary.

Benchmark scores