Models

← All models

GPT-NeoX-20B

EleutherAI · 2022-04-07 · 20.6B parameters

GPT-NeoX-20B is EleutherAI's dense 20,554,567,680-parameter autoregressive language model trained on the Pile, with weights released under Apache-2.0 in April 2022. It uses 44 transformer layers, a 2,048-token sequence length, rotary positional embeddings on 25% of the embedding dimensions, and parallel attention and feed-forward blocks. At paper submission, the authors described it as the largest dense autoregressive model with publicly available weights.

Benchmark scores