MPT-30B
MosaicML · 2023-06-22 · 30B parameters
MPT-30B is MosaicML's Apache-2.0 decoder-only base transformer, released on June 22, 2023, with an 8,192-token trained context and ALiBi-based context extrapolation. It was pretrained on 1T tokens at 2,048 sequence length and then continued for an additional 50B tokens at 8,192; the final stage used 256 H100-80GB GPUs, making it, to MosaicML's knowledge, the first public model partially trained on H100s. The checkpoint uses 48 layers, 64-head multi-head attention with FlashAttention support, and the GPT-NeoX-20B BPE tokenizer.
Benchmark scores
| Benchmark | Score |
|---|---|
| Epoch Capabilities Index (ECI) | 100 |