MPT-30B
MosaicML · 2023-06-22 · 30B parameters
MPT-30B is MosaicML's Apache-2.0 decoder-only base transformer, released on June 22, 2023, with an 8,192-token trained context and ALiBi-based context extrapolation. It was pretrained on 1T tokens at 2,048 sequence length and then continued for an additional 50B tokens at 8,192; the final stage used 256 H100-80GB GPUs, making it, to MosaicML's knowledge, the first public model partially trained on H100s. The checkpoint uses 48 layers, 64-head multi-head attention with FlashAttention support, and the GPT-NeoX-20B BPE tokenizer.
Benchmark scores
| Benchmark | Score |
|---|---|
| ARC AI2 | 50.6 |
| BBH (BIG-Bench Hard) | 38.0 |
| Capability | 91.6 |
| Epoch Capabilities Index (ECI) | 99.3 |
| OpenBookQA | 52.0 |
| PIQA | 81.9 |
| WinoGrande | 71.0 |