MPT-7B
MosaicML · 2023-05-05 · 6.6B parameters
MPT-7B is MosaicML's dense decoder-only base transformer released on May 5, 2023; the published checkpoint contains 6,649,286,656 parameters. MosaicML reports that it was trained from scratch on 1 trillion tokens of text and code using ALiBi and the Lion optimizer. Its 2,048-token base checkpoint was released under Apache-2.0, while the Instruct, Chat, and StoryWriter releases were separate fine-tuned artifacts.