INTELLECT-1
Prime Intellect · 2024-11-29 · 10.2B parameters
INTELLECT-1 is Prime Intellect's open-weight base language model, a 10B-class Llama 3-style dense transformer released on November 29, 2024 and trained collaboratively across five countries and three continents. Prime Intellect reports pretraining on roughly one trillion tokens of English text and code using PRIME's hybrid FSDP2/DiLoCo setup, scaling to up to 112 H100 GPUs; custom int8 pseudo-gradient all-reduce reduced communication bandwidth by 400× relative to traditional data-parallel training.