StarCoder2-3B
Hugging Face · 2024-02-28 · 3B parameters
StarCoder2-3B is an open-weight code language model from the BigCode Project, trained by ServiceNow as the smallest member of the 3B, 7B, and 15B StarCoder2 family. Its report describes an approximately 3.1-trillion-token base stage using 4,096-token sequences followed by an approximately 200-billion-token long-context stage, with fill-in-the-middle training, grouped-query attention, and 4,096-token sliding-window attention within a 16,384-token context. The authors report that it outperforms comparable small code models on most evaluated benchmarks and can match or surpass StarCoderBase-15B.