DeepSeek-V3-Base
DeepSeek · 2024-12-26 · 684.5B parameters
DeepSeek-V3-Base is the pretrained base checkpoint of DeepSeek-V3, released December 26, 2024; DeepSeek reports 671B main-model parameters and 37B activated per token, while the downloadable checkpoint also includes a 14B multi-token prediction module. It was pretrained on 14.8 trillion tokens with auxiliary-loss-free load balancing and multi-token prediction, then extended with YaRN from 4K to 128K context before the SFT and RL stages used to produce the chat model.
Benchmark scores
| Benchmark | Score |
|---|
No benchmark scores recorded.