DeepSeek-V2
DeepSeek · 2024-05-06 · 235.7B parameters
DeepSeek-V2 is DeepSeek's pretrained base checkpoint released May 6, 2024; its downloadable weights contain exactly 235,741,434,880 parameters, while official architecture figures are rounded to 236B total and 21B activated per token. It was pretrained on a reported 8.1T-token corpus with MLA and DeepSeekMoE (2 shared and 160 routed experts per MoE layer, 6 routed experts selected per token), then extended with YaRN from 4K to documented 128K context support; its released configuration declares 163,840 maximum positions.
Benchmark scores
| Benchmark | Score |
|---|---|
| Epoch Capabilities Index (ECI) | 125 |