DeepSeek-V2
DeepSeek · 2024-05-06 · 235.7B parameters
DeepSeek-V2 is DeepSeek's pretrained base checkpoint released May 6, 2024; its downloadable weights contain exactly 235,741,434,880 parameters, while official architecture figures are rounded to 236B total and 21B activated per token. It was pretrained on a reported 8.1T-token corpus with MLA and DeepSeekMoE (2 shared and 160 routed experts per MoE layer, 6 routed experts selected per token), then extended with YaRN from 4K to documented 128K context support; its released configuration declares 163,840 maximum positions.
Benchmark scores
| Benchmark | Score |
|---|---|
| ARC AI2 | 92.2 |
| BBH (BIG-Bench Hard) | 78.8 |
| Capability | 119.3 |
| Epoch Capabilities Index (ECI) | 124 |
| HellaSwag (Unspecified Scoring Protocol) | 87.1 |
| MMLU | 78.4 |
| PIQA | 83.9 |
| TriviaQA | 80.0 |
| WinoGrande | 86.3 |