Baichuan-7B
Baichuan · 2023-06-15 · 7B parameters
Baichuan-7B is Baichuan Intelligent Technology's open-weight bilingual Chinese-English pretrained base language model, released in June 2023 and trained on approximately 1.2 trillion tokens. It is a dense decoder-only Transformer with 7,000,559,616 parameters, a 4,096-token context window, 32 layers, RoPE, SwiGLU, RMSNorm, and multi-head attention. Its 64,000-token tokenizer uses SentencePiece BPE with UTF-8 byte fallback.
Benchmark scores
| Benchmark | Score |
|---|---|
| BBH (BIG-Bench Hard) | 32.5 |
| Capability | 82.4 |
| Epoch Capabilities Index (ECI) | 88.8 |
| GSM8K | 9.2 |
| MMLU | 42.3 |
| PIQA | 76.2 |