Orion-14B-Chat
OrionStar · 2024-01-16 · 14.5B parameters
Orion-14B-Chat is OrionStar's open-weight chat checkpoint, fine-tuned from its multilingual Orion-14B base with AdamW for three epochs over more than 500,000 supervised samples at a 4,096-token sequence length. The base model was pretrained from scratch on a reported 2.5-trillion-token corpus dominated by English and Chinese and also covering Japanese, Korean, and other languages. The dense 40-layer decoder uses LayerNorm, RoPE, multi-head attention, and a SiLU-gated feed-forward network; OrionStar separately released long-context, RAG, plugin, and quantized variants.
Benchmark scores
| Benchmark | Score |
|---|---|
| EQ-Bench v2 | 59.7 |
| EQ-Bench v2 + MAGI-Hard Combined | 50.2 |
| MAGI-Hard | 40.7 |