Yi-34B
01.AI · 2023-11-02 · 34.4B parameters
Yi-34B is 01.AI's open-weight bilingual base language model, released on November 2, 2023 and reported as pretrained from scratch on 3.1 trillion tokens of English and Chinese corpora. It is a dense decoder-only Transformer with grouped-query attention, SwiGLU, and RoPE using a 5,000,000 base period. The checkpoint was trained and configured at 4,096 tokens; 01.AI documents inference extension to 32K tokens, distinct from the separately released Yi-34B-200K checkpoint, but does not state the extension as an exact integer.