Yi-6B
01.AI · 2023-11-02 · 6.1B parameters
Yi-6B is 01.AI's open-weight bilingual English/Chinese base language model, released on November 2, 2023. It is a dense decoder-only Transformer reported as pretrained from scratch on 3.1 trillion tokens and uses GQA, SwiGLU, RMSNorm, and RoPE with a 5,000,000 base period. The checkpoint is configured and pretrained for 4,096-token sequences; 01.AI states that it can be extended to 32K at inference but does not publish an exact maximum beyond the native length.