Qwen2-72B
Alibaba · 2024-06-07 · 72.7B parameters
Qwen2-72B is Alibaba's 72.7-billion-parameter base language model from the Qwen2 series, released June 7, 2024. It is a dense decoder-only Transformer with RMSNorm, SwiGLU, grouped-query attention, and RoPE, pretrained on a multilingual corpus of over 7 trillion tokens with increased code and mathematics coverage. Its concluding pretraining phase used 32,768-token sequences, while YaRN and Dual Chunk Attention enable extrapolated processing up to 131,072 tokens; as a base model it is intended for further post-training and underlies Qwen2-72B-Instruct.