Qwen1.5 MoE A2.7B Chat
Alibaba · 2024-03-28 · 14.3B parameters
Qwen1.5-MoE-A2.7B-Chat is Alibaba's chat-tuned Mixture-of-Experts model from the Qwen1.5 family, released March 28, 2024. The model has 14.3 billion total parameters and activates about 2.7 billion per token through 64 experts (4 shared plus 60 routed, with 4 routed experts selected), and Alibaba measured approximately 1.74× the inference speed of Qwen1.5-7B while reporting comparable performance. It was upcycled from the Qwen-1.8B dense model and supports a 32K-token context window.