Mistral NeMo Base 2407
Mistral AI · 2024-07-18 · 12.2B parameters
Mistral NeMo Base 2407 is the pretrained base checkpoint of Mistral NeMo, a dense text-decoder Transformer jointly developed by Mistral AI and NVIDIA and released on July 18, 2024. The 12,247,782,400-parameter Apache-2.0 checkpoint has a 131,072-token context window, grouped-query attention, and the 131,072-ID Tekken tokenizer, which Mistral says is based on tiktoken and trained on more than 100 languages. Mistral trained the model with quantization awareness to support FP8 inference without performance loss and positioned it as a drop-in replacement for systems using Mistral 7B.