Zephyr 7B Beta
Hugging Face · 2023-10-26 · 7.2B parameters
Zephyr 7B Beta is the second model in Hugging Face H4’s Zephyr assistant series and a full-parameter fine-tune of Mistral 7B v0.1. The final checkpoint was initialized from one epoch of distilled supervised fine-tuning on the filtered UltraChat corpus, then optimized for three DPO epochs on binary preferences constructed from UltraFeedback. It retains the dense 32-layer Mistral architecture with grouped-query attention, a 4,096-token sliding window, and a 32,768-position configuration; the reported experiments used AdamW with no weight decay and bfloat16 precision.