Phi-1.5
Microsoft Research · 2023-09-11 · 1.4B parameters
Phi-1.5 is Microsoft's open-weight 1.3B-class dense base Transformer, released on September 11, 2023 without instruction tuning or RLHF. Microsoft trained it from random initialization for a reported 150B tokens using a 30B-token dataset composed almost entirely of synthetic textbook-like NLP and coding data; the only non-synthetic component was 6B tokens of filtered code data, not generic web-crawl text. Its 24-layer, 32-head architecture uses a 2,048-token context, partial rotary embeddings over half of each attention head, the CodeGen-Mono tokenizer, Adam, and fp16 training.