Falcon 7B
Technology Innovation Institute · 2023-04-24 · 7.2B parameters
Falcon 7B is TII's (Technology Innovation Institute) 7B causal decoder-only base language model, trained on 1.5 trillion tokens of RefinedWeb plus curated corpora and released in early 2023 under the Apache 2.0 license. Its architecture features multi-query attention, FlashAttention, and parallel attention/MLP blocks optimized for inference efficiency, and it served as the foundation for the Falcon-7B-Instruct instruction-tuned variant.