XGen-7B-8K-Base
Salesforce AI Research · 2023-06-28
XGen-7B-8K-Base is Salesforce AI Research's 7B-class decoder-only transformer language model designed for long-sequence modeling. It was initialized from XGen-7B-4K-Base and continued for 300 billion tokens at 8K sequence length, bringing total pretraining to 1.5 trillion text-and-code tokens and extending the practical context beyond the then-common 2K or 4K limits.
Benchmark scores
| Benchmark | Score |
|---|---|
| Epoch Capabilities Index (ECI) | 92.8 |