XGen-7B-8K-Base
Salesforce AI Research · 2023-06-28
XGen-7B-8K-Base is Salesforce AI Research's 7B-class decoder-only transformer language model designed for long-sequence modeling. It was initialized from XGen-7B-4K-Base and continued for 300 billion tokens at 8K sequence length, bringing total pretraining to 1.5 trillion text-and-code tokens and extending the practical context beyond the then-common 2K or 4K limits.
Benchmark scores
| Benchmark | Score |
|---|---|
| ARC AI2 | 41.2 |
| BoolQ | 74.3 |
| Capability | 84.5 |
| Epoch Capabilities Index (ECI) | 91.8 |
| HellaSwag (Unspecified Scoring Protocol) | 74.2 |
| OpenBookQA | 40.2 |
| PIQA | 75.5 |
| WinoGrande | 64.9 |