Cerebras-GPT 13B
Cerebras Systems · 2023-03-28 · 12.9B parameters
Cerebras-GPT 13B is the largest checkpoint in Cerebras Systems' open, compute-optimal GPT family, released on March 28, 2023 and trained on the Pile with a GPT-3-style decoder-only architecture. Its training exposure was reported as 257.1 billion tokens under the Chinchilla scaling recipe on the 16-CS-2 Andromeda AI supercomputer. Cerebras released the weights, training code, and checkpoints under the Apache 2.0 license.
Benchmark scores
| Benchmark | Score |
|---|---|
| ARC AI2 | 32.4 |
| BoolQ | 61.1 |
| Capability | 71.7 |
| Epoch Capabilities Index (ECI) | 81.3 |
| HellaSwag (Unspecified Scoring Protocol) | 59.4 |
| OpenBookQA | 35.8 |
| PIQA | 73.5 |
| WinoGrande | 60.8 |