InternLM 20B
Shanghai AI Laboratory · 2023-09-20 · 20.1B parameters
InternLM 20B is the first-generation base checkpoint released by Shanghai AI Laboratory and collaborators on September 20, 2023. It is a dense 60-layer decoder-only text model pretrained on more than 2.3 trillion English, Chinese, and code tokens, using a SentencePiece byte-level BPE tokenizer and dynamic rotary position embeddings. The checkpoint declares 4,096 native positions, while the official model card says it supports 16k context through inference-time extrapolation without stating an exact maximum token count.
Benchmark scores
| Benchmark | Score |
|---|---|
| ARC AI2 | 81.7 |
| BBH (BIG-Bench Hard) | 52.5 |
| BoolQ | 87.5 |
| Capability | 103.0 |
| Epoch Capabilities Index (ECI) | 111 |
| HellaSwag (Unspecified Scoring Protocol) | 78.1 |
| LAMBADA | 71.8 |
| PIQA | 80.3 |