BALROG — NetHack (LLM)

Name: BALROG — NetHack (LLM)
Published: 2024-11-20

Games · 2024-11-20

Mean per-episode progress (0-100) on the full NetHack Learning Environment, scored by BALROG's dungeon-depth and experience-level progression metric, in BALROG's language-only track with a zero-shot agent. One of the six environments averaged into the BALROG leaderboard score; the episode count depends on the submission's protocol version.

Top models (higher is better)

Model	Score
GPT-6 Astra	13.2
Claude Opus 5	7.3
Gemini 3 Pro Preview	6.8
Gemini 3 Flash Preview	4.0
GPT-5.6 Terra	3.3
GPT-5.6 Sol	3.2
Gemini 3.1 Pro Preview	3.0
Opus 4.5	2.4
Grok 4	1.8
GPT-5.6 Luna	1.7
Gemini 2.5 Pro Experimental 03-25	1.7
Grok 3 (API)	1.6
DeepSeek-R1	1.4
GPT-5	1.3
Haiku 3.5	1.2
Gemini 2.5 Flash	0.8
DeepSeek-R1-Distill-Qwen-32B	0.7
Reka Flash 3	0.6
Claude 3.5 Sonnet (Oct 2024)	0.6
Gemini 1.5 Pro 002	0.4
GPT-4o (2024-05-13)	0.4
Llama 3.3 70B Instruct	0.4
Llama 3.1 70B Instruct	0.3
Mistral NeMo Instruct 2407	0.3
Qwen2.5 72B Instruct	0.3
Haiku 4.5	0.0
Gemini 1.5 Flash 002	0.0
GPT-4o Mini	0.0
Llama 3.1 8B Instruct	0.0
Llama 3.2 11B Vision Instruct	0.0
Llama 3.2 1B Instruct	0.0
Llama 3.2 3B Instruct	0.0
Llama 3.2 90B Vision Instruct	0.0
Phi-4	0.0
Qwen2.5 7B Instruct	0.0
Qwen2 VL 72B Instruct	0.0
Qwen2 VL 7B Instruct	0.0