BALROG — Crafter (LLM)

Name: BALROG — Crafter (LLM)
Published: 2024-11-20

Games · 2024-11-20

Mean per-episode progress (0-100) on Crafter open-world survival: progress is the share of the game's 22 achievements unlocked, in BALROG's language-only track with a zero-shot agent. One of the six environments averaged into the BALROG leaderboard score; the episode count depends on the submission's protocol version.

Top models (higher is better)

Model	Score
GPT-6 Astra	76.8
Claude Opus 5	68.2
Gemini 3 Pro Preview	57.3
Grok 4	57.3
GPT-5.6 Sol	55.9
GPT-5.6 Luna	55.5
Gemini 2.5 Pro Experimental 03-25	55.0
GPT-5.6 Terra	55.0
Gemini 3.1 Pro Preview	55.0
Opus 4.5	49.5
Gemini 3 Flash Preview	45.0
Gemini 2.5 Flash	40.0
GPT-5	39.1
DeepSeek-R1	36.4
Reka Flash 3	33.6
GPT-4o (2024-05-13)	33.1
Claude 3.5 Sonnet (Oct 2024)	32.7
Llama 3.2 90B Vision Instruct	31.7
Llama 3.1 70B Instruct	31.2
Gemini 1.5 Pro 002	30.2
Llama 3.3 70B Instruct	28.6
Mistral NeMo Instruct 2407	27.7
Qwen2.5 72B Instruct	27.3
Haiku 3.5	26.4
Llama 3.2 11B Vision Instruct	26.2
Llama 3.1 8B Instruct	25.5
Grok 3 (API)	25.0
Haiku 4.5	24.1
Qwen2 VL 72B Instruct	22.7
Gemini 1.5 Flash 002	20.0
Llama 3.2 3B Instruct	17.3
Qwen2.5 7B Instruct	16.4
GPT-4o Mini	15.9
DeepSeek-R1-Distill-Qwen-32B	15.0
Phi-4	13.6
Llama 3.2 1B Instruct	12.7
Qwen2 VL 7B Instruct	6.4