BALROG — BabaIsAI (LLM)

Name: BALROG — BabaIsAI (LLM)
Published: 2024-11-20

Games · 2024-11-20

Mean per-episode progress (0-100) on Baba Is AI rule-manipulation puzzles adapted from Baba Is You, where agents push word blocks to rewrite the level's rules, in BALROG's language-only track with a zero-shot agent. One of the six environments averaged into the BALROG leaderboard score; the episode count depends on the submission's protocol version.

Top models (higher is better)

Model	Score
GPT-6 Astra	100.0
GPT-5.6 Terra	97.5
GPT-5.6 Sol	96.7
Claude Opus 5	95.8
Gemini 3.1 Pro Preview	90.0
Gemini 3 Pro Preview	88.3
GPT-5.6 Luna	77.5
Gemini 3 Flash Preview	73.3
Gemini 2.5 Pro Experimental 03-25	56.7
Opus 4.5	50.8
DeepSeek-R1	50.8
Gemini 2.5 Flash	50.0
Grok 4	45.8
Llama 3.2 90B Vision Instruct	43.9
Reka Flash 3	43.3
DeepSeek-R1-Distill-Qwen-32B	40.0
Llama 3.1 70B Instruct	40.0
Claude 3.5 Sonnet (Oct 2024)	37.5
GPT-4o (2024-05-13)	33.7
Haiku 4.5	33.3
Grok 3 (API)	33.3
Gemini 1.5 Pro 002	32.0
Llama 3.3 70B Instruct	29.2
GPT-5	25.8
Mistral NeMo Instruct 2407	20.8
Qwen2.5 72B Instruct	19.3
Llama 3.1 8B Instruct	18.3
Llama 3.2 3B Instruct	17.5
Phi-4	16.7
GPT-4o Mini	15.6
Llama 3.2 11B Vision Instruct	15.6
Gemini 1.5 Flash 002	12.8
Qwen2.5 7B Instruct	12.5
Llama 3.2 1B Instruct	10.8
Qwen2 VL 72B Instruct	10.8
Haiku 3.5	8.3
Qwen2 VL 7B Instruct	7.6