AgentVQA Web Agents
Multimodal · 2026-10-22
The 200 AgentVQA questions in the Web Agents domain, covering Android, iOS, desktop and web GUI steps: choosing the next UI action from a goal, the screen and earlier actions, or grounding an instruction to click coordinates. Questions are manually verified four-option multiple-choice items converted from 23 existing datasets, so offline answers stand in for online agent actions such as choosing the next GUI click or predicting a robot's next step. Models answer with a single letter from the visual input (images, screenshots or video) and any action history; chance is 25%. The score is the percentage of these questions answered correctly.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-5.6 Sol | 47.0 |
| Qwen3.7 Flash | 43.0 |
| Gemini 3.7 Flash | 40.5 |
| Sonnet 5 | 38.0 |
| Opus 4.8 | 35.5 |
| Gemini 3.1 Pro Preview | 34.5 |
| GPT-5.6 Luna | 34.5 |
| Qwen3.5 35B A3B | 34.5 |
| Qwen3.5-27B | 32.5 |
| Qwen3.7-Plus (source unspecified) | 32.5 |
| Qwen3.8 27B | 30.0 |
| Qwen3.5 122B A10B | 26.0 |
| GLM-4.6V | 25.5 |
| GLM-4.5V | 25.0 |
| Qwen3.5 397B A17B | 25.0 |
| Kimi K3 | 22.0 |
| GPT-4o (source unspecified) | 21.5 |
| GLM-4.1V-9B-Thinking | 19.0 |
| Llama 4 Maverick Instruct | 18.5 |