← All benchmarks

iPrefBench (task success)

Agents · 2026-10-26

iPrefBench, from Apple and The Ohio State University, has 130 underspecified user requests across 26 mock iOS apps in an executable simulator seeded around one persona, whose 3,051-event activity history the agent receives as context. Each omitted user-dependent detail is either inferable from that history or known only to a simulated user (Claude Sonnet 5) that the agent can query with an ask_user action. The agent acts from screenshots with standard GUI actions under a 50-step budget, and a Claude Sonnet 5 judge checks each task's action-completion and personalization rubric criteria against the trajectory. The score is the percentage of tasks whose criteria are all satisfied.

Top models (higher is better)