← All benchmarks

OpenDS-World Medium Difficulty

Search · 2026-10-19

The 61 OpenDS-World tasks of medium human difficulty, where exactly one of the five human solvers assigned to the task answered correctly. Each task gives a question and an input video; the answer must be found on the live web through an evidence chain spanning text, images, audio and video in 42 resource formats and seven access types, such as following links, filling forms, downloading files and seeking within videos. Agents use a shared harness with text, image and video search, an interactive browser and video tools under a budget of 3,600 seconds and 100 tool calls, then answer without tools if the budget runs out; a Qwen3.8-27B judge checks semantic equivalence with the reference answer. The score is the percentage of these tasks answered correctly.

Top models (higher is better)