RegretBench (goal success)
General QA · 2026-07-24
RegretBench treats clarifying questions as a sequential policy: the assistant faces an ambiguous request with a hidden user intent (6,287 open-domain QA items from AmbigDocs and CondAmbigQA-2K), may ask up to five questions of a simulated cooperative user (GPT-5 Nano), and then answers. Goal success is the share of rollouts, three per item, whose final selected intent matches the hidden one, in percent.
Top models (higher is better)
| Model | Score |
|---|---|
| Gemini 3.1 Pro Preview | 83.9 |
| Grok 4.1 Fast | 79.9 |
| Llama 4 Scout Instruct | 79.5 |
| GPT-4.1 | 79.3 |
| Llama 4 Maverick Instruct | 77.5 |
| GPT-5 Mini | 77.4 |
| DeepSeek-V4-Flash | 75.4 |
| Gemini 3.1 Flash-Lite | 75.0 |
| K2 Think V2 | 74.5 |
| GPT-5.4 Nano | 71.7 |
| Qwen3.5-Flash (source unspecified) | 71.1 |
| Nemotron 3 Super 120B A12B | 70.7 |
| K2-V2-Instruct | 39.1 |