AURORA Multilingual τ³-bench (Turkish)
Agents · 2026-09-30
LILT's Turkish-locale run of τ³-bench customer-support agents on AURORA, with localized user-simulator prompts, task scenarios and databases, a gemini-3.5-flash user simulator, and reasoning disabled or at its lowest setting. The score is pass^1 over four trials, averaged with equal weight across the airline, retail and telecom base splits and a 20-task banking sample. Kept as a diagnostic: it is one locale of a modified τ³-bench protocol with no published cross-locale aggregate.
Top models (higher is better)
| Model | Score |
|---|---|
| Claude Opus 5.5 | 79.8 |
| GPT-6 Sol | 69.3 |
| Mistral Medium 3.5 | 60.1 |
| GPT-5.6 Sol | 58.5 |
| Gemini 3.8 Flash | 57.9 |