ORCA-bench Hard RCA (Fireworks 324-task subset)
Agents
ORCA-bench, from Traversal, Cornell Tech, and Columbia, tests on-call root-cause analysis on a live OpenTelemetry Astronomy Shop deployment, where an agent gets an ambiguous user report, Grafana access to six days of metrics, logs, and traces, and full source code. Fireworks runs a 324-task subset of the 1,079-task release three times per model with Terminus-2, and GPT-5.4 grades each plausible root cause. The headline is Hard RCA accuracy, the share of the 79 Hard-specificity incidents in which the agent names every plausible root cause, averaged over three runs.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-6 Astra | 50.2 |
| Claude Opus 5 | 44.3 |
| Grok 4.6 | 32.1 |
| Kimi K3 | 31.2 |
| GLM-5.3 | 30.0 |
| Gemini 3.8 Flash | 29.5 |
| GPT-5.6 Sol | 29.5 |
| DeepSeek V4 Pro 0813 | 24.5 |
| Ember-1 | 24.1 |
| DeepSeek V4.1 Flash 1017 | 20.7 |
| DeepSeek V4.1 Flash | 20.3 |
| Muse Spark Open | 3.4 |