ORCA Benchmark

Root-cause analysis on live incidents, narrowing traces, logs, and metrics to the failing change.

Last updated

Main Insights

Results

ORCA Benchmark results, ranked by score (hard rca, avg@3), highest first. Last updated .
RankModelProviderScore (Hard RCA, avg@3)Cost (USD / task)Duration (seconds / task)
1GPT-6 AstraOpenAI50.2%$4.001,144
2Claude Opus 5Anthropic44.3%$5.332,598
3Kimi K3.1 ProMoonshot AI32.9%$0.71730
4Grok 4.6xAI32.1%$2.411,637
5Kimi K3Moonshot AI31.2%$1.241,345
6GLM-5.3Z.ai30.0%$2.312,419
7Gemini 3.8 FlashGoogle29.5%$1.401,140
7GPT-5.6 SolOpenAI29.5%$1.971,436
9DeepSeek V4 Pro 0813DeepSeek24.5%$0.561,128
10Ember-1Fireworks24.1%$0.92895
11DeepSeek V4.1 Flash (1017)DeepSeek20.7%$0.24720
12DeepSeek V4.1 FlashDeepSeek20.3%$0.17915
13Muse Spark OpenMeta3.4%$1.101,431