ORCA Benchmark

Name: ORCA Benchmark leaderboard
Creator: Traversal

Root-cause analysis on live incidents, narrowing traces, logs, and metrics to the failing change.

Last updated Oct 27, 2026

Main Insights

Traversal’s ORCA Benchmark tests 12 models on finding the root cause of live production incidents.
GPT-6 Astra leads with 50.2%, 5.9 points ahead of Claude Opus 5.
The best open model, Kimi K3, is fourth at 31.2%, 19.0 points behind first.

Results

ORCA Benchmark results, ranked by score (hard rca, avg@3), highest first. Last updated Oct 27, 2026.
Rank	Model	Provider	Score (Hard RCA, avg@3)	Cost (USD / task)	Duration (seconds / task)
1	GPT-6 Astra	OpenAI	50.2%	$4.00	1,144
2	Claude Opus 5	Anthropic	44.3%	$5.33	2,598
3	Grok 4.6	xAI	32.1%	$2.41	1,637
4	Kimi K3	Moonshot AI	31.2%	$1.24	1,345
5	GLM-5.3	Z.ai	30.0%	$2.31	2,419
6	Gemini 3.8 Flash	Google	29.5%	$1.40	1,140
6	GPT-5.6 Sol	OpenAI	29.5%	$1.97	1,436
8	DeepSeek V4 Pro 0813	DeepSeek	24.5%	$0.56	1,128
9	Ember-1	Fireworks	24.1%	$0.92	895
10	DeepSeek V4.1 Flash (1017)	DeepSeek	20.7%	$0.24	720
11	DeepSeek V4.1 Flash	DeepSeek	20.3%	$0.17	915
12	Muse Spark Open	Meta	3.4%	$1.10	1,431

How ORCA Benchmark is scored · Traversal source

All Specialized Intelligence Index leaderboards

Benchmarks

View full data table

{{ ind.chartBenchName }}

Main Insights

Domain Leaderboard

Rank by

Scored on

{{ r.displayModel }}*

Fireworks methodology

This methodology is versioned, published alongside the index, and inherited by every benchmark run unless a benchmark or model row records an exception.

1. Scope

The Specialized Intelligence Index measures how models perform on domain benchmarks under one standardized serving configuration and one harness per benchmark. It does not measure the maximum achievable performance on a benchmark, which a team optimizing one model with a purpose-built harness will typically exceed.

The index does not measure general capability and is not a substitute for a generalist index. Benchmarks are contributed by domain partners that use them in production or drawn from recognized public sources. Fireworks runs open, closed, and custom models against benchmarks built by their owners; it does not design the benchmarks.

2. Definitions

Model — a set of weights with a published name and version. A specialized model is adapted for a domain through fine-tuning; its base model is recorded so the effect of adaptation is directly interpretable.
Adapter — trained weights applied over a base model at serving time, most commonly LoRA. Most specialized models on the index are adapters.
Serving provider — the company hosting an endpoint. Open-weight models run on Fireworks; closed models run with their developers. “Open weights” means publicly released weights runnable on controlled infrastructure, not necessarily open-source licensing.
Endpoint — a hosted model instance reachable through an API. The exact endpoint string is recorded because providers can differ in quantization, serving stack, context limits, and price.
Harness — prompt construction, tools, turn management, context assembly, stop conditions, and response parsing. The orchestrator packages benchmarks and executes the harness across models and tasks.
Task — one benchmark problem. A run is one attempt at one task; the program standard is three runs per task.
Condition — a deliberately varied setting, such as reasoning effort, that produces a separate row instead of being averaged away.
Benchmark score — the benchmark’s own named headline metric, reported in its native unit and named explicitly where additional context is needed.
Duration — wall-clock time for one complete task, measured end to end. It includes model generation, tool calls, waiting on tool outputs, and subsequent turns. It excludes judge requests and other evaluation or setup steps. It is not time to first token. Columns are headed Duration / task, or Duration / trial on a board whose source times one trial of a repeated task.
pass@1 — the mean success rate across runs. A task solved on two of three runs contributes 0.67. pass^k requires success on all k runs and is a separate reliability measure. pass@k, success on any of k runs, is not used.
Confidence interval — always 95%, and always printed beside the score it qualifies as a single ± margin rather than in a column of its own. What it resamples depends on what the board retained: a task-level bootstrap where per-task results are held, or run-to-run variation across complete runs where only run headlines are. Each board’s footer names which. No interval covers judge variance, and a board that measured no uncertainty prints none rather than a placeholder.
Executed by identifies who ran the benchmark. Reproducibility tier identifies who can rerun it.
Exception — any departure from the program standard. An exception means the row is not comparable to its neighbors on equal footing.
Index release — a dated board snapshot. Scores are comparable within a release, but not necessarily across releases.
Sampling profile — a named, versioned generation-parameter bundle. Rows record the profile and any deviations.

3. Benchmark admission

Every benchmark is a production evaluation already used by a domain partner or a recognized open benchmark.
Partners deliver tasks and data in Harbor-compatible format; they do not deliver harnesses.
Task diversity and difficulty are reviewed before admission, and the benchmark owner must attest to contamination controls.
A benchmark is retired or moved into a deepening phase when top models cluster within a stated margin and it stops discriminating.

4. Model roster

Open-weight models are selected from top performers on our internal and external evaluations. Closed models run alongside them through first-party APIs, and partner fine-tunes run on their own domain benchmarks.

Inclusion follows a cost-relevance rule based on what an enterprise would plausibly run in production. Materially out-of-range models are excluded explicitly.
New releases enter a qualification pipeline and appear only after clearing it.

FIREWORKS IMPLEMENTATION

5. Scoring standard

Scores on benchmarks that only the provider can rerun are provided to us directly by the provider. On all other benchmarks, we generally collect three runs and report the average (avg@3), along with a 95% confidence interval; benchmark-native aggregations such as pass@3 are named on the card. Where a benchmark’s native metric is not pass/fail, its card names the metric. Metrics are never averaged across benchmarks.

Score carries its 95% margin beside it as ±x.x, with the full interval in the cell's tooltip. Cost and duration name the denominator they were measured against: Cost / task and Duration / task by default, and / trial where the source measured one trial of a repeated task. Both state their exclusions in the table footer. Duration covers the full end-to-end execution time, excluding judge time.

6. Sampling standard

Default profile SII-2026.09:

TEMPERATURE

1.0

TOP_P

0.95 where supported

TOP_K

unset by default

MAX_TOKENS

131,072

REASONING_EFFORT

max

REASONING_HISTORY

preserved

SEED

unset

top_p applies nucleus sampling where supported; top_k is left unset by default. max_tokens is a per-call cap and must leave enough room for reasoning tokens.
The seed is unset so temperature-1 sampling produces the run variance measured by reliability and confidence intervals.
Provider recommendations override defaults where published. Every deviation is recorded on the model row.

7. Serving standard

Open-weight models run on Fireworks serverless endpoints.
Closed models run against their own first-party APIs.
Partner fine-tunes are LoRA adapters on base models and run on Fireworks dedicated endpoints.

8. Harness standard

Fireworks-executed evaluations use open-source harnesses pinned to versioned snapshots. The same harness, orchestrator, tools, and tool definitions apply to every model on a benchmark.

9. Execution standard

We use each benchmark’s designed timeout by default. If that limit systematically cuts off unfinished work and would distort the comparison, we raise it until timeouts are rare, not absent. Any raised limit is published in the benchmark methodology.
Network access is open where required. Public benchmark repositories and known mirrors are blocked during evaluation to prevent retrieval of tasks or grading artifacts.
Concurrency changes throughput only; it does not change prompts, tools, time limits, or scoring.
Every model must complete the same number of tasks. Faster endpoints do not win by finishing more work.

10. Reproducibility

Reproducibility identifies who can rerun a score, not whether a rerun will produce an identical result.

Public — anyone. The data is public, Fireworks ran a pinned open-source harness, and conditions are disclosed.
Fireworks — Fireworks and the benchmark owner. The data is private, but Fireworks ran a pinned open-source harness and can rerun its recorded snapshot.
Partner — the benchmark owner. The owner supplied a score from their own evaluation. Fireworks has not yet rerun it, but plans to do so.

11. Cost and duration standard

Where available, token counts use provider-reported values. Input is split into cached and uncached tokens; output tokens are provided.
Dollar cost is derived with a rate-card source and price date because prices may change after a run.
Duration is wall-clock time for the full end-to-end task, excluding judge time. It is not time to first token.
Duration keeps its source's denominator: per task by default, and per trial where the source times one trial of a repeated task. Independent trials are never summed, because a duration is not a total.
Each board’s methodology names whether the published figure is a mean or a median, and states any remaining exclusions in the table footer.
When timing is invalid for comparison — for example shared multi-tenant concurrency — duration is withheld rather than published as a misleading number.

12. How ranks are shown

The selected “Rank by” metric first orders eligible models within each benchmark. Higher score ranks first; lower cost or duration ranks first. Exact benchmark-score ties receive the average of their occupied positions.

A model’s domain rank is the arithmetic mean of its available benchmark ranks. Every contributing benchmark receives equal weight; missing results are not imputed or counted as zero.
A model is listed only when it has scores on at least two of the domain’s benchmarks, so one strong result cannot lead a domain. A domain with a single benchmark lists every model on it, and Offensive and Customer Support list every model on either of their benchmarks, with an asterisk on each model scored on only one.
If average ranks tie, a normalized score tie-break min–max normalizes each result within its benchmark, reverses lower-is-better metrics, and averages those normalized values. Higher average normalized performance wins, so the larger score differential decides the order.
If both the average rank and normalized tie-break score are equal, models share the displayed position and the following position is skipped.
We don't aggregate metrics into one overall score, as this would apply subjective weighting.
Per-domain leaders are shown; cross-domain leaders are not.

13. Privacy and publication

Benchmark visibility is the owner’s decision. Some partners publish their benchmarks; others keep them private. For private benchmarks, scores are published, while evaluation scripts, prompts, grading logic, and trajectories are not.

ORCA Benchmark

Main Insights

Results

How models handle real work,
on benchmarks written by practitioners

What {{ keyInsightBoards }} benchmarks show

{{ ki.claim }}

Benchmarks

{{ category.name }}

Methodology

Blogs

Want to be on the Index?

Methodology

Fireworks methodology

Benchmark methodologies

{{ group.name }}

Paper, repository, and sources

{{ part.title }}

{{ bench.name }}

Models

A rank tells you where you stand. Training moves it.