Software AI model leaderboards

Which AI model is best for software work? Compare 117 models on 16 Software benchmarks: FrontierSWE V2 (Proximal), Legacy-Bench (Factory), sudo L7 (Surge AI), Senior SWE-Bench (Snorkel AI), SWE-Clar (Upwork), SWE-Bench Pro V2 (Scale AI), APEX-SWE (Mercor), DeepSWE v1.1, ViBench 2.0 (Replit), DesignArena: Full-Stack…

Last updated Oct 28, 2026

FrontierSWE V2 by Proximal

Proximal’s FrontierSWE V2 tests 22 models on 34 expert-level software projects of up to 20 hours each.
GPT-6 Astra leads with 65.5%, 3.2 points ahead of Claude Opus 5.5, which costs 90% less per trial.
The best open model, GLM 5.3, is tenth at 30.2%, 35.3 points behind first.

Legacy-Bench by Factory

Factory’s Legacy-Bench tests 12 models on maintaining enterprise code in COBOL, Fortran, Java 7, and other legacy languages.
Claude Opus 5.5 and Muse Spark 1.3 tie for first at 62.0%, and Claude Opus 5.5 costs 75% less per task.
DeepSeek V4.1 Flash and DeepSeek V4.1 Flash (1017), both open models, rank third and fourth, ahead of every OpenAI, Google, and xAI model.

sudo L7 by Surge AI

Surge AI’s sudo L7 tests 16 models on software work.
Claude Sonnet 5.5 leads with 50.6%, 5.0 points ahead of Claude Opus 5.5.
The best open model, Qwen 3.8 Max, is tied for sixth at 16.7%, 33.9 points behind first.

Senior SWE-Bench by Snorkel AI

Snorkel AI’s Senior SWE-Bench tests seven models on software work.
GPT-6 Astra leads the ranked models at 36.8%; Claude Opus 5.5 (39.3%, reported by Anthropic) is shown but not ranked.
The best open model, Kimi K3, is sixth at 20.2%, 16.6 points behind first.

SWE-Clar by Upwork

Upwork’s SWE-Clar tests nine models on building greenfield software from real Upwork jobs after clarifying incomplete client requests.
GPT-6 Astra leads with 67.0%, 4.0 points ahead of Claude Opus 5.
The best open model, Qwen 3.8 Max, is fifth at 55.0%, 12.0 points behind first.

SWE-Bench Pro V2 by Scale AI

Scale AI’s SWE-Bench Pro V2 tests 16 models on resolving 642 real issues in professional open-source repositories.
The top three models are within 0.3 points of each other, led by Claude Opus 5 at 99.4%.
The best open model, Kimi K3, is fourth at 97.7%, 1.7 points behind first.

APEX-SWE by Mercor

Mercor’s APEX-SWE tests 11 models on repository-scale engineering tasks graded against a maintainer’s patch.
Claude Opus 5.5 leads with 67.6%, 3.8 points ahead of Claude Opus 5, which costs 79% less per task.
The best open model, DeepSeek V4.1 Flash, is fifth at 51.9%, 15.7 points behind first.

DeepSWE v1.1

DeepSWE v1.1 tests 16 models on debugging failures whose cause sits several layers from the symptom.
Claude Opus 5 leads with 72.3%, 1.3 points ahead of GPT-6 Astra, which costs 59% less per task.
Kimi K3 and DeepSeek V4.1 Flash (1017), both open models, rank third and fourth, ahead of every Google and xAI model.

ViBench 2.0 by Replit

Replit’s ViBench 2.0 tests nine models on software work.
GPT-6.1 Sol leads with 64.7%, 4.4 points ahead of GPT-6 Astra.
The best open model, DeepSeek V4.1 Flash, is tied for sixth at 0.0%, 64.7 points behind first.

DesignArena: Full-Stack by DesignArena

DesignArena: Full-Stack tests 15 models on software work.
GPT-6 Astra leads with 1,357 Elo, 31 Elo points ahead of Claude Fable 5.1.
The best open model, Kimi K3, is third at 1,315 Elo, 42 Elo points behind first.

DesignArena: Agentic Game Dev by DesignArena

DesignArena: Agentic Game Dev tests 15 models on software work.
Claude Fable 5 leads with 1,248 Elo, 16 Elo points ahead of GPT-5.6 Sol.
The best open model, Kimi K3, is fourth at 1,218 Elo, 30 Elo points behind first.

MacroscopeBench by Macroscope

MacroscopeBench tests 12 models on catching bugs in real code changes without false reports.
Claude Opus 5.5 leads with 83.9%, 3.6 points ahead of GPT-6 Sol, which costs 68% less per task.
The best open model, GLM 5.3, is sixth at 77.4%, 6.5 points behind first, for 67% less per task.

AI Code Review Benchmark by Qodo

Qodo’s AI Code Review Benchmark tests 11 models on software work.
Claude Opus 5.5 leads with 56.8%, 2.8 points ahead of Claude Sonnet 5.5.
GLM 5.3, DeepSeek V4 Pro 0813, and Kimi K3, all open models, rank fourth, fifth, and sixth, ahead of every OpenAI and Google model.

ORCA Benchmark by Traversal

Traversal’s ORCA Benchmark tests 12 models on finding the root cause of live production incidents.
GPT-6 Astra leads with 50.2%, 5.9 points ahead of Claude Opus 5.
The best open model, Kimi K3, is fourth at 31.2%, 19.0 points behind first.

SecIT Bench by Cribl

Cribl’s SecIT Bench tests 17 models on software work.
DeepSeek V4.1 Flash leads with 82.9%, 1.5 points ahead of GLM 5.3.
Open models take two of the top five places.

o11y-bench by Grafana Labs

Grafana Labs’ o11y-bench tests 18 models on querying metrics, logs, and traces and investigating incidents in a live Grafana stack.
The top three models are within 0.6 points of each other, led by GPT-6.1 Sol at 96.9%.
The best open model, Kimi K3, is third at 96.3%, 0.6 points behind first.

How every benchmark is run and scored · Data: JSON · Markdown · llms.txt

Benchmarks

View full data table

{{ ind.chartBenchName }}

Main Insights

Domain Leaderboard

Rank by

Scored on

{{ r.displayModel }}*

Fireworks methodology

This methodology is versioned, published alongside the index, and inherited by every benchmark run unless a benchmark or model row records an exception.

1. Scope

The Specialized Intelligence Index measures how models perform on domain benchmarks under one standardized serving configuration and one harness per benchmark. It does not measure the maximum achievable performance on a benchmark, which a team optimizing one model with a purpose-built harness will typically exceed.

The index does not measure general capability and is not a substitute for a generalist index. Benchmarks are contributed by domain partners that use them in production or drawn from recognized public sources. Fireworks runs open, closed, and custom models against benchmarks built by their owners; it does not design the benchmarks.

2. Definitions

Model — a set of weights with a published name and version. A specialized model is adapted for a domain through fine-tuning; its base model is recorded so the effect of adaptation is directly interpretable.
Adapter — trained weights applied over a base model at serving time, most commonly LoRA. Most specialized models on the index are adapters.
Serving provider — the company hosting an endpoint. Open-weight models run on Fireworks; closed models run with their developers. “Open weights” means publicly released weights runnable on controlled infrastructure, not necessarily open-source licensing.
Endpoint — a hosted model instance reachable through an API. The exact endpoint string is recorded because providers can differ in quantization, serving stack, context limits, and price.
Harness — prompt construction, tools, turn management, context assembly, stop conditions, and response parsing. The orchestrator packages benchmarks and executes the harness across models and tasks.
Task — one benchmark problem. A run is one attempt at one task; the program standard is three runs per task.
Condition — a deliberately varied setting, such as reasoning effort, that produces a separate row instead of being averaged away.
Benchmark score — the benchmark’s own named headline metric, reported in its native unit and named explicitly where additional context is needed.
Duration — wall-clock time for one complete task, measured end to end. It includes model generation, tool calls, waiting on tool outputs, and subsequent turns. It excludes judge requests and other evaluation or setup steps. It is not time to first token. Columns are headed Duration / task, or Duration / trial on a board whose source times one trial of a repeated task.
pass@1 — the mean success rate across runs. A task solved on two of three runs contributes 0.67. pass^k requires success on all k runs and is a separate reliability measure. pass@k, success on any of k runs, is not used.
Confidence interval — always 95%, and always printed beside the score it qualifies as a single ± margin rather than in a column of its own. What it resamples depends on what the board retained: a task-level bootstrap where per-task results are held, or run-to-run variation across complete runs where only run headlines are. Each board’s footer names which. No interval covers judge variance, and a board that measured no uncertainty prints none rather than a placeholder.
Executed by identifies who ran the benchmark. Reproducibility tier identifies who can rerun it.
Exception — any departure from the program standard. An exception means the row is not comparable to its neighbors on equal footing.
Index release — a dated board snapshot. Scores are comparable within a release, but not necessarily across releases.
Sampling profile — a named, versioned generation-parameter bundle. Rows record the profile and any deviations.

3. Benchmark admission

Every benchmark is a production evaluation already used by a domain partner or a recognized open benchmark.
Partners deliver tasks and data in Harbor-compatible format; they do not deliver harnesses.
Task diversity and difficulty are reviewed before admission, and the benchmark owner must attest to contamination controls.
A benchmark is retired or moved into a deepening phase when top models cluster within a stated margin and it stops discriminating.

4. Model roster

Open-weight models are selected from top performers on our internal and external evaluations. Closed models run alongside them through first-party APIs, and partner fine-tunes run on their own domain benchmarks.

Inclusion follows a cost-relevance rule based on what an enterprise would plausibly run in production. Materially out-of-range models are excluded explicitly.
New releases enter a qualification pipeline and appear only after clearing it.

FIREWORKS IMPLEMENTATION

5. Scoring standard

Scores on benchmarks that only the provider can rerun are provided to us directly by the provider. On all other benchmarks, we generally collect three runs and report the average (avg@3), along with a 95% confidence interval; benchmark-native aggregations such as pass@3 are named on the card. Where a benchmark’s native metric is not pass/fail, its card names the metric. Metrics are never averaged across benchmarks.

Score carries its 95% margin beside it as ±x.x, with the full interval in the cell's tooltip. Cost and duration name the denominator they were measured against: Cost / task and Duration / task by default, and / trial where the source measured one trial of a repeated task. Both state their exclusions in the table footer. Duration covers the full end-to-end execution time, excluding judge time.

6. Sampling standard

Default profile SII-2026.09:

TEMPERATURE

1.0

TOP_P

0.95 where supported

TOP_K

unset by default

MAX_TOKENS

131,072

REASONING_EFFORT

max

REASONING_HISTORY

preserved

SEED

unset

top_p applies nucleus sampling where supported; top_k is left unset by default. max_tokens is a per-call cap and must leave enough room for reasoning tokens.
The seed is unset so temperature-1 sampling produces the run variance measured by reliability and confidence intervals.
Provider recommendations override defaults where published. Every deviation is recorded on the model row.

7. Serving standard

Open-weight models run on Fireworks serverless endpoints.
Closed models run against their own first-party APIs.
Partner fine-tunes are LoRA adapters on base models and run on Fireworks dedicated endpoints.

8. Harness standard

Fireworks-executed evaluations use open-source harnesses pinned to versioned snapshots. The same harness, orchestrator, tools, and tool definitions apply to every model on a benchmark.

9. Execution standard

We use each benchmark’s designed timeout by default. If that limit systematically cuts off unfinished work and would distort the comparison, we raise it until timeouts are rare, not absent. Any raised limit is published in the benchmark methodology.
Network access is open where required. Public benchmark repositories and known mirrors are blocked during evaluation to prevent retrieval of tasks or grading artifacts.
Concurrency changes throughput only; it does not change prompts, tools, time limits, or scoring.
Every model must complete the same number of tasks. Faster endpoints do not win by finishing more work.

10. Reproducibility

Reproducibility identifies who can rerun a score, not whether a rerun will produce an identical result.

Public — anyone. The data is public, Fireworks ran a pinned open-source harness, and conditions are disclosed.
Fireworks — Fireworks and the benchmark owner. The data is private, but Fireworks ran a pinned open-source harness and can rerun its recorded snapshot.
Partner — the benchmark owner. The owner supplied a score from their own evaluation. Fireworks has not yet rerun it, but plans to do so.

11. Cost and duration standard

Where available, token counts use provider-reported values. Input is split into cached and uncached tokens; output tokens are provided.
Dollar cost is derived with a rate-card source and price date because prices may change after a run.
Duration is wall-clock time for the full end-to-end task, excluding judge time. It is not time to first token.
Duration keeps its source's denominator: per task by default, and per trial where the source times one trial of a repeated task. Independent trials are never summed, because a duration is not a total.
Each board’s methodology names whether the published figure is a mean or a median, and states any remaining exclusions in the table footer.
When timing is invalid for comparison — for example shared multi-tenant concurrency — duration is withheld rather than published as a misleading number.

12. How ranks are shown

The selected “Rank by” metric first orders eligible models within each benchmark. Higher score ranks first; lower cost or duration ranks first. Exact benchmark-score ties receive the average of their occupied positions.

A model’s domain rank is the arithmetic mean of its available benchmark ranks. Every contributing benchmark receives equal weight; missing results are not imputed or counted as zero.
A model is listed only when it has scores on at least two of the domain’s benchmarks, so one strong result cannot lead a domain. A domain with a single benchmark lists every model on it, and Offensive and Customer Support list every model on either of their benchmarks, with an asterisk on each model scored on only one.
If average ranks tie, a normalized score tie-break min–max normalizes each result within its benchmark, reverses lower-is-better metrics, and averages those normalized values. Higher average normalized performance wins, so the larger score differential decides the order.
If both the average rank and normalized tie-break score are equal, models share the displayed position and the following position is skipped.
We don't aggregate metrics into one overall score, as this would apply subjective weighting.
Per-domain leaders are shown; cross-domain leaders are not.

13. Privacy and publication

Benchmark visibility is the owner’s decision. Some partners publish their benchmarks; others keep them private. For private benchmarks, scores are published, while evaluation scripts, prompts, grading logic, and trajectories are not.

Software AI model leaderboards

FrontierSWE V2 by Proximal

Legacy-Bench by Factory

sudo L7 by Surge AI

Senior SWE-Bench by Snorkel AI

SWE-Clar by Upwork

SWE-Bench Pro V2 by Scale AI

APEX-SWE by Mercor

DeepSWE v1.1

ViBench 2.0 by Replit

DesignArena: Full-Stack by DesignArena

DesignArena: Agentic Game Dev by DesignArena

MacroscopeBench by Macroscope

AI Code Review Benchmark by Qodo

ORCA Benchmark by Traversal

SecIT Bench by Cribl

o11y-bench by Grafana Labs

How models handle real work,
on benchmarks written by practitioners

What {{ keyInsightBoards }} benchmarks show

{{ ki.claim }}

Benchmarks

{{ category.name }}

Methodology

Blogs

Want to be on the Index?

Methodology

Fireworks methodology

Benchmark methodologies

{{ group.name }}

Paper, repository, and sources

{{ part.title }}

{{ bench.name }}

Models

A rank tells you where you stand. Training moves it.