MacroscopeBench

Name: MacroscopeBench leaderboard
Creator: Macroscope

MacroscopeBench evaluates a model’s ability to identify bugs in real code changes while avoiding incorrect bug reports. Each task reviews a commit from a public repository; the evaluation includes commits containing known, subsequently fixed bugs and control commits with no known bugs recorded in the dataset. LLM judges determine whether the model identified the known bugs and assess the validity of all reported findings. The performance of a model is scored based on the harmonic mean of the known bug recall and overall bug detection precision as evaluated by the LLM judge.

Last updated Oct 20, 2026

Main Insights

MacroscopeBench tests 12 models on catching bugs in real code changes without false reports.
Claude Opus 5.5 leads with 83.9%, 3.6 points ahead of GPT-6 Sol, which costs 68% less per task.
The best open model, GLM 5.3, is sixth at 77.4%, 6.5 points behind first, for 67% less per task.

Results

MacroscopeBench results, ranked by score (avg@3), highest first. Last updated Oct 20, 2026.
Rank	Model	Provider	Score (avg@3)	Cost (USD / task)	Duration (seconds / task)
1	Claude Opus 5.5 · max	Anthropic	83.9%	$11.58	608
2	GPT-6 Sol · max	OpenAI	80.3%	$3.68	297
3	Claude Opus 5.5 · xhigh	Anthropic	78.9%	$3.51	172
4	GPT-6.1 Sol · max	OpenAI	78.5%	$1.05	239
5	GPT-6 Astra · max	OpenAI	78.0%	$7.87	229
6	GPT-5.6 Sol · high	OpenAI	77.8%	$3.90	180
7	GLM 5.3 · max	Z.ai	77.4%	$3.86	750
8	GPT-6.1 Sol · xhigh	OpenAI	77.2%	$0.79	163
9	GPT-6 Sol · xhigh	OpenAI	77.2%	$2.00	186
10	GPT-6.1 Sol · high	OpenAI	76.7%	$0.56	112
11	Grok 4.6 · xhigh	xAI	75.9%	$4.68	881
12	Grok 4.6 · high	xAI	75.1%	$3.51	930
13	GPT-6 Sol · high	OpenAI	74.3%	$1.23	123
14	GPT-6.1 Sol · medium	OpenAI	73.5%	$0.28	66
15	Claude Opus 5.5 · high	Anthropic	73.1%	$1.30	63
16	Kimi K3 · max	Moonshot AI	72.9%	$2.82	493
17	Grok 4.6 · medium	xAI	72.8%	$2.56	519
18	GPT-6 Luna · max	OpenAI	72.5%	$0.13	241
19	DeepSeek V4.1 Flash · max	DeepSeek	72.0%	$0.75	977
20	Gemini 3.8 Flash · medium	Google	70.7%	$3.38	663
21	DeepSeek V4.1 Flash · low	DeepSeek	70.7%	$0.38	489
22	GPT-6.1 Sol · low	OpenAI	70.6%	$0.15	46
23	DeepSeek V4.1 Flash · high	DeepSeek	70.5%	$0.38	460
24	Claude Opus 5.5 · medium	Anthropic	70.1%	$0.97	44
25	Claude Opus 5 · high	Anthropic	69.9%	$6.50	258
26	GLM 5.3 · high	Z.ai	69.4%	$2.02	329
27	GPT-6 Sol · medium	OpenAI	69.2%	$0.69	91
28	Kimi K3 · high	Moonshot AI	68.6%	$1.90	261
29	GPT-6 Luna · xhigh	OpenAI	68.2%	$0.063	122
30	GPT-5.6 Sol · low	OpenAI	66.1%	$0.81	64
31	GPT-6 Luna · high	OpenAI	65.6%	$0.047	100
32	Claude Opus 5 · medium	Anthropic	65.5%	$3.50	132
33	GPT-6 Luna · medium	OpenAI	62.4%	$0.033	74
34	GPT-6 Sol · low	OpenAI	61.8%	$0.36	58
35	Claude Opus 5.5 · low	Anthropic	61.4%	$0.68	29
36	Claude Opus 5 · low	Anthropic	59.4%	$1.40	56
37	Grok 4.6 · low	xAI	57.1%	$0.30	55
38	GLM 5.3 · low	Z.ai	54.6%	$0.41	82
39	Kimi K3 · low	Moonshot AI	53.5%	$0.48	63
40	Gemini 3.8 Flash · low	Google	51.3%	$0.90	158
41	GPT-6 Luna · low	OpenAI	44.7%	$0.019	42

How MacroscopeBench is scored · Macroscope source

All Specialized Intelligence Index leaderboards

Benchmarks

View full data table

{{ ind.chartBenchName }}

Main Insights

Domain Leaderboard

Rank by

Scored on

{{ r.displayModel }}*

Fireworks methodology

This methodology is versioned, published alongside the index, and inherited by every benchmark run unless a benchmark or model row records an exception.

1. Scope

The Specialized Intelligence Index measures how models perform on domain benchmarks under one standardized serving configuration and one harness per benchmark. It does not measure the maximum achievable performance on a benchmark, which a team optimizing one model with a purpose-built harness will typically exceed.

The index does not measure general capability and is not a substitute for a generalist index. Benchmarks are contributed by domain partners that use them in production or drawn from recognized public sources. Fireworks runs open, closed, and custom models against benchmarks built by their owners; it does not design the benchmarks.

2. Definitions

Model — a set of weights with a published name and version. A specialized model is adapted for a domain through fine-tuning; its base model is recorded so the effect of adaptation is directly interpretable.
Adapter — trained weights applied over a base model at serving time, most commonly LoRA. Most specialized models on the index are adapters.
Serving provider — the company hosting an endpoint. Open-weight models run on Fireworks; closed models run with their developers. “Open weights” means publicly released weights runnable on controlled infrastructure, not necessarily open-source licensing.
Endpoint — a hosted model instance reachable through an API. The exact endpoint string is recorded because providers can differ in quantization, serving stack, context limits, and price.
Harness — prompt construction, tools, turn management, context assembly, stop conditions, and response parsing. The orchestrator packages benchmarks and executes the harness across models and tasks.
Task — one benchmark problem. A run is one attempt at one task; the program standard is three runs per task.
Condition — a deliberately varied setting, such as reasoning effort, that produces a separate row instead of being averaged away.
Benchmark score — the benchmark’s own named headline metric, reported in its native unit and named explicitly where additional context is needed.
Duration — wall-clock time for one complete task, measured end to end. It includes model generation, tool calls, waiting on tool outputs, and subsequent turns. It excludes judge requests and other evaluation or setup steps. It is not time to first token. Columns are headed Duration / task, or Duration / trial on a board whose source times one trial of a repeated task.
pass@1 — the mean success rate across runs. A task solved on two of three runs contributes 0.67. pass^k requires success on all k runs and is a separate reliability measure. pass@k, success on any of k runs, is not used.
Confidence interval — always 95%, and always printed beside the score it qualifies as a single ± margin rather than in a column of its own. What it resamples depends on what the board retained: a task-level bootstrap where per-task results are held, or run-to-run variation across complete runs where only run headlines are. Each board’s footer names which. No interval covers judge variance, and a board that measured no uncertainty prints none rather than a placeholder.
Executed by identifies who ran the benchmark. Reproducibility tier identifies who can rerun it.
Exception — any departure from the program standard. An exception means the row is not comparable to its neighbors on equal footing.
Index release — a dated board snapshot. Scores are comparable within a release, but not necessarily across releases.
Sampling profile — a named, versioned generation-parameter bundle. Rows record the profile and any deviations.

3. Benchmark admission

Every benchmark is a production evaluation already used by a domain partner or a recognized open benchmark.
Partners deliver tasks and data in Harbor-compatible format; they do not deliver harnesses.
Task diversity and difficulty are reviewed before admission, and the benchmark owner must attest to contamination controls.
A benchmark is retired or moved into a deepening phase when top models cluster within a stated margin and it stops discriminating.

4. Model roster

Open-weight models are selected from top performers on our internal and external evaluations. Closed models run alongside them through first-party APIs, and partner fine-tunes run on their own domain benchmarks.

Inclusion follows a cost-relevance rule based on what an enterprise would plausibly run in production. Materially out-of-range models are excluded explicitly.
New releases enter a qualification pipeline and appear only after clearing it.

FIREWORKS IMPLEMENTATION

5. Scoring standard

Scores on benchmarks that only the provider can rerun are provided to us directly by the provider. On all other benchmarks, we generally collect three runs and report the average (avg@3), along with a 95% confidence interval; benchmark-native aggregations such as pass@3 are named on the card. Where a benchmark’s native metric is not pass/fail, its card names the metric. Metrics are never averaged across benchmarks.

Score carries its 95% margin beside it as ±x.x, with the full interval in the cell's tooltip. Cost and duration name the denominator they were measured against: Cost / task and Duration / task by default, and / trial where the source measured one trial of a repeated task. Both state their exclusions in the table footer. Duration covers the full end-to-end execution time, excluding judge time.

6. Sampling standard

Default profile SII-2026.09:

TEMPERATURE

1.0

TOP_P

0.95 where supported

TOP_K

unset by default

MAX_TOKENS

131,072

REASONING_EFFORT

max

REASONING_HISTORY

preserved

SEED

unset

top_p applies nucleus sampling where supported; top_k is left unset by default. max_tokens is a per-call cap and must leave enough room for reasoning tokens.
The seed is unset so temperature-1 sampling produces the run variance measured by reliability and confidence intervals.
Provider recommendations override defaults where published. Every deviation is recorded on the model row.

7. Serving standard

Open-weight models run on Fireworks serverless endpoints.
Closed models run against their own first-party APIs.
Partner fine-tunes are LoRA adapters on base models and run on Fireworks dedicated endpoints.

8. Harness standard

Fireworks-executed evaluations use open-source harnesses pinned to versioned snapshots. The same harness, orchestrator, tools, and tool definitions apply to every model on a benchmark.

9. Execution standard

We use each benchmark’s designed timeout by default. If that limit systematically cuts off unfinished work and would distort the comparison, we raise it until timeouts are rare, not absent. Any raised limit is published in the benchmark methodology.
Network access is open where required. Public benchmark repositories and known mirrors are blocked during evaluation to prevent retrieval of tasks or grading artifacts.
Concurrency changes throughput only; it does not change prompts, tools, time limits, or scoring.
Every model must complete the same number of tasks. Faster endpoints do not win by finishing more work.

10. Reproducibility

Reproducibility identifies who can rerun a score, not whether a rerun will produce an identical result.

Public — anyone. The data is public, Fireworks ran a pinned open-source harness, and conditions are disclosed.
Fireworks — Fireworks and the benchmark owner. The data is private, but Fireworks ran a pinned open-source harness and can rerun its recorded snapshot.
Partner — the benchmark owner. The owner supplied a score from their own evaluation. Fireworks has not yet rerun it, but plans to do so.

11. Cost and duration standard

Where available, token counts use provider-reported values. Input is split into cached and uncached tokens; output tokens are provided.
Dollar cost is derived with a rate-card source and price date because prices may change after a run.
Duration is wall-clock time for the full end-to-end task, excluding judge time. It is not time to first token.
Duration keeps its source's denominator: per task by default, and per trial where the source times one trial of a repeated task. Independent trials are never summed, because a duration is not a total.
Each board’s methodology names whether the published figure is a mean or a median, and states any remaining exclusions in the table footer.
When timing is invalid for comparison — for example shared multi-tenant concurrency — duration is withheld rather than published as a misleading number.

12. How ranks are shown

The selected “Rank by” metric first orders eligible models within each benchmark. Higher score ranks first; lower cost or duration ranks first. Exact benchmark-score ties receive the average of their occupied positions.

A model’s domain rank is the arithmetic mean of its available benchmark ranks. Every contributing benchmark receives equal weight; missing results are not imputed or counted as zero.
A model is listed only when it has scores on at least two of the domain’s benchmarks, so one strong result cannot lead a domain. A domain with a single benchmark lists every model on it, and Offensive and Customer Support list every model on either of their benchmarks, with an asterisk on each model scored on only one.
If average ranks tie, a normalized score tie-break min–max normalizes each result within its benchmark, reverses lower-is-better metrics, and averages those normalized values. Higher average normalized performance wins, so the larger score differential decides the order.
If both the average rank and normalized tie-break score are equal, models share the displayed position and the following position is skipped.
We don't aggregate metrics into one overall score, as this would apply subjective weighting.
Per-domain leaders are shown; cross-domain leaders are not.

13. Privacy and publication

Benchmark visibility is the owner’s decision. Some partners publish their benchmarks; others keep them private. For private benchmarks, scores are published, while evaluation scripts, prompts, grading logic, and trajectories are not.

MacroscopeBench

Main Insights

Results

How models handle real work,
on benchmarks written by practitioners

What {{ keyInsightBoards }} benchmarks show

{{ ki.claim }}

Benchmarks

{{ category.name }}

Methodology

Blogs

Want to be on the Index?

Methodology

Fireworks methodology

Benchmark methodologies

{{ group.name }}

Paper, repository, and sources

{{ part.title }}

{{ bench.name }}

Models

A rank tells you where you stand. Training moves it.