Should Kimi K3.1 Pro replace the model powering your agent, or work alongside it? We compared it with Kimi K3 and leading alternatives using practitioner-built benchmarks in the Specialized Intelligence Index (SII), examining task performance, model cost and execution time.
K3.1 Pro scores higher than K3 on fourteen of seventeen shared benchmarks, with gains that include more completed legacy-code tasks and more precise actions in customer support in banking. Against other leading models, the opportunity varies: it approaches Claude Opus 5.5’s web-research score at substantially lower model cost, but its higher resolution rate in banking comes with greater spending and longer waits.
Those differences also create opportunities to combine models’ strengths. In our routing simulations, assigning work between K3.1 Pro and GPT-6 Astra or GPT-6.1 Sol produced higher combined scores than either model alone. Pairing it with Opus cut model spending by more than half. The findings help distinguish where to consider K3.1 Pro as a replacement and where retaining another model for some tasks may offer more value.
What we evaluated
The study draws on 21 benchmark leaderboards, comparing K3.1 Pro with its predecessor and other models on quality, model cost and execution time where available. We examined task results and agent traces more closely on four: Sierra’s τ³-Banking for customer support in banking, Epiq’s CorporateBench for company records and emails, Rogo’s Big Finance Benchmark for financial research, and Factory’s Legacy-Bench for legacy-code maintenance. These task results also underpin the routing simulations. Letta and Surge AI supplied additional analyses of memory creation and software-engineering behavior, respectively. See Methodology and limitations for evaluation details.
K3.1 Pro improves on K3 across most benchmarks, with important exceptions
K3.1 Pro scores higher than K3 on fourteen of seventeen shared benchmarks. The task-level analysis shows both more completed work and more precise actions, but also regressions in memory creation and contract review.

Where K3.1 Pro improves
Legacy-code maintenance. Factory’s Legacy-Bench tests bug fixes, feature additions and migrations in older languages and environments, including COBOL, Fortran and Java 7. K3.1 Pro completes 15% more tasks than K3, with a completion rate of 62.5% versus 54.3% across the same 97 tasks. Model cost is 48% lower than K3’s.
Customer support in banking. Sierra’s τ³-Banking tests agents using bank policies and account-management tools to resolve customer requests. The inspected logs show K3.1 Pro following requests more precisely: the share of account-changing tool calls containing unrequested fields fell by about 69%, from 8.5% with K3 to 2.6% with K3.1 Pro.
A separate example makes the consequence concrete. In a conversation involving three of a customer’s bank cards, K3 closed all three. K3.1 Pro froze the cards, checked the transactions and closed only the one that required closure, avoiding unnecessary closure of the other two.
Where the upgrade falls short
Memory creation. Letta’s FrontierMemory tests whether an agent can turn records of previous attempts into memories that help another agent solve coding and knowledge-work tasks. A coordinating agent distributes those records to workers, which extract and save useful information. The benchmark measures the later agent’s task success using those memories.
With K3.1 Pro creating the memories, the benchmark score is 22% lower than with K3. Inspected runs reveal a potential contributing problem: K3.1 Pro leaves many source records unassigned to workers.
K3.1 Pro was less thorough than K3 in this coordinating role. It assigned only about 50-65% of the available records, while K3 assigned the full set in two of three runs. For one task with 22 records, K3.1 Pro assigned just one record in one run and none in another. Neither run produced memories for that task, leaving the later agent without the benefit of those records.
Contract review. Legal Agent Benchmark (LAB) tests assignments a first-year legal associate might receive across practice areas. K3.1 Pro scores 18% higher than K3, rising from 13.1 to 15.4. But on the separate LAB: Contracts benchmark, which tests contract review, redlining and issue spotting, its score falls 64%, from 8.3 to 3.0. For teams using K3 for contract review, the broader LAB improvement is not evidence that they should switch.
These regressions argue against treating K3.1 Pro as an automatic replacement for K3 throughout an application, even when its overall benchmark record is stronger.
How K3.1 Pro compares with leading alternatives
Web research: near Opus 5.5’s score at 72% lower model cost
The Parallel Search Capability Leaderboard tests models on difficult questions requiring web search. K3.1 Pro costs 72% less than Claude Opus 5.5 ($0.27 versus $0.95 per question), with a score 1.2 points lower (74.2 versus 75.4). For teams using Opus on high-volume web research, that makes K3.1 Pro a candidate for reducing model spending.

Financial research: GPT-6.1 Sol scores higher and costs less than K3.1 Pro
Rogo’s Big Finance Benchmark measures how accurately models answer financial analysts’ questions using Rogo’s research tools. K3.1 Pro matches Claude Opus 5.5’s 48.2% score at 81% lower model cost ($0.17 versus $0.91 per question). GPT-6.1 Sol scores 52.0% at about $0.05 per question, scoring higher while costing roughly 69% less than K3.1 Pro. Among these models, Sol offers the stronger combination of score and cost for this workload.

Corporate research: higher scores than Opus 5.5 at 83% lower model cost
Epiq’s CorporateBench tests whether agents can reconstruct company activity from the email archives of four synthetic companies. Tasks include identifying employees’ reporting relationships, classifying documents by topic and answering questions that require connecting facts across messages. Answers are checked against known company records, with each company and capability weighted equally in the headline score.

K3.1 Pro earns a higher average score than Claude Opus 5.5, 78.4 versus 75.4, at 83% lower model cost ($0.47 versus $2.80 per task). GPT-6.1 Sol scores higher than both, at 82.2, and costs $0.41 per task.
Employee histories are a particular weakness. On 16 tasks asking models to reconstruct employees’ reporting relationships and role start dates, K3.1 Pro averages 78 out of 100, compared with GPT-6 Astra’s 96 and Sol’s 93. In inspected logs, K3.1 Pro sometimes concludes that an employee has no manager because no document explicitly names one. Astra instead infers the relationship from evidence such as whom the employee asks for direction.

Customer support in banking: K3.1 Pro resolves more requests than Opus 5.5, at higher cost and latency
Sierra’s τ³-Banking tests whether agents can resolve customer requests using bank policies and account-management tools. K3.1 Pro resolves 13% more requests than Claude Opus 5.5, completing 63.3% of attempts versus 56.0%.
Higher completion on difficult requests. The benchmark includes 12 conversations that at least 15 of the 20 evaluated models failed to resolve in any run. Across repeated evaluation runs, K3.1 Pro resolved an average of five of those 12 conversations, while no other model averaged more than two.

The inspected logs show K3.1 Pro consulting bank policies more extensively, averaging 22 knowledge-base lookups per conversation, compared with 10-17 for the other models in the analysis. That may help explain its performance on complex requests, but the logs do not establish that additional lookups caused the completion advantage.
Higher spending and longer waits. K3.1 Pro processes roughly three times as many input tokens per conversation as Opus, outweighing its lower token prices. Even with its higher completion rate, it costs $1.51 per successful resolution versus Opus’s $0.80, about 89% more. This divides total model spending, including failed attempts, by successful resolutions. Average execution time per attempt is also more than twice as long.
For a support team, additional resolutions could justify the premium if they avoid costly human handoffs, but longer customer waits may rule out K3.1 Pro where response-time requirements are strict.
Software engineering: identified risks went unresolved and unreported in 64% of inspected attempts
A coding agent can complete a requested change while leaving its reviewer unaware of a problem it already noticed. Surge AI found this pattern in K3.1 Pro across 256 software tasks, each attempted three times: in 64% of attempts, the model identified a risk in its reasoning but neither addressed it nor mentioned it in its final response.
Surge AI could inspect K3.1 Pro’s full reasoning, while the comparison models exposed only their written working notes. The frequency of unreported risks is therefore not directly comparable across models.
Surge AI’s separate sudo L7 benchmark evaluates changes to production codebases, including what the agent tells the user afterward. K3.1 Pro’s pass rate is 37% higher than K3’s, but it remains well behind Claude Opus 5.5 and Claude Sonnet 5.5: 16.7% versus 45.6% and 50.6%, respectively. The broader behavioral analysis does not establish how much unreported risks contributed to that result.
For teams evaluating coding agents, the final response deserves scrutiny alongside the code. Reviewers need the agent to report consequential problems it noticed but did not fix so they can decide whether the change is ready to ship.
Incident diagnosis: K3.1 Pro finds the exact cause less often than GPT-6 Astra
Traversal’s ORCA Benchmark tests diagnosis of production incidents from logs, traces and metrics. K3.1 Pro ranks third, but identifies the exact root cause in only 32.9% of cases, compared with 44.3% for Claude Opus 5 (Opus 5.5 was not evaluated on ORCA) and 50.2% for GPT-6 Astra. Its position on the leaderboard masks a substantial gap: Astra identifies the root cause in roughly half of cases, while K3.1 Pro does so in roughly a third.
What changes when you pair K3.1 Pro with your current model?
The workload comparisons show reasons to use K3.1 Pro and reasons to retain another model. Could pairing K3.1 Pro with your current model improve on using either model alone? We tested that question in a routing simulation.

How the simulation worked
We simulated three two-model ensembles across the same 461 tasks, pairing K3.1 Pro separately with Claude Opus 5.5, GPT-6 Astra and GPT-6.1 Sol. A router selected one model to handle each task; the models did not review each other’s answers or retry each other’s failures.
The benchmarks supplied 24 task categories. For each task, the router chose between models using their recorded scores and costs on other tasks in the same category. The task’s own result was excluded from that decision.
The router balanced benchmark performance against model spending by assigning a hypothetical value of $20 to each score point. This allowed it to choose a more expensive model when the expected score improvement justified the extra cost. The $20 was a simulation assumption, not a spending limit or measured business value.
What each pairing changes
For each pairing, we compared using the existing model on all 461 tasks with routing each task to either that model or K3.1 Pro.

The Astra and Sol pairings scored higher than either model alone. We resampled the task set 2,000 times and recalculated routing decisions to test how much that result depended on the tasks evaluated. The advantage held in 98% of resamples for Astra and 89% for Sol, giving stronger support to the Astra pairing’s improvement. The Opus pairing beat Opus alone in 99% of resamples and cost less in all of them, but it beat K3.1 Pro alone in only 55%.
Where the score gains and savings came from
Corporate research. On Epiq’s CorporateBench, the K3.1 Pro-Opus pairing assigned seven task categories to K3.1 Pro and kept Opus on all 16 questions about employees’ managers and role start dates, where K3.1 Pro struggled. Together, the models earned a combined score of about 100 out of 128, 11 points higher than Opus alone, while reducing model spending from $358 to $100. The Astra pairing made the same split, keeping Astra on the employee-history questions.
Customer support in banking. All three pairings assigned all 97 conversations to K3.1 Pro. This completed more requests than using Opus, Astra or Sol alone. It increased spending on banking relative to Opus and Sol, but reduced it relative to Astra.
Legacy-code maintenance. The Astra pairing assigned 93 of 97 tasks to K3.1 Pro and the remaining four to Astra. It completed about five additional tasks while cutting model spending by about 82%: 61 tasks for $43 versus 56 for $235 using Astra alone.
These results come from simulations using recorded results and known task categories. A production router would need to obtain or predict those categories. The resampling analysis measures variation in the task sample, not run-to-run variability. The study also did not test whether K3.1 Pro adds value to a larger ensemble already routing among several models.
Should you switch to K3.1 Pro?
The case for K3.1 Pro is stronger when adoption means assigning it useful work, rather than replacing an existing model everywhere. The task-level findings reveal opportunities that an overall ranking misses: where a higher score comes with higher costs, where a weakness warrants retaining another model, and where combining their strengths improves the result. That makes the scope of adoption as important as the choice of model.
Try K3.1 Pro on Fireworks to evaluate it on your workload, or explore Fireworks Nexus to use an ensemble of open and closed models inside your existing coding and co-work tools.
Methodology and limitations
Benchmark coverage and scoring
Results reflect the study’s November 9, 2026 snapshot of the Specialized Intelligence Index. Benchmark coverage, model rosters and evaluation dates vary. Scores are compared within benchmarks, whose definitions of success differ.
Task-level analysis covers Sierra’s τ³-Banking, Factory’s Legacy-Bench, Epiq’s CorporateBench and Rogo’s Big Finance Benchmark. The study uses 97 banking conversations, the 97 legacy-code tasks (3 tasks failed to build across all models, and thus are not included), 128 CorporateBench questions and 139 financial-research questions. For the routing analysis, each task contributes up to one point to the combined score. CorporateBench retains its partial-credit grading: a score of 0.75 contributes 0.75 points. The combined score sums task scores averaged across evaluation runs.
Evaluation runs and behavioral analysis
Updated K3.1 Pro results include three runs on customer support in banking, Legacy-Bench and CorporateBench, and three trials per question on Big Finance. Behavioral observations from agent logs use selected earlier runs; they illustrate observed actions and do not establish the cause of every score difference.
Letta supplied the memory-creation analysis from its setup, where an orchestrator distributes records of earlier task attempts to worker agents that extract information to preserve. These observations concern K3.1 Pro’s behavior as the orchestrator in that workflow, rather than its ability to use memories generally.
Surge AI supplied the software-engineering analysis, covering three runs on 256 tasks, including the 60-task sudo L7 set. That broader analysis is distinct from the sudo L7 leaderboard score. Comparisons involving visible reasoning and final responses should account for differences in what providers expose.
Model cost and execution time
K3.1 Pro ran on Fireworks; comparison models ran through their providers’ APIs on different dates. Model costs reflect recorded token usage under each benchmark’s pricing and caching assumptions, including Fireworks list prices for K3.1 Pro. They exclude judge costs and, where applicable, simulated-customer costs. Execution-time differences reflect serving conditions and evaluation settings as well as model behavior. Similar scores do not by themselves establish statistical equivalence.
Routing assumptions
For each task, the router compared the models’ average scores and model costs on other tasks of the same type, valuing each score point at $20. The task’s own result was excluded from the decision. Combined scores changed little when that value varied from $10 to $100, but model spending changed materially. The reported 53% and 65% savings use the $20 setting. The simulation uses recorded results rather than live traffic.
Uncertainty and limitations
The routing analysis assumes benchmark-defined task categories are available. We resampled tasks within each benchmark and recalculated routing decisions in each of 2,000 resamples, excluding every copy of the task being routed from its routing decision.
The GPT-6 Astra and GPT-6.1 Sol pairings scored higher than both constituent models alone in 98% and 89% of resamples, respectively. For the Claude Opus 5.5 pairing, the 90% interval ranged from seven points lower to eight points higher than K3.1 Pro alone. Its cost savings relative to Opus alone remained reliable.
Each task’s score was held at its average across runs, so this check measures variation in the task sample, not run-to-run variability. Combined scores are rounded to whole numbers. This check does not validate request classification or routing on production traffic.


