Legacy-Bench

Maintenance work on the enterprise code that still runs payroll, settlement, and billing: fixing bugs, implementing features, and migrating code across COBOL, Java 7, BASIC, C89, Fortran, and Assembly. Each task ships a natural-language instruction and a containerized environment, and a task passes only when hidden verification tests the agent never sees all pass.

Last updated

Main Insights

Results

Legacy-Bench results, ranked by score (avg@3), highest first. Last updated .
RankModelProviderScore (avg@3)Cost (USD / task)Duration (seconds / task)
1Kimi K3.1 ProMoonshot AI63.0%$0.38466
2Claude Opus 5.5Anthropic62.0%$0.49434
3Muse Spark 1.3Meta62.0%$1.98744
4DeepSeek V4.1 FlashDeepSeek57.7%$0.14691
5DeepSeek V4.1 Flash (1017)DeepSeek56.3%$0.13357
6GPT-6 AstraOpenAI55.7%$2.33629
7GPT-6.1 SolOpenAI55.0%$0.45629
8Grok 4.7xAI54.0%$3.001,013
9Kimi K3Moonshot AI52.7%$0.741,418
10Gemini 3.8 FlashGoogle52.3%$1.251,038
11GLM 5.3Z.ai50.7%$0.951,659
12Muse Spark OpenMeta50.0%$0.841,554
13Gemini 4 ArgonGoogle44.3%$0.313,562