DeepSWE v1.1

Deep debugging tasks where the failure cause sits several layers from the symptom.

Last updated

Main Insights

Results

DeepSWE v1.1 results, ranked by score (avg@3), highest first. Last updated .
RankModelProviderScore (avg@3)Cost (USD / task)Duration (seconds / task)
1Claude Opus 5Anthropic72.3%$6.181,309
2GPT-6 AstraOpenAI71.0%$2.56681
3Kimi K3Moonshot AI70.2%$5.402,340
4DeepSeek V4.1 Flash (1017)DeepSeek69.6%$0.17505
5GPT-6.1 SolOpenAI69.3%$1.071,149
6Claude Opus 5.5Anthropic68.4%$13.163,110
7GLM 5.3Z.ai68.1%$4.732,294
8Gemini 4 ArgonGoogle67.8%$5.433,165
9GPT-6 SolOpenAI67.3%$2.31997
10Ember-1Fireworks67.0%$4.191,712
11GPT-6 LunaOpenAI66.4%$0.191,240
12DeepSeek v4.1 FlashDeepSeek63.4%$0.921,317
13GPT-5.6 SolOpenAI62.0%$2.01545
14Gemini 3.8 FlashGoogle61.6%$4.241,180
15Grok 4.6xAI57.8%$6.035,240
16Muse Spark OpenMeta53.4%$2.441,568
17DeepSeek V4 Pro 0813DeepSeek45.1%$1.121,264