dfbench

A held-out evaluation of three jobs a security engineer actually does: finding vulnerabilities in a codebase, judging whether a reported finding is real, and working out what a diff changed. Detection and differential analysis run on separate task sets with their own ground truth, so the two are scored and costed independently rather than averaged into one number.

Last updated · Source leaderboard checked daily

Main Insights

Results

dfbench results, ranked by score (detection recall), highest first. Last updated .
RankModelProviderScore (detection recall)Cost (USD / task)
1Mythos 5†Anthropic69.0%—
2GPT 5.6 SolOpenAI · xhigh65.7%$27.02
3dfs-large1depthfirst62.2%$4.05
4GPT 6 SolOpenAI · xhigh59.3%$8.60
5Grok 4.7xAI · xhigh59.0%$13.25
6DeepSeek v4.1 FlashDeepSeek · high57.3%$1.20
7Grok 4.5xAI · high57.2%$5.54
8MiMo-v2.6 ProXiaomi55.0%$0.75
9Opus 5.5Anthropic · high54.9%$5.10
10MiMo-v2.6 FlashXiaomi53.5%$0.47
11GPT 5.6 LunaOpenAI · xhigh52.4%$1.57
12Kimi K3Moonshot AI · max48.0%$7.72