SecIT Bench

Thirty incident investigations across intrusion and breach, covert data egress, service errors, and performance degradation, each built from telemetry captured on real infrastructure. Some scenarios are deliberately benign. Three LLM judges grade every root-cause report against binary diagnostic conditions, so the score is the share of conditions met rather than a pass rate.

Last updated · Source leaderboard checked daily

Main Insights

Results

SecIT Bench results, ranked by score (avg@3), highest first. Last updated .
RankModelProviderScore (avg@3)Cost (USD / task)Duration (seconds / task)
1DeepSeek V4.1 FlashDeepSeek82.9%$1.13716
2GLM 5.3Z.ai81.4%$2.70838
3GPT-6 AstraOpenAI81.0%$8.21499
4Grok 4.6xAI80.7%$3.27426
5GPT-6.1 SolOpenAI79.9%$1.94626
6Claude Opus 5Anthropic79.2%$6.11315
7Qwen 3.8 MaxAlibaba78.5%$3.39663
8Muse Spark 1.3Meta77.5%$3.65508
9Kimi K3Moonshot AI77.0%$3.13632
10Mistral Large 4Mistral76.4%$1.78668
11MiMo V2.6 FlashXiaomi76.0%$0.331,013
12GLM 5.3 FlashZ.ai75.5%$0.28689
13GPT-5.6 SolOpenAI75.4%$4.62209
14Claude Sonnet 5.5Anthropic75.1%$0.72165
15Ember-1Fireworks74.5%$3.01350
16Gemini 3.8 FlashGoogle69.0%$19.24869
17GPT-6 SolOpenAI59.3%$2.46497
18GPT-6 LunaOpenAI58.0%$0.13835