o11y-bench

Sixty-four observability tasks in a running Grafana stack with Prometheus, Loki, and Tempo: querying metrics, logs, and traces, investigating incidents across them, and creating, editing, and repairing dashboards. Grading checks the agent’s claims against facts read from the same environment and inspects the dashboards it saves, so a plausible answer alone does not pass. The score is the share of tasks passed on all three attempts.

Last updated · Source leaderboard checked daily

Main Insights

Results

o11y-bench results, ranked by score (pass^3), highest first. Last updated .
RankModelProviderScore (pass^3)Cost (USD / task)Duration (seconds / task)
1Claude Opus 5.5Anthropic87.5%$1.69282
2GPT-6.1 SolOpenAI85.9%$0.056111
3Claude Sonnet 5.5Anthropic84.4%$1.34329
4GPT-6 LunaOpenAI84.4%$0.0063117
5GPT-6 AstraOpenAI84.4%$0.33112
6Claude Haiku 5.5Anthropic82.8%$0.072297
7Kimi K3Moonshot AI81.3%$0.18209
8GLM 5.3Z.ai81.3%$0.14257
9Mistral Large 4Mistral81.3%$0.10223
10Gemini 4 ArgonGoogle81.3%$0.37217
11DeepSeek V4.1 Flash (1017)DeepSeek81.3%$0.059181
12Muse Spark OpenMeta79.7%$0.12155
13Qwen 3.8 MaxAlibaba78.1%$0.25122
14MiniMax M3.1 FlashMiniMax64.1%$0.027235
15Qwen 3.8 27BAlibaba60.9%$0.062253
16Gemma 4 31BGoogle53.1%$0.050220
17Qwen 3.6 27BAlibaba51.6%$0.091232
18Qwen 3.6 35B-A3BAlibaba43.8%$0.021202
19Grok 4.7xAI26.6%$0.14110