AutomationBench

Six hundred fifty-seven held-out business workflows across sales, marketing, operations, support, finance, and HR, each in a small simulated company spread over 47 SaaS apps. The agent gets one request, finds the API endpoints it needs through search, follows rules that may sit in an inbox or a spreadsheet, and writes to each system. Deterministic checks on the final state decide the task, and it passes only if every check holds, including the ones that catch a notice sent to the wrong team.

Last updated · Source leaderboard checked daily

Main Insights

Results

AutomationBench results, ranked by score (strict pass rate), highest first. Last updated .
RankModelProviderScore (strict pass rate)Cost (USD / task)
1Gemini 4 CarbonGoogle53.9%$2.12
2Claude Fable 5.2Anthropic51.4%$4.64
3Gemini 4 ArgonGoogle51.3%$1.70
4Gemini 4 HydrogenGoogle51.1%$0.82
5Claude Sonnet 5.5Anthropic44.7%$1.14
6Claude Opus 5.5Anthropic42.5%$1.44
7GPT-6 AstraOpenAI41.4%$1.73
8GPT-6 SolOpenAI33.2%$0.27
9Gemini 3.7 FlashGoogle30.4%$0.61
10Gemini 3.8 FlashGoogle29.7%$0.55
11GPT-5.6 SolOpenAI28.8%$0.67
12Claude Opus 5Anthropic26.9%$3.05
13GPT-5.6 TerraOpenAI23.6%$0.51
14Kimi K3Moonshot AI22.7%$0.43
15Claude Fable 5.1Anthropic22.4%$2.45