τ³-Banking

Policy-bound banking agents handle realistic customer requests with database tools and a simulated user. The board covers the 97-task banking_knowledge subset, with four official seeded trials per task and binary rewards for correct final state and required natural-language assertions.

Last updated

Main Insights

Results

τ³-Banking results, ranked by score (mean trial reward), highest first. Last updated .
RankModelProviderScore (mean trial reward)Cost (USD / trial)Duration (seconds / trial)
1Kimi K3.1 ProMoonshot AI63.3%$0.95387
2Gemini 4 ArgonGoogle58.2%$0.58578
3Claude Opus 5.5Anthropic56.0%$0.45169
4GLM-5.3Z.ai51.6%$0.49282
5Kimi K3Moonshot AI51.5%$1.02423
6Gemini 3.8 FlashGoogle51.4%$0.53285
7Qwen 3.8 MaxAlibaba51.3%$0.87294
8Grok 4.6xAI50.7%$1.32398
9Claude Opus 5Anthropic49.1%$2.59537
10GPT-6 SolOpenAI47.0%$0.67366
11DeepSeek V4.1 Flash (1017)DeepSeek46.8%$0.085182
12GPT-6 AstraOpenAI44.8%$1.80411
13GPT-6.1 SolOpenAI44.7%$0.30380
14DeepSeek V4 Pro 0813DeepSeek42.7%$0.32506
15Ember-1Fireworks38.1%$0.66183
16GPT-6 LunaOpenAI37.7%$0.055455
17Muse Spark OpenMeta37.2%$0.36331
18DeepSeek V4.1 FlashDeepSeek35.7%$0.069331
19GPT-5.6 SolOpenAI35.7%$1.14334
20Muse Spark 1.2Meta35.4%$0.29220