τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

The gap it targets

Language agents are systems that act by calling external tools through APIs. Benchmarks for such agents had not tested two things that matter in deployment: how an agent handles a back-and-forth with a human user, and whether it follows rules specific to a domain ^[1]. τ-bench was built to fill exactly that gap ^[1].

How the benchmark works

A τ-bench task is a conversation. The user's side is played by a language model, so the exchange can develop differently depending on what the agent says and does. The agent's side is given two things: a set of domain-specific API tools it can call, and policy guidelines — written rules of the kind a company hands to its support staff ^[1]. Because both the user and the tools are live parts of the setup, the benchmark emulates dynamic conversation rather than a fixed script the agent reads through once ^[1].

Scoring does not look at the wording of the agent's replies. At the end of the conversation, the benchmark compares the state of the database with the annotated goal state for that task ^[1]. What counts is whether the right changes were actually made through the tools — whether the order was cancelled, the record corrected, the account updated — not whether the agent sounded helpful or produced a plausible sentence. That makes the target of evaluation the outcome in the world the agent is acting on.

A metric for consistency: pass^k

The paper proposes a new metric, pass^k, to evaluate how reliable an agent's behavior is across multiple trials ^[1]. Ordinary accuracy answers a narrower question: did the agent get this task right once? A system that is run again and again by real users has to do better than that, and pass^k is designed to register the difference by looking at repeated attempts at the same task rather than a single attempt.

What the results showed

The headline finding is that strong agents are far from reliable on this kind of work. Even state-of-the-art function-calling agents such as GPT-4o succeed on less than 50% of the tasks, and they are quite inconsistent: in the retail domain, pass^8 is below 25% ^[1]. In other words, an agent that manages a task on one attempt often fails when the same task comes around again, and the drop is large enough to be visible in the metric built to measure it.

Why it matters

The benchmark puts two abilities side by side that are usually measured separately: carrying a conversation with a user whose requests unfold over several turns, and staying inside the rules of a domain while doing so ^[1]. It also ties success to database state instead of to generated text ^[1]. Together with pass^k, the setup shifts attention from whether an agent can perform a task at all to whether it can be depended on to perform it consistently ^[1]. For anyone building or deploying tool-using agents, that is the question that decides whether the system can be put in front of real users.