τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment

τ²-Bench evaluates an agent helping a user who can also change the world. In its telecom environment, the agent has backend account tools while the user has phone tools. Fixing a connection may require both an account change and a device change, with neither participant directly observing everything the other can see.

This differs from a benchmark where the user only supplies information and the agent performs all operations. The assistant must explain an action, elicit the right observation and adapt when the result differs from its expectation. A correct plan can still fail through poor coordination.

The benchmark constructs tasks from smaller components with initialization steps, known solution actions and final assertions. A tool-grounded user simulator performs device actions against the same evolving environment. This reduces reliance on a language model simply claiming that an instruction worked.

For GPT-4.1, the reported telecom success rate is 34% in the normal interaction setting and 52% when the agent directly controls all tools. The comparison identifies a coordination burden in this environment. It is not a general estimate of how much human users reduce AI performance. The official repository provides the implementation needed to inspect and reproduce the task contract.

Many enterprise workflows share control. A service desk assistant can reset an account but cannot physically reconnect a cable. A customer can change a device setting but cannot alter the backend subscription. Evaluating only the agent's own tool calls misses the handoff that determines whether the workflow finishes.

The useful design lesson is to model each actor's authority and observations explicitly. Do not let a user simulator report success merely because the assistant asked confidently. Tie observations to environment state, and make incomplete or mistaken actions visible to the agent through the same interface it would have in deployment.

For an agent gym, this also changes the test plan. Score the terminal state, inspect the trajectory and measure repeated success. A good answer in a transcript can coexist with an unresolved customer problem, an unauthorized action or instructions the user could not follow.