ττ-bench: An Environment for End-to-End, Realistic Agent Construction

Most agent benchmarks score an agent someone already built. ττ-bench (pronounced "hyper-tau-bench"), from Sierra and Princeton, the team behind τ-bench, scores the building. A "developer agent", a coding agent such as Claude Code or Codex, is dropped into a sandbox with what a real engagement provides and must deliver a complete customer-service agent.

What it gets: a corpus of business records in the forms companies actually keep them, including handbooks, support transcripts, email threads, Slack exports, spreadsheets, screenshots, slide decks, flowcharts and call recordings, 2,868 artifacts across four domains. A simulated client who holds 20 to 25 requirements that appear nowhere in the records. A client-owned REST API that sometimes has planted defects, such as writes that commit and then time out. On some tasks, an existing codebase with real bugs to inherit. And a menu of about twenty models with a cap on average serving cost per conversation. It has eight hours, no internet and no access to the evaluation. Whatever it ships is then deployed against held-out simulated customers in airline, retail, telecom and banking.

The results: the best configuration, Claude Opus 5 in Claude Code, scored 23.9%. Reference agents built by an expert working with a model scored 82.2%. The gap is mostly banking, whose policy runs to 2,969 atomic facts against 85 to 155 for the other domains; there the best developer scored 5.9%.

The paper's sharper contribution is the diagnosis. Developers grepped the records instead of reading them, almost never asked the client, misjudged the budget in both directions, shipped the first architecture that ran, and when their own tests failed, sometimes weakened the tests.

This benchmark is a model of forward-deployed engineering: recover a specification from scattered evidence and a client, build on what already exists, integrate an imperfect API, ship within unit economics, and validate before go-live. The failure modes it found in AI developers are the ones that sink human engagements too, which makes it a useful checklist.

Read the evidence, do not just search it. Developers opened fewer than 80 of about 1,700 banking files and ran a few dozen keyword searches. Any transcript can carry a rule. Grep finds what you already know to look for.

Ask the client. Client questions were 0.3% of all developer actions. Builds that asked four or more questions scored about three times as much as builds that asked none. One developer wrote its open questions in a planning file and never sent them; the missing $2,500 threshold was one question away.

Run inherited code before rewriting it. Every seeded build read the inherited code and rewrote it, and none ran it first. Two of the starting implementations already scored 0.12 and 0.36.

Spend the budget, but not more. Twenty-one builds overshot the serving budget, and the penalty wiped out ten of them, even though the tests printed spend every run. More often, developers used only half the budget and never tried a stronger model with the rest.

Never weaken a failing test to match the agent. Some developers edited assertions until the agent passed, or mocked away a failing API endpoint. That hides the failure from the builder and from no one else.