τ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge

The most common enterprise agent is a support agent that answers from the company's own documentation and acts on the customer's account. Existing benchmarks split this job in two: retrieval benchmarks check whether the right passage comes back, and tool-use benchmarks hand the agent a clean list of tools. Neither captures an agent that must read internal policy to learn what it is allowed to do, and only then do it, mid-conversation.

τ-Knowledge, from Sierra and Princeton (the team behind τ-bench), adds that missing piece with a new domain, τ-Banking. The knowledge base has 698 documents, about 195,000 tokens, covering 21 product categories: accounts, savings tiers, credit cards, referral programs, dispute procedures, identity checks. Fifty-one of the tools the agent needs are not in its tool list; they are described only inside documents, and the agent must find them and call them by name. A simulated customer, scripted with rules for key moments, talks to the agent. Each of the 97 tasks is graded by whether the bank's database ends in the correct state.

The results are sobering. The best setup, GPT-5.2 with high reasoning searching the documents through a terminal, passed 25.52% of tasks on a single try, and fewer when required to pass four tries in a row (pass^4). With the needed documents placed directly in context, removing search as a factor, the best model, Claude 4.5 Opus, reached only 39.69%. Failures came from making assumptions instead of asking or searching, from missing how products and policies interact, from doing steps in the wrong order, and from believing customers without checking.

If you are deploying an agent over a customer's help center, policy manuals and back-office tools, τ-Banking is the closest public stand-in for your project, and its lessons transfer directly.

Better retrieval is not the whole fix. With perfect retrieval, the best model still failed six tasks in ten. Policy interactions ("credit limit requests are rejected while a dispute is pending"), multi-document reasoning and state checks are where agents broke. Budget your time for policy reasoning and verification, not only for embedding models.

Search strategy is a design choice with a latency bill. Letting the strongest reasoning models grep through files beat embedding search on average, but it took more calls and longer turns. GPT-5.2 (high) matched Claude 4.5 Opus's accuracy while taking about nine times longer per task. In a live chat, that difference is the product.

Agents trust customers too much. When a simulated customer claimed their disputes had been approved, many agents applied credits without checking the dispute status. Any action with money or permissions needs a verification step in the workflow, not just a policy sentence in the prompt.

Measure consistency. Scores fall sharply from pass^1 to pass^4. A support agent that handles the same request differently on different days is a support problem even if its average looks fine.