AI Agents That Matter

By mid-2024, agent papers competed on leaderboards: HumanEval for coding, WebArena for web tasks, many others. The Princeton team (Sayash Kapoor and Arvind Narayanan's group) looked at how those results were produced and found four problems.

Cost was ignored. Language models are stochastic, so simply calling one several times raises accuracy on any task where answers can be checked. The authors built three trivial baselines for HumanEval: retry up to five times on failed tests; "warming", which retries while raising the temperature; and escalation, which starts with a cheap model and moves to more expensive ones only on failure. These matched the best published agents, which used reflection, planning and debugging. Those agents cost more, one of them over 50 times more, and none of their papers compared against retrying or reported dollar costs.

Model evaluation and procurement were conflated. Researchers want to know which technique is better; buyers want to know what to deploy at what price. A benchmark built for one misleads the other.

Benchmarks lacked held-out sets. Many agent benchmarks are small and public, so agents can overfit them. The top WebArena agent, for example, hard-coded site-specific URL patterns.

Evaluations were not reproducible. Scripts, versions and setups differed enough that results could not be compared.

The prescription: plot accuracy against cost and pick from the Pareto frontier, optimize both jointly, report token counts so costs can be recalculated, and build benchmarks with secret held-out sets that reflect how the system will be used.

A customer will show you a leaderboard and ask for the top agent. This paper explains why that is the wrong question and what to measure instead.

Always run the simple baseline first. If the task has a check (tests pass, the API call succeeds, the extracted total matches), retrying the same model, or escalating from a small model to a large one on failure, is often as good as an elaborate agent and much cheaper. If a complex design does not beat retry-on-failure at the same cost, it is not earning its complexity.

Report cost per task, with token counts. Prices change monthly. Token counts do not. Recording input and output tokens per task lets anyone recompute the bill at current prices and keeps an evaluation useful after the next price cut.

Benchmarks are built for someone else. A model-research benchmark can distort a deployment decision badly. On NovelQA, which asks every question about a novel in a single prompt, retrieval looked about half the cost of long context. Asked one question at a time, as real users ask, it was more than 20 times cheaper at similar accuracy. Build the customer's evaluation around how their users will actually use the system.

Watch for shortcuts. If an agent's score depends on knowing the test environment's specifics, it will not survive the customer's environment changing.