Towards a Science of AI Agent Reliability

A benchmark score is one number: the share of tasks an agent completed. It cannot tell an agent that always fails the same five tasks from one that fails a different five every run, or a harmless formatting mistake from deleting a production database. Deployed agents have done the second kind of thing; the paper cites Replit's assistant deleting a production database against instructions and OpenAI's Operator making a purchase without confirmation.

Sayash Kapoor, Arvind Narayanan and colleagues at Princeton started from fields that already certify reliability: aviation, nuclear power, automotive and process control. Across those, four dimensions recur. Consistency: does it do the same thing when run again? Robustness: does it degrade gracefully when conditions change? Predictability: does it know when it is likely to fail? Safety: when it fails, how bad is it? The paper turns these into twelve metrics, designed to be independent of raw accuracy, plus an overall score that deliberately leaves safety out so a rare catastrophic failure is never averaged away.

They then ran 15 models from OpenAI, Google and Anthropic, released between early 2024 and mid 2026, on two benchmarks: GAIA (general assistant tasks with browsing, files and code) and a cleaned 26-task subset of τ-bench (airline customer service). Every task ran five times, plus runs with paraphrased instructions, injected tool faults and changed tool formats.

The finding: over 24 months of releases, reliability improved far more slowly than accuracy, barely at all on GAIA. Calibration and safety improved in recent models. Outcome consistency stayed modest everywhere, models were thrown by reworded instructions more than by real tool failures, and the ability to tell a likely success from a likely failure did not improve on GAIA.

Customers ask "how accurate is it?" The questions that decide whether a deployment survives are different: will it do the same thing tomorrow, will it break when someone rephrases a request, will it tell us when it is unsure, and how bad is the worst thing it can do? This paper gives each of those a definition and a way to measure it.

Run every evaluation more than once. The paper sets temperature to zero and still finds that agents which can solve a task often fail it on another run. A single pass over your test set hides this. Five runs per task is the paper's protocol and a sensible default.

Test paraphrases, not just faults. Models handled injected tool errors and API changes better than reworded instructions. Real users never phrase a request the way your test set does, so paraphrase robustness is the robustness that matters most for user-facing agents.

Do not trust confidence without checking discrimination. Models have become better calibrated, meaning their average confidence matches their average success. That is not the same as knowing which specific runs will fail, which is what you need to route uncertain cases to a person, and on GAIA that did not improve.

Match the bar to the autonomy. The authors argue reliability matters most where no human reviews the output. A coding assistant can be inconsistent because a developer approves each change; an agent issuing refunds cannot.