Measuring Agents in Production

Agent research leans toward autonomy: long-horizon planning, reinforcement learning, open-ended tool use. This study asks what the teams with agents in front of real users actually do, and the answer is mostly the opposite.

The authors, a group from Berkeley, UIUC, Stanford, IBM Research and the bank Intesa Sanpaolo, ran 20 semi-structured interviews with teams whose agents were in production (14) or pilot (6), across insurance, HR, customer care, scientific workflows, software operations and more. They also surveyed 306 practitioners between July and October 2025, and kept the 86 responses describing systems in production or pilot. They call the study MAP, Measuring Agents in Production.

The findings, briefly. Teams build agents to save time: 80% cite productivity and 72% fewer human hours. Most agents serve people (93%), often employees first, and most tolerate slow answers, because a few minutes still beats a human doing the task. Seventeen of 20 case-study teams use closed frontier models, 14 of 20 never fine-tune, and humans write most prompts. Sixteen of 20 use structured workflows rather than open-ended planning, 68% of surveyed systems run fewer than 10 steps before a human intervenes, and 17 of 20 teams wrote their own agent code instead of using a framework. Evaluation is mostly people: 74% use human-in-the-loop checks, and 15 of 20 teams have no formal benchmark.

Reliability is still the top challenge. The paper's summary: practitioners get reliability from system design, such as constraints, workflows and human checkpoints, rather than from smarter models or new algorithms.

This is the closest thing to a field survey of your job. When a customer asks how other companies run agents, these are real answers, collected under confidentiality from teams that shipped.

It licenses the boring design. A fixed sequence of steps, each handled by a model, with a human approving the output, is not a compromise. It is what the majority of deployed systems look like. One insurance case runs a fixed sequence of coverage lookup, medical necessity review and risk identification; the agent is autonomous within each step, and the sequence never changes.

It tells you where to start a deployment. Internal users first (52% of systems serve employees), read-only actions where possible, and humans as the final verifiers. Teams described this as deliberate risk management, not caution for its own sake.

It explains why the model version is a risk. Teams avoid fine-tuning partly because it must be redone when the base model changes, and one team runs legacy models beside new ones because upgrades broke their workflows. Prompts, scaffolds and evaluations lock onto one model's behavior.

It sets expectations about evaluation. Most teams cannot buy a benchmark for their task. Plan to build a golden set with the customer's experts, collect production feedback, and route uncertain outputs to people. That is the norm, not a gap in your project.