Measuring AI Ability to Complete Long Software Tasks

An agent's fifty-percent task horizon is the human duration of tasks it completes with roughly even odds under the evaluation setup. If the horizon is an hour, that describes work taking the reference humans about an hour. It does not mean the agent can safely operate for an hour, or that it takes an hour to produce its answer.

METR measures human performance on software and research tasks, runs agents repeatedly on those tasks, and fits a relationship between human task time and agent success. That produces a more interpretable capability measure than a single average benchmark score. The paper then studies how the fitted horizon changes across historical model releases.

The reviewed version finds a strong historical growth trend, with an estimated doubling time of roughly seven months. The authors also discuss substantial limits: the tasks are selected, the reference humans matter, and less structured work can behave differently. Extrapolating the trend is a conditional forecast rather than a delivery commitment.

The important deployment lesson is to state the required reliability. A fifty-percent success boundary can help compare research systems, but it is far below what many unattended customer actions require. You need a local evaluation and an explicit handoff policy before using a headline horizon to scope automation.

Customers describe work in familiar units: a short ticket, an afternoon migration, a recurring weekly analysis. Relating agent performance to human task time can make an initial scoping conversation more concrete. It also forces you to define what successful completion means and whose effort supplies the baseline.

That baseline can change the apparent difficulty. A maintainer with repository context may finish quickly; an equally skilled newcomer may spend most of the time finding the relevant files. An agent given a prepared workspace and explicit tests has another information advantage. Record those differences before comparing durations.

Use the paper's method to structure a pilot, then add the bounded autonomy and human review controls the workflow needs. Evaluate permissions and recovery separately from task completion. An agent that reaches the desired output after an unauthorized action has failed the deployment contract even if the benchmark grader accepts its final artifact.