Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

METR recruited 16 developers who maintain large open-source projects: repositories averaging 23,000 stars, 1.1 million lines of code and about 10 years of history, where each developer had worked for 5 years on average and made about 1,500 commits. Each developer listed real issues they wanted to work on, typically bug fixes and features of up to two hours. Before anything was assigned, they forecast how long each issue would take with and without AI tools.

Each issue was then randomly assigned. If AI was allowed, the developer could use any tool or none; most used Cursor Pro with Claude 3.5 or 3.7 Sonnet, between February and June 2025. If AI was disallowed, no generative AI was permitted. Developers recorded their screens, submitted pull requests that went through their projects' normal review, and reported the time spent before and after review. They were paid $150 an hour.

Across 246 completed issues (136 AI-allowed, 110 not), allowing AI increased implementation time by 19%. Developers had forecast a 24% reduction. After the study, they estimated AI had cut their time by 20%. Economics experts (34 of them) and ML experts (54) forecast reductions of about 39% and 38%.

To explain the gap, the authors labeled 143 hours of screen recordings at roughly 10-second resolution and examined 21 candidate explanations. They found evidence that five contributed: over-optimism about AI, developers' deep familiarity with their code, the size and complexity of the repositories, low reliability of AI output (developers accepted under 44% of generations), and context the AI did not have. They caution that the result is specific to this setting and to the tools of early 2025.

Measure the outcome, not the impression. Developers who were measurably slower reported being faster, both before and after. If a customer's case for an AI coding rollout rests on a survey of how fast people feel, this paper is the reason to ask for a measured comparison.

Fix the task before you measure. The trial defined each issue before assignment, so AI could not change what counted as done. Lines of code, commits and pull request counts can all rise without more work being finished; the paper found 47% more lines per forecasted hour on AI-allowed issues.

Expect the effect to depend on the user and the codebase. The slowdown was larger on issues where developers rated themselves expert and needed no references. Developers said AI helped most on unfamiliar parts of the code. A rollout plan should expect gains for newcomers and on unfamiliar code before it expects them for the people who know the system best.

Count review time as part of the cost. On labeled recordings, developers spent about 9% of AI-allowed time reviewing and cleaning AI output and 4% waiting for it. Those minutes belong in any productivity estimate.

Plan the measurement before the pilot. A trial of 246 tasks produced a confidence interval roughly 37 points wide. A customer pilot with a few dozen tasks cannot distinguish a 10% gain from a 10% loss, so decide in advance what result would change the decision and how many tasks it needs.