SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

By 2023, coding benchmarks were close to saturated. HumanEval asks a model to write one short function from a docstring, and frontier models passed most of it. Real software work is different: find the relevant code in a large repository, understand how pieces interact, change several places at once, and not break anything.

SWE-bench, from Princeton (Carlos Jimenez, John Yang, Ofir Press, Karthik Narasimhan and colleagues), builds evaluation tasks from that kind of work. The authors scraped about 90,000 pull requests from 12 popular Python projects, including Django, scikit-learn, sympy, matplotlib and Sphinx. They kept merged pull requests that resolve a GitHub issue and add or change tests, then ran each candidate: at least one test had to go from failing before the fix to passing after it, and the project had to install and run cleanly. That left 2,294 tasks.

A model receives the issue text and the repository at the commit before the fix, and must produce a patch. The patch is applied and the project's tests run. The task counts as resolved only if the patch applies, the tests the human fix made pass now pass (fail-to-pass), and the existing tests that passed before still pass (pass-to-pass).

The results were humbling. Using BM25 to retrieve likely relevant files, Claude 2 resolved 1.96% of tasks. Even when given exactly the files the human fix edited, it resolved 4.8%. A later version of the paper added Claude 3 Opus at 3.79%. Models often produced valid diffs that did not fix anything, struggled to locate the code to change, and did worse as more code was put in their context.

SWE-bench scores are now quoted in every coding-model and coding-agent announcement. Knowing what the number measures, and what it leaves out, is part of advising a customer on coding tools.

It is the template for a good task evaluation. Each task pairs a real request with a check written by someone who understood the fix: tests that must start passing and tests that must keep passing. You can build the same thing from a customer's own history: closed tickets with merged fixes and tests become an evaluation set for their codebase, which is more predictive than any public leaderboard.

Pass-to-pass tests catch the greedy fix. In the paper's example, a model changed the right function in Sphinx but made it behave as if a configuration setting were always on, instead of checking the setting. The new test passed; an existing test caught the regression. Customer evaluations need both kinds of test, or they reward fixes that break something else.

Localization is the hard part. Given the right files, models did much better than when they had to find them, and more context hurt. That points to the retrieval, search and navigation around a model, which is the integration work FDEs do, as the lever for coding agents on large customer codebases.

It is Python, open source and tested. A customer's monorepo in another language, with thin test coverage, may behave very differently from the leaderboard.