The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason

SWE-Bench asks a model to resolve real GitHub issues in real repositories, with the project's own tests deciding success. Its Verified subset, 500 tasks from 12 Python repositories checked by human annotators, became the standard number for coding agents, with scores above 70% by mid-2025. Those repositories and their fixes are also public, and likely in the training data of the models being tested.

The authors design three tasks that should be nearly impossible without having seen the answers. File-path identification: given only the repository name and the issue text, name the file that the fix changes. Function reproduction: given the issue and a file with the changed functions removed, plus their names, write them back; success means matching the real fix word for word, which reasoning alone should rarely do. Prefix completion: given the code before a changed hunk, continue it, and check for verbatim matches.

They run these on SWE-Bench Verified and on comparison sets: 200 other SWE-Bench tasks, 217 newer issues from the same repositories, 245 tasks they built from seven other popular repositories (numpy, pandas, pytorch, scipy, celery, aiohttp, jupyter notebook), an internal C# benchmark, and a refactoring benchmark. Ten OpenAI and Anthropic models scored 60% to 76% on file paths for Verified, lower on every other set, and still clearly higher on Verified after removing issues that mention a path. Reproduced functions shared up to 34.9% of their 5-word sequences with the real fix on Verified, against at most 18% elsewhere. The authors conclude that Verified scores are partly inflated by memorization of both specific instances and these repositories.

Treat public coding-benchmark scores as an upper bound. A model that has seen a repository's history can locate and patch bugs there partly from memory. A customer's private codebase offers no such help, so expect lower performance there than on SWE-Bench Verified.

Test on the customer's own repositories. Pick recently closed issues from the customer's code, after the model's training cutoff, and score the agent with their tests. That is the number to put in a proposal.

Use the paper's probes as cheap checks. Asking for the file path from the issue text alone costs one call per task and needs no execution environment. If a model does much better on one benchmark than on comparable fresh tasks, distrust that benchmark for that model.

Compare like with like. The paper's sets differ in more than freshness: Verified issues were selected by annotators for being clearly written, and refactoring tasks have 15-token descriptions. When you build comparison sets for a customer, match on issue quality and repository size, or the gap will mix memory with difficulty.

Mind the curated subset. The highest scores came on the most widely used, most discussed subset. Any heavily reported test set accumulates exposure over time.