Agentless: Demystifying LLM-based Software Engineering Agents

In 2024, the race on SWE-bench, the benchmark of real GitHub issues in Python projects, was a race of agents: systems that give a model a shell, file tools and a test runner, and let it decide step by step what to do. Chunqiu Steven Xia, Lingming Zhang and colleagues at the University of Illinois asked whether that autonomy was earning its keep. Current models plan poorly over long horizons, agents can wander and get stuck, and each extra turn costs money.

Agentless is their answer: a fixed pipeline with no model deciding the next action. Localization narrows the search hierarchically: the model ranks suspicious files (combined with embedding retrieval), then reads a compressed "skeleton" of those files to pick classes and functions, then picks specific edit lines, sampled four times for variety. Repair generates ten candidate patches per set of locations, forty in all, as search-and-replace edits. Validation picks one: it drops patches that break the repository's existing tests, keeps patches that pass a reproduction test the model wrote from the issue, normalizes the remaining patches so cosmetic differences do not split the vote, and submits the most common.

With GPT-4o this resolved 96 of the 300 SWE-bench Lite issues (32.0%) for $0.70 on average, ahead of every open-source agent at the time and cheaper than most. On SWE-bench Verified it resolved 38.8%. The paper also audited the benchmark itself and found issues whose descriptions contained the exact fix, lacked necessary information, or suggested a wrong fix.

A customer asking for "an agent" usually has a task with a known shape. This paper is the strongest public evidence that, for such tasks, a well-designed fixed pipeline can beat a free-roaming agent on quality, cost and debuggability.

Start with the pipeline, add autonomy if it pays. Write the stages the task obviously has, measure, and only let the model choose steps where the fixed structure demonstrably fails. Agentless outperformed agents that cost three times as much.

Generate many candidates, select with evidence. Most of the gain came from choosing among 40 patches with tests, not from generating better ones. When a customer's task has a checker, such as tests, a schema, a reconciliation or a business rule, use it to pick among samples.

Narrow before you generate. Hierarchical localization with compressed views (file skeletons instead of whole files) kept prompts small and accurate. The same idea applies to finding the right document, record or code path in customer data.

Audit the benchmark before trusting it. About one in five SWE-bench Lite issues was flawed. Any customer eval set built from historical tickets will have the same problems: answers in the question, missing information, misleading hints.

Cheap per item changes what is possible. At 70 cents an issue, running every incoming ticket through the pipeline is a budget conversation, not a research project.