SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering

Human engineers do not fix bugs in a raw terminal. They use editors with search, syntax highlighting and linting. The Princeton team asked whether language model agents also need tools built for them, and found the answer was a clear yes.

They call the tool layer an agent-computer interface (ACI): the commands an agent can run and the format of what comes back. SWE-agent's ACI is deliberately small. find_file and search_dir return a short summary of where a term appears, and refuse when there are too many hits, asking for a narrower query. A file viewer shows 100 lines at a time with line numbers and how many lines are above and below. An edit command replaces a range of lines, shows the result, and runs a linter; if the edit introduces a syntax error, it is rejected and the agent is told why. Plus the usual shell for running code and tests.

On SWE-bench, which asks agents to resolve real GitHub issues in real Python repositories and checks the result with the project's tests, SWE-agent with GPT-4 Turbo resolved 12.5% of issues. The best earlier approach, which retrieved files and generated a patch without interacting with the repository, resolved 3.8%. On the 300-issue Lite subset, the same model with only a shell resolved 11.0%; with the ACI, 18.0%.

The ablations are the lasting contribution: they show which interface choices help and which hurt, with the model held fixed.

Most agent projects start by picking a model and wiring up the customer's existing APIs as tools. This paper says the tools themselves deserve as much design effort as the prompt, and shows what good tool design looks like.

Feedback format changes behavior. A search command that showed results one at a time, like an editor's find-next, made the agent page through every match and burn its budget. It scored worse than having no search command at all (12.0% against 15.7%). A search that returned a compact summary scored best. Customer APIs that return pages of JSON need the same treatment: summarize, cap and tell the agent how to narrow its request.

Guardrails stop cascades. Rejecting edits that break syntax, with a clear message, added three points on its own. After one failed edit, the chance of eventually succeeding fell from 90.5% to 57.2%. Validate tool inputs before they change anything, and return actionable errors.

Less context can be better. Showing whole files instead of a 100-line window cut performance from 18.0% to 12.7%; keeping the full history rather than the last five observations cut it to 15.0%. More information is not more help.

Failure is slow and expensive. Successful runs finished at a median of 12 steps and $1.21; failed runs averaged 21 steps and $2.52, and 93% of successes submitted before hitting the budget. Raising the budget mostly buys more expensive failures.