An Empirical Study of Harness Design for Coding Agents

A coding agent is a model inside a harness: the loop that sends the model a prompt, runs the tool calls it emits, and feeds back the results. Harnesses make many choices at once, such as whether the model keeps a plan, whether it gets file tools or only a shell, and what happens when the conversation outgrows the context window. When two products score differently, those choices are tangled together with the model. This paper untangles them.

The authors, from UMass Amherst, Emory and UNC Charlotte working with Zoom, built a lightweight harness on LangGraph with a fixed ReAct loop and three swappable parts. Planning: an explicit plan the model maintains through an update_plan tool, on or off. Action space: eight predefined tools (read, write, edit, list, glob, grep, fetch, bash) or bash alone. Context management: five tiers, from none (T0) to elision, recall, summarization, and a staged combination (T4). They ran three sizes of NVIDIA's Nemotron-3 (30B, 120B, 550B) and Mistral-Medium-3.5 on SWE-Bench Verified (500 GitHub issues) and Terminal-Bench 2.1 (89 terminal tasks), at four context budgets from 32k to 128k tokens.

Four findings came out. Context management matters most when the window is small, mostly by preventing overflow. Cheap rule-based elision before LLM summarization gives the best accuracy for the money. Planning keeps a weak model working long enough to try a fix, while it mainly trims redundant checking for strong models. A predefined tool set props up models that are poor at writing shell commands but adds overhead for models that are good at it.

Customers rarely get to choose the strongest model. They get the one their contract, region or budget allows, often a smaller open-weight model served in their own cloud. This paper is a guide to adjusting the harness for that model rather than copying the settings a frontier model needs.

Context handling is a must when windows are small. With a 32k window and no management, most SWE-Bench runs died on overflow before reaching a fix. If the customer's deployment caps context, compaction is not an optimization, it is what lets the agent finish.

Use cheap compaction first. Replacing stale tool outputs with stubs, and summarizing only when that is not enough, matched the accuracy of heavier strategies at the lowest cost. The fancier option, letting the model re-fetch elided content, was almost never used.

Test planning and tools per model. The same plan scaffold raised a 30B model's success by 11.6 points and, for the largest model, mostly cut cost. Giving a bash-fluent model a bare shell halved its cost; giving the same shell to another model cost 23 points. Nothing here is a universal default.

Measure on enough tasks. Many differences the authors describe on the 89-task benchmark are not statistically significant. A customer eval with a few dozen tasks will mislead you the same way.