Meta-Harness: End-to-End Optimization of Model Harnesses

The same model can score very differently depending on the code wrapped around it: what examples it is shown, what it remembers between steps, how retrieved material is formatted. That code, the harness, is usually tuned by an engineer reading failures and editing by hand. Yoonho Lee, Chelsea Finn and colleagues at Stanford, with Omar Khattab at MIT, ask whether a coding agent can do that work itself.

Meta-Harness is a deliberately simple outer loop. A coding agent (Claude Code with Opus 4.6 in all experiments) is given a directory containing every harness tried so far: its source code, its scores, and the full execution traces of its runs, including prompts, tool calls and model outputs. It explores that directory with ordinary tools, decides what went wrong, and writes new harness programs. Those are evaluated on a search set of tasks, the results are written back into the directory, and the loop repeats. A typical run evaluates about 60 harnesses over 20 iterations.

The authors' argument is that earlier optimizers threw away the evidence. Methods such as OPRO, TextGrad, GEPA and OpenEvolve see a score, the current candidate, or a short summary. A harness decision often fails many steps later, and only the raw trace shows why. Their ablation supports this: an agent shown full traces found far better harnesses than the same agent shown scores or LLM-written summaries.

Most of an FDE's tuning time goes into the harness, not the model: the retrieval step, the prompt assembly, the memory policy. This paper shows that a capable coding agent can do some of that iteration, and it tells you what to give it.

Keep the traces, not just the scores. Whether a person or an agent does the tuning, the most useful artifact is the full record of what the system saw and did on each failure. Summaries lose the detail that explains the failure. Log runs so they can be read later.

Separate the tasks you tune on from the tasks you report. Text classification and math here used held-out test sets. TerminalBench-2 did not: the harness was searched and scored on the same 89 tasks, and its lead over the best hand-built harness was about two tasks. When you tune on a customer's eval set, the score on that set will flatter you.

Read what the optimizer wrote. Found harnesses are code. You can review them for hard-coded answers, brittle special cases or leaked labels, which is a real advantage over tuning weights.

Budget for evaluation. Each candidate must be run on many tasks. The authors note that evaluation, not proposal, was the main cost.