Self-Improving Agent Harnesses: Harness², MemoHarness and Meta-Harness Compared, With Our Own Runs

A self-improving agent harness lets a model rewrite the instructions, skills, scripts and hooks around another model, keep the changes that helped and remember why. Three 2026 research systems show three ways to do it: Meta-Harness searches offline with labels, MemoHarness searches offline and then adapts per case at test time, and Harness² improves the harness task by task with no labels at all. We ran Harness²'s credential-free demo, recomputed MemoHarness's cost table and redid both papers' arithmetic. Our view: all three work, the gains are real but smaller than the headlines, and the best thing to take from them is a discipline, not a framework. Tie every harness edit to a moment in a trajectory. Run it on the same task before you keep it. Record whether it helped, hurt or is confounded. Run the expensive loop offline, then ship a frozen harness.

Why this matters now

A year ago "the harness" meant a system prompt and some tools. Now it means a tree of files: always-on instructions, standing rules, memory files, tool guidance, on-demand skills, helper scripts, subagents, and plugins that hook the tool loop. We covered how teams build these by hand in our post on harness engineering. Every one of those files is something a capable model can edit. The question these papers answer is whether letting a model do that editing makes the agent better, and under what evidence.

For forward deployed engineers the question is concrete. You arrive at a customer, the agent fails on their documents in ways nobody predicted, and you have a week. Hand-tuning the harness for each failure does not scale past the first few. Automatic improvement promises to, if it can tell a good edit from a bad one without an answer key.

Three approaches side by side

The column that matters most is the fourth from the top: how each one knows an edit helped. With labels, you measure. Without them, Harness² compares two runs of the same task and reads the difference, for example "the reference run never reopened its output file, the candidate did, and its totals now reconcile". Self-improvement without labels is self-consistency. It reliably improves process (did the agent check its work, use the right tool, finish the deliverable). It cannot tell you the answer is true.

What we ran

Harness² ships harness2-demo, which runs the real recursion and library code with deterministic local editors and real Python executions on two small accounting tasks, in both parallel and sequential modes. No model credentials are needed. We installed the repository at commit 1cfe254 in a clean environment and ran it at three budgets.

The counts match the method exactly. Per task, the two modes share k reference runs, then parallel recursion adds 2k candidate runs and k composed runs, and sequential recursion adds 3 runs per step: 7k per task, or 4k for one mode. That is the multiplier you pay. At k = 5, one task costs twenty agent runs and seventeen editor sessions in a single mode.

The library entry it wrote is the best illustration of the method we have seen. For the second ledger task, the demo's EXPERIENCE.md records the adopted harness, then one row per edit: the component (scripts/summarize.py), the change (dropped a filter that summed only positive amounts), what was observed (the probe's total included the credits the reference run omitted), and the verdict, helped. Then a transferable lesson with a cue ("when a ledger includes credits and debits") and the evidence it rests on. Note also what the demo shows about reuse: the second task still started from the base harness and still needed the fix to be proposed and probed. The library guides the next edit; it does not pre-apply it.

The numbers, with the noise left in

Harness² (Google Cloud AI Research and Northwestern, October 2026) tested two executors, Gemini 3.5 Flash and GLM-5, under two stock harnesses on three benchmarks, with Claude Opus 4.8 as the editor. One improvement step beat the base harness in all 12 settings, by 1.2 to 11.8 points; five candidates per scope added 2.6 to 6.7 more. In our breakdown of the Harness² paper we tested each one-step margin over the strongest competing method using the reported means and standard deviations (three runs each): only 1 of 12 is clearly larger than the noise. The sign pattern is the stronger evidence: ahead of the base in 12 of 12 settings and ahead of the best baseline in 11 of 12. One more caveat: on JobBench the grader is also Claude Opus 4.8, the same model that edits harnesses and picks the final answer.

MemoHarness (Notre Dame, LMU Munich and USC, July 2026) searched a six-dimensional harness space with GPT-5.3-Codex and adapted it per case at test time. On an 18-task held-out Terminal-Bench split it scored 0.806 against 0.722 for Codex, the strongest baseline: about one and a half more tasks per run. FinanceAgent rose from 0.600 to 0.767 and LiveCodeBench from 0.900 to 0.967. The authors are unusually candid: their limitations section says the main split is 18 tasks, results are point estimates without confidence intervals, and the experience bank and test-time adaptation are not separately ablated in every setting. They also report the validation-selected harness rather than the in-training peak (0.833 at iteration 4), which is the right choice.

The cost claim is a caching claim. MemoHarness reports $6.89 on the Terminal-Bench split against $10.28 for Codex and $9.51 for Claude Code, so it looks cheaper as well as better. It also uses the most input tokens of any system in its table, 14.18 million, of which 13.32 million (94%) were cache hits. We backed the per-token prices out of the table (they fit all five rows to within six cents: about $1.77 per million input tokens, $0.18 cached and $13.36 output) and recomputed each bill with no cache hits.

Source: MemoHarness Table 4. Reported costs come from the paper; the no-cache estimates are our calculation using rates inferred from its token counts.

With no cache hits, MemoHarness would cost $28.06 against Codex's $17.12. The authors flag that their cost result "depends on the observed cacheability"; the chart shows how much. Retrieved experience is mostly stable text, so high cache rates are plausible. But every cost table for a memory-heavy agent is a claim about cache hit rates, and you should price the worst case before you plan on the best one.

What goes wrong

The Harness² authors list nine regressions in their own runs, and they are the most useful part of either paper for a practitioner.

• Blind selection takes the weaker output. On one coding task the selector chose a candidate scoring 55 over one scoring 95, preferring the "plainest implementation".
• A wrong premise spreads. An editor wrote a rule that a key must not be "fixed"; every candidate inherited it and every candidate lost points.
• A checker shares its author's blind spot. An editor wrote a verification script that reported PASS three times on code that failed the real tests. In the paper's words, "A checker written by the editor cannot catch the editor's assumption."
• Lessons harden into rules. A skill distilled from unlabeled runs pre-decided a legal position that the grading criterion rewarded the opposite of.
• Edits without runs hurt. Editing the harness with no reference runs and no library lowered WB-code from 73.4 to 63.2. An edit you never ran is a guess with good grammar. It is the same lesson as the SkillsBench result that agent-written skills, written cold, made agents worse; see SkillsBench.

The list maps directly onto controls: show the selector the source material and an explicit checklist, rotate or diversify the editors so one premise cannot reach every candidate, keep the grader out of the editor's hands, and require a person to approve any lesson that becomes an always-on rule.

Our recommendation for customer deployments

Copy the discipline before the framework. The rules in the Harness² prompts work with any harness and any model, and they cost nothing:

1. Every edit names the trajectory moment that motivated it. If you cannot name it, the edit is speculation.
2. Prefer, in order, a divergence (some runs already did it right), a repeated failure, then a visible risk. Never fix what the runs already got right.
3. Choose the cheapest form: an on-demand skill for a triggered procedure, a script for repeated deterministic work, a subagent for a large input with a small answer, a hook only when prose has demonstrably been ignored. Always-on prose last, because every word of it is paid for on every run.
4. Write facts about this customer's data as facts, labeled as such, and procedures for the whole domain.
5. Record each edit as helped, hurt, no observed effect or confounded, and prefer confounded when one run stands behind it.
6. Keep at most one transferable lesson per task, with its cue and evidence, and only when a contrast supports it.

Run the loop offline, then freeze. Take 30 to 50 representative customer tasks, run Harness² or your own version at k = 2 to 5, have an engineer review every adopted edit and lesson, and ship the resulting harness as a versioned artifact. Re-run on a schedule or when production quality drops. Per-request self-improvement in production multiplies cost by an order of magnitude and makes the agent's behavior change between two identical requests, which customers notice.

Keep the boundary fixed. Permissions, data access, network egress and safety checks stay outside the editable surface. Harness² lets the editor write TypeScript plugins that run inside the tool loop; on a customer system that is executable code from a model and needs code review like any other.

One library per customer. The library holds facts from the tasks it learned on. Sharing it across customers moves one customer's data into another's prompts.

Licenses matter. Harness² is Apache-2.0. MemoHarness had no license file when we checked on 21 October 2026, which means reading and learning are fine but copying its code into a client project needs the authors' permission.

Pros and cons

Self-improving harnesses are the most practical research direction in agents this year, because they work with the models you already have and produce edits a person can read. They are not a reason to stop reviewing. If anything, they create more for engineers to review, faster. If you want to be ready to talk about this in an interview, be able to explain the difference between improving with labels and without, why consistency across settings beats a single margin, and what you would keep outside the editable surface. Our Harness Engineering course and the agentic evals and trajectories concept cover the foundations.