IN THIS BREAKDOWN · 10 SECTIONS
You can read 19% of this breakdown.
The paper in one minute
The paper starts from a familiar annoyance. You correct a coding agent ("don't leave debug files behind"), it says it will remember, and in a later session it does the same thing again. The authors call this the access-compliance gap: having a preference in context is not the same as complying with it.
They measure it first. From two months of one AI researcher's coding-agent transcripts (32 transcripts of roughly a million tokens each) they extracted 142 cases where a user correction was paired with a violation, and built 19 held-out tasks with 29 preference checks. Across six models, agents satisfied 31.6% of preferences with no rules, 42.5% with Mem0 memory, 54.0% with the relevant rules in context, 55.0% with every rule in context, and 70.1% when the rules were compiled into checks.
TRACE (Test-time Rule Acquisition and Compiled Enforcement) is the pipeline that produces those compiled rules. A small model (Gemma 4 31B) scans each user message for a correction. A correction becomes an atomic rule with an applicability condition. A resolver reconciles it with the user's library using five actions: Noop, Update, Supersede, Split or New. Each rule then compiles into three parts: a check for whether it applies, an instruction injected into the agent's context, and a verifier attached to the agent runtime's hooks. If the verifier fails, the hook blocks the event and returns the violation, and the agent must fix it before it can finish.
It ships as a skill for Claude Code and Codex CLI. With simulated users, it cut violations and repeated corrections on two benchmarks without slowing tasks down.
Research notes: the problem, approach and results
preference violations on held-out ClawArena rounds, Mem0 memory versus TRACE; no memory violated 100%
the same on unseen ClawArena scenario families, where TRACE had zero violations on three of four models
user turns per round on unseen ClawArena families, no memory versus TRACE; 1.00 means never corrected
Coding agents forget nothing and still repeat mistakes. A user says "clean up the debug files before you stop", the agent stores it, and next week it retrieves the memory and leaves the files anyway. In a diagnostic built from one user's transcripts, Mem0 memory still left 57.5% of applicable preference checks violated.
Change the representation. Detect corrections in the chat, rewrite each as an atomic rule with a condition for when it applies, reconcile it with the user's existing rules, and compile it into a verifier attached to agent hooks (prompt, tool use, file write, stop). A failed verifier blocks the action and tells the agent which rule it broke.
On ClawArena coding tasks, violation fell to 37.6% in distribution and 2.0% out of distribution, against 51.1% and 30.7% for Mem0 and 100% for no memory. Users needed 1.37 and 1.02 turns per round instead of 2.00, at about the same wall-clock time.
Why it matters to an FDE
Every agent you leave with a customer accumulates "house rules": run the linter before committing, never touch the migrations folder, ask before deleting anything, use the staging bucket. Teams usually put these in an instructions file or a memory store and hope. This paper measures the hope: in its diagnostic, memory and context left close to half the preferences broken.
The practical lesson is to split house rules by how they can be checked.
Deterministic rules become hooks. "No files left in tmp/", "tests pass before commit", "never run git push --force" can be verified from commands, file names or workspace state. A check that blocks the action is far more reliable than a sentence in the prompt.
Judgment rules stay as instructions. "Keep the code readable" or "match our naming style" need a model to judge, and the paper's deployment never actually used its model-judged tier.
Rules need a lifecycle. Preferences change, conflict and get refined. Without an update and supersede step, a library of corrections ends up holding both "clean the logs" and "keep the logs", and no run can satisfy both.
Read the headline numbers against the right baseline. "100% to 37.6%" compares against an agent with no memory at all. Against Mem0, the drop is 51.1% to 37.6% in distribution and 30.7% to 2.0% on unseen families: smaller in one case, larger in the other, and real in both.
Understand the mechanism. Try it on a real problem.
Unlock this breakdown and 149 more papers.
Premium opens every paper in full, plus every answer, concept, course and guide on the site. One payment for six months of access.
6 months · One payment · No auto-renewal
- ◇1 architecture diagram
- </>1 runnable code experiment
- ▦2 results tables
- ?5 interview questions with answers
- 03Key takeaways
What was measured, how rules are built and maintained, the honest baselines, and misreadings
Free account - 04How it works
Detection, atomization, the resolver's decision protocol, compilation and the hook loop
Premium - 05The numbers that matter
The diagnostic, the four result panels, efficiency, simulator fidelity and the store sizes
Premium - 06Try it yourself
Re-baseline the headlines, run the paper's own compiled rule, and replay a correction stream
Premium - 07Where it breaks
Single-user data, simulated users, unequal retries, brittle checks and paper slips
Premium - 08Using it on a deployment
Turning a customer's house rules into hooks, with a lifecycle and tests
Premium - 09Interview angles
Five questions this paper prepares you for, with answer hints
Premium - 10Read next
Where to go from here on this site
Premium