Your AI Coding Agent Says Done. What Evidence Should You Trust?

A coding agent has finished a task when current, trustworthy evidence shows that its submitted change meets the agreed requirements. Its final message is a useful handoff, but acceptance belongs to the system running the checks. That system must know what was tested, which rules applied, whether execution completed, and whether the candidate changed afterward.

This is a practical part of harness engineering: making completion observable. The same discipline helps an agent stop before an unsafe action, recover after an interrupted tool call, and explain why it needs a person. It also gives you a way to remove expensive machinery when it stops helping.

We will follow a fictional customer change from request to release decision. The numbers and records below are teaching fixtures, not measured results from a deployed agent.

Start with a requirement someone can check

Suppose a procurement team asks: “Update the approval threshold in our internal tool.” A plausible implementation changes a label. Another changes the API but leaves the form using the old rule. Both can produce a convincing screenshot.

The missing information is the behavior. For this example, the customer confirms that an amendment of 100,000 cents or more requires review, the threshold is configurable, and the form and API must agree. Below-threshold requests still need the ordinary authorization checks. This change must not create approvals or contact vendors.

Write down the accepted behavior and the permitted work before starting the worker:

A specification does not become useful merely because it has more fields. Each field should resolve a choice the worker would otherwise guess. Ask a colleague to read this contract and predict the equality case. If two readers disagree, the worker will have the same ambiguity.

The free harness boundary lesson develops this separation between the proposed work and the host's control of execution. For a full requirements-to-release journey, use the Autonomous ADLC guide.

Make the repository navigable from a clean session

The worker needs to find the policy function, the UI consumer and the verification command. A short repository entry file can point to those locations and to the relevant setup instructions. Detailed rules belong near the area they govern, with clear ownership and version history.

Test that map. Start from a clean checkout with the intended permissions, follow its commands, and confirm that they reach the behavior in the contract. A reference to a deleted script is a concrete defect. A diagram of the architecture will not compensate for it.

OpenAI's engineering account describes a short entry map backed by structured repository documentation and automated checks for its maintenance. The useful lesson is discoverability and upkeep; it does not establish a universal file length. OpenAI's harness engineering account.

Keep historical traces available without loading every incident into every task. A previous failure can justify a check, but the permanent rule should describe the invariant. If an earlier worker bypassed the policy service, an import-boundary check can reject that dependency. Repeating “be careful with policy” in several documents leaves the same bypass available.

The scoped instruction lesson includes a worked exercise on turning an observed mistake into a small, maintainable rule.

Separate local feedback from acceptance evidence

The worker should run cheap checks while editing. Fast feedback makes repairs easier. It may also add exploratory tests, propose cases the original specification missed, or show a minimal reproduction of an apparent bug in the acceptance suite.

Those are contributions to the review. The authority to accept the deliverable stays outside the worker-editable environment.

For our threshold change, a protected runner takes an immutable candidate and the approved test bundle. It runs the boundary cases, checks a changed configuration, and exercises the UI-to-API path. It records the result itself. If the worker prints PASS, that text remains part of the worker's log; it cannot create an accepted result.

The diagram separates two identities that a green badge often hides: the thing being tested and the rules used to test it. A result for candidate c17 under evaluator e4 says nothing about an untested candidate c18. A later evaluator e5 may add a missing equality case; the old result does not establish that c17 passes it.

An evidence record should bind at least:

• The candidate artifact identity, including the relevant source and generated artifacts.
• The task contract, evaluator, fixtures and environment identities needed to reproduce the check.
• The required checks and their actual outcomes, including interruption or missing evidence.
• The trusted runner identity, run identifier and retained logs or output references.

A commit hash is useful only if it identifies everything that mattered. Uncommitted files, a floating dependency, a mutable container tag or a changed external fixture can invalidate that assumption. Use the artifact and environment identifiers appropriate to the system, and record the remaining uncertainty.

A hash proves correspondence to some bytes. It does not prove who produced the bytes or whether the test was honest. Protect the result writer and its credentials. Running candidate code in a fresh container helps reproducibility, but the container still needs a security boundary that prevents candidate code from modifying the evaluator, writing its result or stealing its credentials. GitHub's workflow security guidance explains why executing untrusted code alongside privileged workflow capabilities needs particular care. GitHub secure use reference.

Treat “not verified” as its own outcome

Imagine four records arriving at the release controller:

B and C both block release, but they require different repairs. B contains evidence of an implementation defect. C may be an infrastructure problem, an agent-induced resource failure or a broken evaluator. Preserve that distinction instead of turning every interrupted run into a model-quality score.

Integration can invalidate a result even when nobody edits the agent's patch. If the base branch changes an API that the patch calls, testing the branch before that change is insufficient evidence about the combined result. Configure the repository's required checks and merge policy deliberately; do not assume the platform always tests the artifact you intend to release. GitHub protected branches.

Try the stale verification interview drill, then use Eval Regression Suites and CI Gates for the broader distinction between invariant checks and noisy quality scores.

Let a reviewer challenge what the checks missed

A reviewer with a fresh conversation can inspect the contract, diff and observed results without inheriting the builder's entire explanation. Give it a precise question: which requirement lacks evidence, which changed behavior creates a new risk, and what observation would resolve that concern?

Fresh context is useful, but it is not independent authority. Two models can share a mistaken assumption. An executable test can encode that assumption too. Include deliberate defects that the verifier must reject, and keep a reviewed path for correcting the specification when a worker finds a genuine contradiction.

For the running example, change >= to > and confirm the equality check fails. Then hard-code 100,000 and confirm the changed-threshold case fails. Finally, leave the API correct but keep the UI label stale: the user-visible check should catch that mismatch. Each mutation demonstrates a particular detection capability. It does not certify the absence of other bugs.

The protected evaluator lesson adds an evidence-admission exercise, while the verifier-gaming question examines deliberate attempts to weaken the acceptance signal.

Recover the operation before restarting the loop

Suppose the worker has permission to create a draft change request through a service. It submits the request, the service commits it, and the reply is lost. The local trace ends with a timeout.

Starting a fresh session and submitting again with a new operation identifier can create a second draft. The error message describes what the caller observed, not what the service committed.

In this diagram, the amber state remains unresolved until evidence from the action service changes it. The next step is to query the existing operation or use the service's documented idempotency behavior. Retrying with the same identifier is safe only under that contract, including its payload-matching and retention rules. If the service offers neither dependable reconciliation nor safe deduplication, hand off with the uncertainty intact.

AWS's discussion of idempotent APIs explains the importance of preserving caller intent, handling late requests and rejecting reuse of a request identifier with different parameters. These details belong in the tool adapter, where they can be enforced. AWS Builders' Library.

For other failures, choose a response from the evidence. A transient read failure may justify bounded backoff. A schema error needs a corrected request. A permission denial needs an authorized change or a handoff. Repeating an unchanged test failure needs a new diagnostic step, not an unlimited repair loop.

A repeated observation is only a clue. A healthy asynchronous job can return running several times. Combine that observation with job identity, the polling contract, the deadline and any progress evidence. The agent failure modes concept works through this counterexample.

Keep handoff notes separate from operational truth

A progress note can tell the next worker what was attempted and where to look. It should link to the candidate, the current contract, unresolved operations and the last trusted result. It should not independently assert that a remote action failed or that an approval remains valid.

On resume, compare the note with the durable task record and authoritative service receipts. If the note says “draft creation failed” but the service has a committed draft under the operation key, repair the note and continue from the known effect. If policy changed while the run was suspended, recheck authority before the next action.

This distinction prevents a common failure: treating an articulate summary as a database. Summaries help navigation. Operational records decide what happened and what may happen next. Practice the recovery boundary in the kill-and-recover project.

Remove scaffolding with an experiment

A stronger model may no longer benefit from a particular planning reminder or extra review pass. Keeping it can add cost and delay. Removing it blindly can expose a failure that your recent tasks did not happen to exercise.

Use a component ablation: compare the system with and without that one component. For a model upgrade, run both configurations on both models. Keep the task population, acceptance suite and authority controls fixed so that the comparison answers a specific question.

These supplied numbers illustrate an interaction. On 40 synthetic task records, the older model has 30 accepted tasks with the review reminder and 24 without it. The newer model has 34 accepted tasks in either configuration. That pattern makes the reminder a candidate for removal on the newer model; it does not establish equivalence. Different cases could account for those 34 successes, and repeated trials may reveal a rare failure.

Inspect paired wins and losses, task slices, uncertainty and all attempt costs. A rollout can be limited to the task class where the change is supported. Preserve the ability to restore the previous profile.

Authority enforcement is outside this optimization. A lower observed rate of unauthorized proposals does not justify giving the worker production credentials. Test defenses with controlled fault injection in isolation, but hold the production permission boundary fixed while measuring a productivity change.

LangChain reported a 52.8% to 66.5% Terminal-Bench 2.0 improvement with GPT-5.2-Codex fixed and several harness changes. That is useful evidence that the surrounding system matters, not a promise that one review hook produces a 13.7-point gain on your tasks. Research on model-harness interactions likewise supports evaluating the pairing on the actual workload. LangChain's experiment, Finding the Right Fit.

Build a small acceptance packet this week

Take one bounded change from a real backlog. Write its observable behavior, relevant boundary cases, permitted actions and blocked outcome. Make the setup and verification commands discoverable from a clean checkout. Then retain the exact candidate, evaluator and result together.

Before accepting it, run three drills: introduce a defect the suite must detect; change the candidate after a passing result; interrupt one tool operation after submission. For each drill, record the observed state and the next permitted action. These exercises expose gaps that a successful demonstration can hide.

Continue with the paired harness comparison project. The Premium Harness Engineering field guide includes printable exercises for evidence binding, ambiguous outcomes and safe simplification, alongside its reference companion and operational checklists.