Write the acceptance contract before the first agent run

TL;DR: Write acceptance as a checkable relation between a task, its input revision, the allowed changes and the evidence produced. Treat a passing result as version-bound evidence that can become stale when the task or artifact changes.

Where you are. You have separated the model from the harness; now define what the Northstar coding worker is allowed to finish.

From request to observable claim

The supplied request is small: the procurement workbench should require buyer review when an order total reaches the review limit. Today, an order exactly at the limit skips review. Northstar's synthetic rule uses integer cents and says total_cents >= limit_cents. The fixture limit is 100000 cents. There is no currency conversion or tax calculation in this exercise.

“Fix the approval bug” is too vague to hand to an autonomous worker. It does not say whether an order exactly at the limit needs review, which files may change, or what counts as a finished repair. Given that much room, a worker could change the limit, weaken the test, or add a label that looks right while the API behaves as before.

A useful task contract names the target behavior and rules out those shortcuts. For task T-17, the candidate may change the threshold implementation and its explanatory UI text. It may not change the acceptance cases, the configured limit, the user's role or the procurement service's authorization checks. The output is a patch for review; deployment is a separate operation.

The contract should also name the legitimate reasons to stop. Missing dependency access is an environment block, and an ambiguous purchasing policy calls for a clarification request. Neither should be recorded as “passed with assumptions” just to make the task list look tidy.

Evidence bound to what was tested

Keep a small evidence record:

These labels are illustrative identities. A real release record needs resolvable revisions or content digests and a trusted writer. A digest detects a byte difference when compared with a trusted value; it does not establish that whoever supplied the value was honest.

A boolean checklist cannot express whether a result is still current, and that is the decision that matters later. A task accepted last week may need revalidation after a new patch changes a shared dependency. Keep the earlier evidence as history and change the task's current status. Do not rewrite history to say it never passed, and do not treat the old pass as proof for the new revision.

Checks that resist shortcuts

The boundary cases below isolate the equality defect. They are necessary for this task, but they are not a complete procurement test suite. Negative amounts, unsupported currencies and role checks belong to other explicit contracts.

cases = [(99999, False), (100000, True), (100001, True)]

def needs_review(total_cents, limit_cents):
    return total_cents >= limit_cents

observed = [needs_review(amount, 100000) for amount, _ in cases]
expected = [answer for _, answer in cases]
assert observed == expected
print(observed)

The output is [False, True, True]. Keep the acceptance cases outside the worker's editable files. A worker can add its own unit tests, but those tests cannot be the sole authority for accepting its patch. The independent runner should also report a failing process exit or missing result as a failure to obtain evidence.

The shortcuts named earlier are concrete enough to run. The table takes five candidate patches, including the three shortcuts, and runs each against the boundary cases, against the same cases at a second limit, and against a check that the protected files are untouched.

Every cell comes from running the five candidates as predicates on the cases at both limits and comparing their edited files with the protected list. D fails the cases because the runner reads its own copy of them, not the worker's edited one.

Row B is the one to study. A patch that compares the total to the literal 100000 passes every boundary case and edits nothing protected, so a contract built only on the original fixture accepts it. It would fail later, when the customer moves the limit, which is the "change the input" exercise below arriving as a production incident. The fix costs one loop: run the same three cases at a second limit during acceptance, so the constant fails today.

def needs_review(total_cents, limit_cents):
    return total_cents >= limit_cents

for limit in (100000, 150000):
    cases = [(limit - 1, False), (limit, True), (limit + 1, True)]
    assert [needs_review(t, limit) for t, _ in cases] == [e for _, e in cases]
print("ok")

This prints ok; swap in the hard-coded version and the assertion fails on the second limit. Row C shows the other half of the contract earning its place. Moving the configured limit to 99999 and keeping > passes the cases at 100000, and only the protected-file check rejects it, which is why protected surfaces are a line in the contract rather than a convention.

A small patch does not earn a wider scope. Changing one comparison character can alter approval behavior for every order at the boundary, so scope follows the operation and deployment the change affects, not the number of edited lines.

Do this before moving on

Deliverable: write a six-line acceptance contract for T-17: outcome, allowed edits, protected surfaces, required evidence, completion boundary and blocked conditions. Use the supplied amounts and exact fixture limit.

Worked review. A satisfactory contract requires review for totals of 100000 cents or more, permits only the threshold implementation and explanatory UI text, protects the limit and evaluator, requires the three boundary results plus a browser observation of the review state, finishes at a review-ready patch, and blocks on unavailable checks or unclear policy. The browser observation tests the wiring; the numeric cases test the predicate.

Change the input. The customer raises the limit to 150000 cents after the patch passes. Keep the old result as historical evidence, update the contract and run cases at 149999, 150000 and 150001. A hard-coded implementation that only works at 100000 must fail. A generic comparison should produce the same three booleans at the new boundary.

Kept this way, a task ledger stays useful across sessions: it records what still has to be proved, so the next worker does not have to infer it from an optimistic progress summary.

Go deeper

• Evaluation gates: Connect the task’s acceptance evidence to release regression checks.
• Verifier gaming: Inspect how a worker can satisfy a weak check without satisfying the customer.
• The evaluation harness comes first: Practice turning customer expectations into an agreed evaluation before implementation.

Key takeaways

• Bind acceptance evidence to the specification, artifact, fixtures and verifier.
• Preserve past results while invalidating their use for a changed revision.
• Protect acceptance checks and state the boundary between a patch and deployment.