Write the acceptance contract before the first agent run

TL;DR: Write acceptance as a checkable relation between a task, its input revision, the allowed changes and the evidence produced. Treat a passing result as version-bound evidence that can become stale when the task or artifact changes.

Where you are. You have separated the model from the harness; now define what the Northstar coding worker is allowed to finish.

Turn a request into an observable claim

The supplied request is small: the procurement workbench should require buyer review when an order total reaches the review limit. At present, an order exactly at the limit bypasses review. Northstar's synthetic rule uses integer cents and says total_cents >= limit_cents. The fixture limit is 100000 cents. There is no currency conversion or tax calculation in this exercise.

“Fix the approval bug” is too vague to hand to an autonomous worker. It does not specify whether equality requires review, what files may change, or what counts as a completed repair. A worker could change the limit, weaken the test, or add a label that looks correct while leaving the API unchanged.

A useful task contract names the target behavior and excludes those shortcuts. For task T-17, the candidate may change the threshold implementation and its explanatory UI text. It may not change the acceptance cases, the configured limit, the user's role or the procurement service's authorization checks. The output is a patch for review. Deployment remains a separate operation.

The contract should also name legitimate stops. Missing dependency access is an environment block. Ambiguous purchasing policy is a clarification request. Neither should be converted into “passed with assumptions” merely to produce a tidy task list.

Bind proof to the thing that was tested

Keep a small evidence record:

These labels are illustrative identities. A real release record needs resolvable revisions or content digests and a trusted writer. A digest detects a byte difference when compared with a trusted value; it does not establish that whoever supplied the value was honest.

The “still current” decision is the part a boolean checklist loses. A previously accepted task can need revalidation after a later patch changes a shared dependency. Preserve its earlier evidence as history while changing its current status. Do not rewrite history to say it never passed, and do not pretend an old pass proves the new revision.

Choose checks that resist easy shortcuts

The boundary cases below isolate the equality defect. They are necessary for this task, but they are not a complete procurement test suite. Negative amounts, unsupported currencies and role checks belong to other explicit contracts.

cases = [(99999, False), (100000, True), (100001, True)]

def needs_review(total_cents, limit_cents):
    return total_cents >= limit_cents

observed = [needs_review(amount, 100000) for amount, _ in cases]
expected = [answer for _, answer in cases]
assert observed == expected
print(observed)

The output is [False, True, True]. Keep the acceptance cases outside the worker's editable files. A worker can add its own unit tests, but those tests cannot be the sole authority for accepting its patch. The independent runner should also report a failing process exit or missing result as a failure to obtain evidence.

Do not grant extra scope because the patch is small. A one-character comparison can change approval behavior for every order at the boundary. Scope follows the affected operation and deployment, not the number of edited lines.

Do this before moving on

Deliverable: write a six-line acceptance contract for T-17: outcome, allowed edits, protected surfaces, required evidence, completion boundary and blocked conditions. Use the supplied amounts and exact fixture limit.

Worked review. A satisfactory contract requires review for totals of 100000 cents or more, permits only the threshold implementation and explanatory UI text, protects the limit and evaluator, requires the three boundary results plus a browser observation of the review state, finishes at a review-ready patch, and blocks on unavailable checks or unclear policy. The browser observation tests the wiring; the numeric cases test the predicate.

Change the input. The customer raises the limit to 150000 cents after the patch passes. Keep the old result as historical evidence, update the contract and run cases at 149999, 150000 and 150001. A hard-coded implementation that only works at 100000 must fail. A generic comparison should produce the same three booleans at the new boundary.

This is also how a task ledger becomes useful across sessions. It records what remains to prove, rather than asking the next worker to infer that from an enthusiastic progress summary.

Go deeper

• Evaluation gates: Connect the task’s acceptance evidence to release regression checks.
• Verifier gaming: Inspect how a worker can satisfy a weak check without satisfying the customer.
• The evaluation harness comes first: Practice turning customer expectations into an agreed evaluation before implementation.

Key takeaways

• Bind acceptance evidence to the specification, artifact, fixtures and verifier.
• Preserve past results while invalidating their use for a changed revision.
• Protect acceptance checks and state the boundary between a patch and deployment.