Choose proof that reaches the user-visible behavior

TL;DR: Choose verification that reaches the behavior the customer actually uses. Unit checks, service integration and browser observation cover different failure surfaces; tie each result to the tested artifact and report the important paths that remain untested.

Where you are. Northstar's runtime can recover. You are deciding what evidence is sufficient to accept a coding change or a procurement outcome.

A correct function can still produce a broken product

The threshold predicate passes at 99999, 100000 and 100001 cents, yet the browser shows “Below review threshold” for every amount because the UI reads the wrong response field. The unit test is right about the function and silent about the product.

Now reverse the situation. A browser test clicks through one order at 100001 cents and sees the expected label, while the equality defect at 100000 is still there. The browser test is right about the path it observed and silent about the numeric boundary.

Neither result can stand in for the other. Use fast deterministic checks for the calculation, integration checks for the request and response contract, and browser checks for the actual interaction, and add property-based or adversarial tests where the input space is large. Choose tests by the failures they can catch; no single level of testing covers all the others.

An evidence ladder for one task

For T-17, the verification packet can be small and specific:

The packet identifies the candidate revision, fixture version and environment. The verifier should inspect an immutable snapshot of the candidate, so that a passing result cannot end up attached to bytes edited after the check ran. If the application has multiple deployed components, identify the relevant versions and configuration together.

The checks feed one acceptance decision without claiming to prove the absence of defects. A result is most convincing when a reviewer can say what it would have caught and what it would have missed.

That question can be answered by experiment rather than by argument. The script below builds a three-layer toy of the workbench (predicate, HTTP response, UI label), injects one fault at a time, and runs four checks against each version:

LIMIT = 100000

def build(fault=None):
    def predicate(total):
        return total > LIMIT if fault == "equality" else total >= LIMIT
    def http(total):
        key = "review_required" if fault == "renamed field" else "needs_review"
        return {key: predicate(total)}
    def ui(total):
        field = "review_required" if fault == "ui reads wrong field" else "needs_review"
        shown = http(total).get(field, False)
        return "Buyer review required" if shown else "Below review threshold"
    return predicate, http, ui

def predicate_cases(p, h, u):
    return [p(t) for t in (99999, 100000, 100001)] == [False, True, True]
def http_contract(p, h, u):
    return h(100000) == {"needs_review": True}
def browser_one_order(p, h, u):
    return u(100001) == "Buyer review required"
def browser_boundaries(p, h, u):
    return u(100000) == "Buyer review required" and u(99999) == "Below review threshold"

checks = [predicate_cases, http_contract, browser_one_order, browser_boundaries]
for fault in (None, "equality", "ui reads wrong field", "renamed field"):
    app = build(fault)
    row = ["pass" if check(*app) else "FAIL" for check in checks]
    print(f"{str(fault):22}", *row)

The two opening scenarios of this lesson are rows two and three. The single browser journey misses the equality fault because 100001 sits on the correct side under both comparisons, and the predicate cases miss both wiring faults because they never leave the function. The last column catches all three only because it was designed around the boundary. A browser check placed at an arbitrary amount would have inherited the third column's blind spot.

Exercising the product as a sequence

For the synthetic workbench, open the page, enter 100000 cents, run the check and observe “Buyer review required.” Change the amount to 99999 and observe “Below review threshold.” Then ask about the purchasing policy, open its citation, copy the answer and ask an unsupported shipping question.

Check empty inputs and failure states as well. The answer history should retain the earlier question. An unavailable clipboard should produce an honest error. A delayed response should remain associated with the submitted question even if the input field changes while waiting.

These details matter because users experience the whole sequence, not the function on its own. A technically correct answer behind a broken copy control, or attached to the wrong question, still makes for a poor workflow. Conversely, a polished page can display a plausible answer from the wrong tenant, so keep usability checks and authority checks as separate entries in the acceptance record.

Do this before moving on

Use these supplied observations:

• Predicate checks at the three boundary amounts pass.
• HTTP returns needs_review: true for 100000.
• The browser reads review_required and shows the below-threshold label.
• A screenshot exists, but no interaction was recorded.

Deliverable: identify the smallest repair and the checks needed afterward.

Worked review. Correct the UI's response-field mapping, then rerun the HTTP contract and both browser boundary journeys against the candidate. Preserve the passing predicate evidence if its exact artifact and dependencies remain unchanged, or rerun it if that is simpler. A screenshot alone does not show that entering an amount and clicking the action produced the visible state.

Change the input. The browser journey passes, but the user cannot copy the answer. The acceptance decision depends on the stated task contract. If copy is a required part of the evidence workflow, the feature remains incomplete. Record the failing interaction and repair it. Passing numeric checks do not offset the failure of a required usability step.

When a check cannot run, record it as “unverified” with the reason and an owner. That is more useful than an unqualified completion claim, and it gives the next worker a precise starting point.

Extending the journey: history that survives a reload

The supplied workbench keeps answers only in the current page. To extend it, define a history record with an immutable request ID, submitted question, answer, source references, completion timestamp and optional parent request ID. Capture the question before awaiting the response. A follow-up should point to the prior answer the user selected, not to whichever answer happened to finish last.

Exercise: design storage and verify two responses on opposite sides of midnight, a reload, a changed policy and a storage failure. Use synthetic questions only. State whether date groups follow UTC or the viewer's time zone.

Worked solution: persist the record after a successful response, render stored text as text, and preserve citation identity through reload. Group using the declared time zone rather than slicing an arbitrary local timestamp. Show the answer's historical policy revision; revalidate access and freshness before using it as evidence for a new action. On storage failure, retain the visible answer and report that history was not saved. A saved answer is not a fresh permission grant.

A minimal persistence check runs without a browser:

import json
import sqlite3
import tempfile
from pathlib import Path
from datetime import datetime, timezone

with tempfile.TemporaryDirectory() as folder:
    path = Path(folder) / "history.db"
    db = sqlite3.connect(path)
    db.execute("CREATE TABLE history(id TEXT PRIMARY KEY, parent TEXT, question TEXT, answer TEXT, sources TEXT, completed TEXT)")
    db.execute("INSERT INTO history VALUES(?,?,?,?,?,?)", (
        "Q-1", None, "When is review required?", "At least 100000 cents.",
        json.dumps([{"id": "P-4", "revision": 4}]), "2026-10-17T23:59:59+00:00"))
    db.execute("INSERT INTO history VALUES(?,?,?,?,?,?)", (
        "Q-2", "Q-1", "Does that include equality?", "Yes, in revision 4.",
        json.dumps([{"id": "P-4", "revision": 4}]), "2026-10-18T00:00:01+00:00"))
    db.commit()
    db.close()
    reopened = sqlite3.connect(path)
    rows = reopened.execute("SELECT id,parent,sources,completed FROM history ORDER BY completed,id").fetchall()
    reopened.close()
    days = [datetime.fromisoformat(row[3]).astimezone(timezone.utc).date().isoformat() for row in rows]
    assert days == ["2026-10-17", "2026-10-18"]
    assert rows[1][1] == rows[0][0]
    assert json.loads(rows[1][2])[0]["revision"] == 4

Change the input: policy P-4 advances to revision 5 after Q-1. Preserve Q-1 as a historical answer, but fetch and verify revision 5 before answering a new operational question. Check the browser's reload, citation, copy and keyboard flows separately; a database assertion does not establish that the UI works.

Go deeper

• Evaluation gates: Connect task-level proof to the release conditions the customer accepts.
• The HTTP 200 lab: Practice distinguishing transport success from the intended user-visible outcome.
• Verifier gaming question: Challenge checks that can pass while the application still fails its purpose.

Key takeaways

• Use complementary checks for calculation, service wiring and user interaction.
• Identify the exact candidate and scope behind every passing result.
• Keep missing evidence and required usability failures visible in acceptance.