Test replay, slow dependencies and changing data before adding load

TL;DR: Inspect what happens when work repeats, dependencies consume time and data violates its contract. Test the relevant failure and recovery paths; a key, timeout or validator is useful only with the surrounding semantics it needs.

Where you are. You have a trace and boundary register. Use them to choose failure tests before the next module develops the production code at those boundaries.

Failures the ordinary demonstration misses

Replay, slow dependencies and changed data are useful investigation categories, not an exhaustive list or a measured ranking of every customer's incidents. All three can be tested with controlled fixtures, so none of them has to stay invisible until production.

Start with the actual write paths, resource limits and input contracts, because a successful request under clean data and low load leaves these questions unanswered. Ask what evidence would establish the behavior under the failure, including what the operator sees and how recovery works.

Repeated delivery and repeated intent are different

A queue may redeliver under its delivery contract; an operator may rerun a batch; a caller may retry after losing a response. The intended business operation can be the same even when the transport attempt is new, and two legitimate operations may just as well share identical payloads.

Choose identity for the logical operation, with appropriate tenant/source scope and payload consistency. A shipment ID alone cannot distinguish “tag shipment” from “send update” or successive valid events for that shipment. A timestamp generated anew per retry defeats deduplication; a timestamp that is part of a source's genuine unique event identity may be meaningful. Do not reduce key design to “timestamps bad, record IDs good.”

Checking a key before a write is not enough if the check and the effect are not coordinated. Run this deliberately broken, single-threaded simulation of two interleaved callers, which shows the race without threads or a real service.

seen = set()
effects = []
key = ('tenant-a', 'source-x', 'event-7', 'review-tag')

# Both callers check before either commits.
a_should_write = key not in seen
b_should_write = key not in seen
if a_should_write:
    effects.append('tag applied')
    seen.add(key)
if b_should_write:
    effects.append('tag applied')
    seen.add(key)
print('business effects:', len(effects))
print('remembered keys:', len(seen))
assert len(effects) == 2 and len(seen) == 1
# business effects: 2
# remembered keys: 1

The set contains one key, yet two effects occurred. A durable unique operation record coordinated atomically with a local state change can address this class of race. The mechanism is that a unique constraint makes the second insert fail inside the same transaction as the effect, so the check and the write can no longer be separated in time; the database refuses the duplicate at the moment of writing rather than at the moment of looking. Remote effects need the receiver's idempotency contract or explicit reconciliation; a local database transaction cannot roll back an email already sent elsewhere.

Test duplicate delivery, concurrent attempts, a crash before and after the effect, conflicting payloads under one key, changed authorization and expired deduplication retention. The later replay and action lessons have fuller implementation exercises; here, record the operation contract and the failure evidence you still need.

Bounding the time and the work

A dependency can fail quickly, fail slowly or never return, and a “dead” endpoint may hang until a network timeout. The dangerous behavior is consuming shared resources beyond what the system can tolerate, regardless of the label.

Inspect connection, request and overall deadlines, including what the client actually covers. A timeout limits waiting; it neither proves remote cancellation nor prevents all accumulated work. Bound concurrency, queue size and retries, and decide how overload is rejected or deferred. Protect unrelated work where dependencies share a pool.

Marc Brooker's AWS article on timeouts and retries documents an instructive case: a low timeout encountered secure-connection setup after deployments, while reused connections usually succeeded. The team first adjusted the handling and later established connections before taking traffic. What to take from it is to inspect the work included in a timeout rather than copying a number from another service.

Here is a separate synthetic deadline calculation. The request has a 1,000 ms total budget, 100 ms of local work and three permitted attempts, each bounded at 250 ms, with backoff waits of 50 and 100 ms between them. These are illustrative allocations, not recommended defaults.

budget_ms = 1000
local_ms = 100
attempt_limits_ms = [250, 250, 250]
backoff_ms = [50, 100]
planned_ms = local_ms + sum(attempt_limits_ms) + sum(backoff_ms)
print('planned bound:', planned_ms)
assert planned_ms == budget_ms

# If local work actually consumes 300 ms and the first attempt uses 250,
# only 450 ms remain. After 50 ms backoff, at most 400 remain for all work.
remaining_ms = budget_ms - 300 - 250
print('remaining before next backoff:', remaining_ms)
assert remaining_ms == 450
assert remaining_ms - 50 == 400
# planned bound: 1000
# remaining before next backoff: 450

An implementation must compute the remaining monotonic deadline before each wait and attempt, leaving room for required response handling. It must not launch the whole original retry schedule after local work overruns. Actual timeout coverage and cancellation behavior need testing, because the arithmetic is only the plan.

Retries at multiple layers can multiply load. If three layers each make up to three total attempts, one top-level request can cause up to 27 deepest calls when all retries are exercised: three attempts at the top, each able to trigger three in the middle, each able to trigger three at the bottom, 3 × 3 × 3. Coordinate retry ownership, bound attempts and use an appropriate backoff/jitter policy. A retry is also wrong when the operation's outcome is unknown and no valid repeat-safe contract protects it.

The circuit-breaker model, and its limits

The existing widget shows closed, open and half-open behavior: closed passes calls through, open blocks them after enough failures, and half-open lets a few probe calls through to test whether the dependency has recovered. Trigger failures, watch calls being blocked, and then watch limited probing before normal calls resume.

A breaker can reduce calls to a failing dependency, but it does not automatically cancel in-flight work, bound every queue, isolate all resources or establish business-effect safety. Its thresholds, scope, probe behavior and recovery interaction all need tests, and it is one possible control, not a mandatory second step after every timeout.

The agreed meaning, not only the shape

An added field may be backward-compatible under one contract and forbidden under another. A status changing from ACTIVE to Active may be valid if the documented contract is case-insensitive. If the contract requires exact enum values, treating it as active without review is an unsupported assumption.

Write down the accepted representations and the canonical result explicitly, and preserve the original input and transformation provenance where required. Never silently convert unknown values into a plausible default. A structurally valid value can still be semantically wrong, such as a valid-looking customer ID belonging to another tenant.

For this small exercise, the agreed input contract accepts exactly ACTIVE and Active as aliases for the canonical value active; INACTIVE is canonicalized to inactive. Every other value is rejected. This is a supplied contract, not a general recommendation to accept mixed case.

aliases = {'ACTIVE': 'active', 'Active': 'active', 'INACTIVE': 'inactive'}

def canonical_status(value):
    if not isinstance(value, str) or value not in aliases:
        raise ValueError('Unsupported status representation')
    return aliases[value]

assert canonical_status('ACTIVE') == 'active'
assert canonical_status('Active') == 'active'
assert canonical_status('INACTIVE') == 'inactive'
for value in ['active', ' ActIve ', None, 1]:
    try:
        canonical_status(value)
    except ValueError:
        print('rejected:', repr(value))
    else:
        raise AssertionError('Unexpected acceptance')

The lowercase string is rejected because this particular input contract did not include it, even though it equals the canonical output. If that representation should be accepted, change the contract and tests deliberately rather than inferring the input language from the output format.

Quarantine needs a reason, an owner, a retention and access policy and a replay path. Some invalid records can be isolated, while other failures require holding the whole batch or the affected workflow. Alert on actionable conditions without copying sensitive input into broad logs. Reconcile received, accepted, rejected and unresolved work so failures cannot disappear from the count.

The spine above runs all three families as one test sequence, because they share a starting point: the real write paths, limits and contracts rather than the demonstration path.

Do this before moving on

Run all three fixtures. Explain why one remembered key failed to prevent two effects; calculate 27 deepest attempts from the three-layer plan; then change the status contract to accept lowercase explicitly and add its test.

For your trace, write one failure test and one recovery check for each category. Include a lost response after a write, a slow call that consumes the remaining deadline, and a valid-looking value with the wrong domain meaning. State which controls belong to the producer, consumer and operating owner.

Your inventory passes when it identifies exact effects, deadlines and contracts, and distinguishes tested behavior from proposals. Finding no defect in a small fixture is not evidence that all three categories are handled in the real deployment.

Go deeper

• Idempotency develops operation identity and repeat-safe effects in depth.
• Circuit breaker covers what to do once you have timeouts and still have a degrading dependency.
• Exponential backoff is the retry behavior that makes the twice problem common in the first place.
• Data quality is the discipline behind validating at the boundary rather than coercing.
• At-least-once versus exactly-once delivery is the interview question that starts from the same place this lesson does.

Key takeaways

• Identify logical operations and coordinate deduplication with effects; a prior key check can race.
• Timeouts limit waiting, while concurrency, queues and retry budgets bound accumulated work.
• Circuit breakers need explicit scope and recovery tests rather than automatic adoption.
• Accept normalization only under a defined contract and reject unknown meanings visibly.
• Pair every failure test with reconciliation and a usable recovery path.