Test replay, slow dependencies and changing data before adding load

This is lesson nine of Foundations of Forward Deployed Engineering. Inspect what happens when work repeats, when dependencies consume time, and when data violates its contract. Test the relevant failure and recovery paths, because a key, a timeout or a validator is useful only with the surrounding semantics it needs.

You have a trace and a boundary register. Use them to choose failure tests, before the next module develops the production code at those boundaries.

Start by choosing the failures an ordinary demonstration misses. Replay, slow dependencies and changed data are useful investigation categories.

They aren't an exhaustive list, or a measured ranking of every customer's incidents. And they can be tested with controlled fixtures. They aren't inherently invisible until production.

Start with the actual write paths, resource limits and input contracts, because a successful request under clean data and low load leaves these questions unanswered. So ask what evidence would establish the behavior under the failure, including what the operator sees and how recovery works.

Repeated delivery and repeated intent are different. A queue may redeliver under its delivery contract, an operator may rerun a batch, and a caller may retry after losing a response.

The intended business operation can be the same, even when the transport attempt is new. Conversely, two legitimate operations may use identical payloads.

So choose identity for the logical operation, with the appropriate tenant and source scope, and payload consistency. A shipment ID alone can't distinguish tagging a shipment from sending an update, or successive valid events for that shipment. A timestamp generated anew on each retry defeats deduplication, while a timestamp that's part of a source's genuine unique event identity may be meaningful. So don't reduce key design to "timestamps bad, record IDs good."

Checking a key before a write is insufficient if the check and the effect aren't coordinated. The lesson runs a deliberately broken, single-threaded simulation of two interleaved callers. Both callers check the set before either commits. Caller one sees no key, applies the tag and remembers the key. Caller two already decided to write, so it applies the tag too.

It prints two business effects and one remembered key. The set contains one key, while two effects occurred. That's the race. It demonstrates the race without requiring threads or touching a real service.

A durable unique operation record, coordinated atomically with a local state change, can address this class of race. Here's the mechanism. A unique constraint makes the second insert fail inside the same transaction as the effect, so the check and the write can no longer be separated in time. The database refuses the duplicate at the moment of writing, rather than at the moment of looking.

Remote effects need the receiver's idempotency contract, or explicit reconciliation. A local database transaction can't roll back an email already sent elsewhere.

Test duplicate delivery, concurrent attempts, a crash before and after the effect, conflicting payloads under one key, changed authorization, and expired deduplication retention, then record the operation contract and the failure evidence you still need, since later lessons give the fuller implementation exercises.

Now bound the time and the work. A dependency can fail quickly, slowly, or never return. A "dead" endpoint may also hang until a network timeout.

The dangerous behavior is consuming shared resources beyond what the system can tolerate, regardless of the label, so inspect connection, request and overall deadlines, including what the client actually covers. A timeout limits waiting. It doesn't prove remote cancellation, or prevent all accumulated work.

Bound concurrency, queue size and retries, decide how overload is rejected or deferred, and where dependencies share a pool, protect unrelated work.

An AWS article on timeouts and retries documents an instructive case, where a low timeout encountered secure-connection setup after deployments, while reused connections usually succeeded. The team first adjusted the handling, and later established connections before taking traffic. The lesson is to inspect the work included in a timeout, rather than copying a number from another service.

Now a separate synthetic deadline calculation. The request has a 1,000 millisecond total budget. It has 100 milliseconds of local work, and three permitted attempts, each bounded at 250. The two backoff waits are 50 and 100 milliseconds. So 100, plus 750 of attempts, plus 150 of backoff, is exactly 1,000. These are illustrative allocations, not recommended defaults.

Now suppose local work actually consumes 300 milliseconds, and the first attempt uses its full 250. Only 450 milliseconds remain. After the 50 millisecond backoff, at most 400 remain, for all the remaining work.

So an implementation must compute the remaining monotonic deadline before each wait and each attempt, leaving room for the required response handling. It must not launch the entire original retry schedule after local work overruns. Actual timeout coverage and cancellation behavior require testing. The calculation is only the plan.

Retries at multiple layers can multiply load. Suppose three layers each make up to three total attempts: that's three attempts at the top, each able to trigger three in the middle, each able to trigger three at the bottom. So one top-level request can cause up to 27 deepest calls, when all the retries are exercised.

So coordinate retry ownership, bound the attempts, and use an appropriate backoff and jitter policy. A retry is also inappropriate when the operation's outcome is unknown, and no valid repeat-safe contract protects it.

Next, the circuit-breaker model, with its limits. It has three states: closed, open and half-open.

Closed passes calls through. Open blocks them after enough failures. And half-open lets a few probe calls through, to test whether the dependency has recovered. In the lesson's widget, trigger failures, observe calls being blocked, then watch limited probing before normal calls resume.

A breaker can reduce calls to a failing dependency, but it doesn't automatically cancel in-flight work, bound every queue, isolate all resources, or establish business-effect safety. Its thresholds, scope, probe behavior and recovery interaction need tests. It's one possible control, not a mandatory second step after every timeout.

Last, validate the agreed meaning, not only the shape. An added field may be backward-compatible under one contract, and forbidden under another.

A status changing from all-caps active to capitalized active may be valid, if the documented contract is case-insensitive. If the contract requires exact enum values, treating it as active without review is an unsupported assumption.

Write the accepted representations and the canonical result explicitly, preserve the original input and transformation provenance where required, and don't silently convert unknown values into a plausible default. And a structurally valid value can still be semantically wrong, such as a valid-looking customer ID that belongs to another tenant.

For this small exercise, the agreed input contract accepts exactly two spellings, all-caps and capitalized, as aliases for the canonical lowercase active. All-caps inactive becomes lowercase inactive. Other values are rejected. It's a supplied contract, not a general recommendation to accept mixed case.

Look at the lowercase string. It's rejected, because this particular input contract didn't include it, even though it equals the canonical output. If that representation should be accepted, change the contract and the tests deliberately. Don't infer the input language from the output format.

Quarantine needs a reason, an owner, a retention and access policy, and a replay path. Some invalid records can be isolated, while some failures require holding the whole batch, or the affected workflow. Alert on actionable conditions without copying sensitive input into broad logs. And reconcile received, accepted, rejected and unresolved work, so failures can't disappear from the count.

The lesson runs all three families as one test sequence, because they share a starting point: the real write paths, limits and contracts, rather than the demonstration path.

Your turn. Pause the video here. Run all three fixtures. Explain why one remembered key didn't prevent two effects, calculate 27 deepest attempts from the three-layer plan, then change the status contract to accept lowercase explicitly, and add its test. For your trace, write one failure test and one recovery check for each category, including a lost response after a write, a slow call that consumes the remaining deadline, and a valid-looking value with the wrong domain meaning. State which controls belong to the producer, the consumer and the operating owner. And remember that finding no defect in a small fixture isn't evidence that all three categories are handled in the real deployment.

To recap. Identify logical operations and coordinate deduplication with effects, because a prior key check can race, and remember that timeouts limit waiting, while concurrency, queues and retry budgets bound accumulated work. Circuit breakers need explicit scope and recovery tests, not automatic adoption, and you accept normalization only under a defined contract, and reject unknown meanings visibly. And pair every failure test with reconciliation and a usable recovery path.

That's lesson nine. Next in the series, translate source data without losing its meaning. The full lesson, with its exercise and drill, is on fdeinterviews.com.