Three failure modes are worth finding before you design anything

TL;DR: Look for three things in any system you are about to build on: whether a re-run does damage, whether a slow dependency is handled differently from a dead one, and what happens when a field changes shape. Each has a standard fix, and each is cheap now and expensive later.

Where you are. Last lesson of module three. You can orient and you can see the boundaries. This is the short list of things to actively go looking for, because they decide how much of your time the deployment will consume after it ships, and the next module is about writing code that handles all three.

Why these three

Production surprises in customer environments cluster. Not because those systems are badly built, but because these three failures share a property: they are invisible in testing and obvious in hindsight. Each one behaves perfectly on a laptop with clean data and a fast network, which is, not coincidentally, exactly where the estimate was made.

Sun Microsystems catalogued the general version of this in the 1990s as the fallacies of distributed computing, a list of things every engineer assumes without noticing: the network is reliable, latency is zero, bandwidth is infinite, the topology never changes. The list has aged well because the assumptions are not stupid, they are simply true on a laptop and false across a customer's estate. The three failures below are the ones from that family that cost forward deployed engagements the most time, which is why they get a lesson rather than a footnote.

One: the work happens twice

Somewhere in the system, something will run again. A job retries after a timeout, an operator reruns yesterday's load because the first attempt looked wrong, a queue delivers the same message twice because that is what queues do.

The question to ask of any write path is plain: if this runs twice with the same input, what is different afterwards? If the answer is "an extra row" or "the customer is charged again", you have found the most common serious defect in enterprise integrations.

The fix is a stable key derived from the input, checked before the write. The important part is choosing a key that means the same thing on both runs, which is a design decision rather than a library call. A timestamp is not a key. The source system's record identifier usually is.

Two: the dependency gets slow rather than dying

Engineers handle the dead dependency. The connection is refused, an exception is raised, the error path runs and someone gets paged. This case is usually fine.

The dangerous case is the dependency that answers in forty seconds instead of two hundred milliseconds. Nothing raises an exception. Requests queue behind it, the pool fills, threads that had nothing to do with that dependency start waiting, and a service that was ninety-nine percent healthy stops responding entirely. From the outside it looks like your service failed, and in the customer's environment you will not have the access to prove otherwise quickly.

So look for two things: is there a timeout on every outbound call, and is there anything that stops calling after repeated failures. In an unfamiliar codebase, missing timeouts are common, and they are the single highest-value thing to fix before you add load.

Three: the data changes shape

An upstream team adds a field, changes a nullable column, starts sending dates in a different format, or renames a value from ACTIVE to Active. Nobody tells you, because from their side it was a small, correct change.

This one is nastier than the other two because it usually fails silently. Nothing throws. The pipeline runs green and the output is quietly wrong, and the discovery happens weeks later when somebody notices the numbers do not match.

The fix is to validate structure at the boundary and refuse to guess. A record that does not match the contract goes to a quarantine table with the reason attached, and somebody is told. Silent coercion is what turns a data problem into a trust problem, because by the time it surfaces, the customer no longer knows which of your outputs to believe.

Do this before moving on

Take the trace from the last two lessons. For each write in it, answer the twice question. For each outbound call, check for a timeout. For each input, ask what happens if a field arrives in a shape you did not expect. You will usually find at least one of the three unhandled, and knowing which one before you start building tells you where your first week actually goes.

Go deeper

• Idempotency is the first failure mode and its standard fix in depth.
• Circuit breaker covers what to do once you have timeouts and still have a degrading dependency.
• Exponential backoff is the retry behaviour that makes the twice problem common in the first place.
• Data quality is the discipline behind validating at the boundary rather than coercing.
• At-least-once versus exactly-once delivery is the interview question that starts from the same place this lesson does.

Key takeaways

• Three failures dominate in customer environments, and all three look fine on a laptop with clean data.
• For every write path, ask what is different if it runs twice with the same input.
• A slow dependency is more dangerous than a dead one, because nothing raises an exception while your service stalls.
• Validate shape at the boundary and quarantine what does not match; silent coercion turns a data problem into a trust problem.

Check yourself

Answer before you look. Recalling it is what makes it stick; recognising it does not.

1. 1Why is a dependency that responds in forty seconds more dangerous than one that is completely down?

   Answer it, then reveal

2. 2An upstream team changes a status value from ACTIVE to Active without telling anyone. What is the failure mode and the correct handling?

   Loud failure; add a retry so it recoversSilent wrongness; validate shape at the boundary, quarantine mismatches and alert rather than coercingNo real failure; normalise case on read and move on

3. 3What is the question to ask of every write path in an unfamiliar system, and what makes a good idempotency key?

   Answer it, then reveal