You have two days, no context, and everybody has a different theory

This is lesson 17 of Hard Deployments. Contain active harm when needed and then build an assessment that separates observed facts, tested mechanisms and unresolved explanations. Agree a near-term update you can deliver and make repair forecasts conditional on evidence, access and the customer's change process.

You're joining an engagement that already has software, history and disappointed expectations and the task is to establish its current state and a feasible next decision, not to prove that the previous team was wrong.

Part one: diagnose the workflow and the working relationship. This is a fictional freight engagement where a five-month-old deployment has not reached its intended operating scope.

Operators still resolve many exceptions manually, the original engineer now supports another account, and the sponsor wants a decision about continuing. Those facts describe a situation to investigate. They do not establish the cause or whether the software is salvageable.

Technical performance, workflow fit and confidence in delivery can all need attention. They influence one another but each needs evidence. A working parser does not establish that users have access, that the workflow is useful or that the customer accepts the remaining support burden.

For system behavior, seek versioned inputs and outputs, failures, reproduction and current configuration and avoid inferring that the loudest complaint must name the mechanism. For workflow fit, seek work attempted, assistance and unresolved cases; a successful demo is no proof that routine work is covered.

For delivery confidence, seek missed and kept commitments, unanswered decisions and stated concerns; a missed meeting or a delegated attendee does not prove a loss of trust. For feasibility, seek data, access, owners, change windows and remaining effort and avoid assuming the problem must be small because a new engineer found it.

Ask the sponsor what decision they need and how progress will be judged. Ask operators to show examples of unfinished work and ask the technical owners which access and recovery controls are available. A delegate may be the right decision-maker; establish their authority instead of diagnosing the relationship from seniority or attendance.

Part two: containment can come before a complete diagnosis. If the system is sending incorrect notifications, corrupting data or exposing private information, waiting two days to finish an assessment can increase harm.

Use the authorized incident process to reduce exposure: pause the affected action, route work to a staffed fallback or apply a verified reversible mitigation. Preserve useful evidence when feasible without making evidence collection a prerequisite for urgent containment.

Google's incident-response chapter describes stopping impact before completing root-cause investigation. It also shows why mitigation and understanding the underlying fault are separate tasks. Apply the principle to the actual risk and available controls; a rollback can itself be unsafe when state has changed.

A well-understood fix can be appropriate on arrival. Record the trigger, evidence, scope, validation, owner and recovery plan. An impressive-looking patch with no reproduction or acceptance test is no substitute for diagnosis. The clock alone decides neither case.

Part three: use the first two days as an assessment cadence. The two-day outline is a planning example.

On day one listen separately in the morning to the sponsor, operators, their IT and your own team; in the afternoon verify evidence: logs, data, deploy history and the code. On day two form a hypothesis and test it in the morning then write and deliver the assessment with confidence levels. At any point, contain active harm and verify the result.

Access restrictions, incident urgency or source retention can change the order. If a required export will arrive later, deliver a partial assessment on the agreed date and say what cannot yet be established. Do not promise an exact diagnosis date that depends on unavailable evidence.

Speak with the sponsor, operators, customer technical owners and your own delivery team. Separate conversations can make it easier to hear constraints while a joint review can resolve contradictory definitions. Neither format guarantees candor. Ask for the workflow, expected behavior, a recent concrete failure and changes around its onset.

Keep a small hypothesis register. Choose the next test by how much it can distinguish plausible explanations at acceptable risk and cost. One active experiment can keep the work focused; a rule allowing only one hypothesis encourages tunnel vision.

Part four: grade the supplied evidence packet. D1 to D5 are complete synthetic UTC days, not dates from a real customer incident.

In E1, D2 has twice the inputs and twice the failures of D1 so absolute failures doubled while both rates remain 2 percent. In E2, source format S2 begins at D3 so D3's 22 percent failure rate coincides with the recorded format change. In E3, application B begins at D4 while D3 still used A, so B's recorded deployment is later than this observed rate increase.

In E4, application A, held fixed, accepts a string region and rejects the same code nested in an object: a reproduced parser-shape mechanism on a paired record. E5 breaks down the D3 to D5 failures which supports a conditional 500 of 550 recovery target for a shape repair. In E6, no destination-side action ledger has been provided so replay safety for external effects remains unestablished.

E2 supplies a correlation: two things changed on the same day and the packet cannot say which caused which. E4 demonstrates one mechanism because it holds everything else constant and changes only the shape of one input which is what separates a discriminating test from a coincidence in a timeline. E5 predicts a bounded recovery opportunity only if the full selected population matches that breakdown and passes replay. And every item falls short of establishing that it is safe to resend every failed-looking record into production.

Practice with three statements. Application B caused the onset? The supplied timeline contradicts that narrow onset claim assuming the deployment records are complete and correctly aligned. B could still introduce another problem later. The model needs replacement? Not established by this packet. Test the parser path and retain any independent model-quality concern as a separate hypothesis.

And the compatible parser will fix 95.5 percent of failures? Unsupported. Unknown-region labels cover 525 of 550 failures, but only 500 are the supplied shape-compatible subset, which is about 90.9 percent of failures.

A confidence label should describe its basis. "Reproduced on one paired input; population recovery pending full replay" is more useful than an unexplained "90 percent confident." It also tells the next engineer what evidence would strengthen or weaken the claim.

Part five: choose tests that can change the decision. To test whether the S2 shape triggers parser rejection, keep the same logical input and fixed app and change only the region representation. If both shapes pass investigate the production path and configuration.

Does load explain the failure increase? Look at arrival bursts, service times and resource saturation on aligned windows; a repeatable load failure not explained by input shape adds another mechanism. Did model quality cause unresolved cases? Use paired reviewed outputs after successful ingestion. Wrong answers among correctly parsed cases justify a separate model investigation.

And has failed-looking work already caused external effects? Look for destination receipts keyed to the selected records and actions; applied or unknown effects change the replay plan. Do the safe local replay before changing a production parser.

Ask for the highest-value missing evidence with a precise scope and purpose. A request for all logs from the last six months can be slow, expensive and unnecessarily sensitive; start with the failing interval and representative identifiers, then expand as justified.

Part six: write an assessment that travels accurately. The lesson's example memo is titled: parser mechanism confirmed, full recovery still conditional.

Current impact: the D3 failure rate is 22 percent of distinct received records compared with 2 percent on D1 and D2 and the D3 to D5 set has 550 failed records out of 2,500 inputs. Established mechanism: application A, in a paired local replay, rejects the new nested region representation and the recorded application B deployment occurs later than the observed onset.

Proposed next step: confirm the source contract and replay all 550 selected failures in an isolated output store with external actions denied; the breakdown predicts 500 shape-compatible recoveries and 50 residual exceptions; record the result rather than promising it in advance. The unresolved boundary: no destination action receipts yet, no peak-window behavior and no evidence for other source versions.

Production replay waits for effect reconciliation and the approved change process; an unknown external effect is a recovery constraint and gives no permission to try the action again. The next commitment: deliver the isolated replay report at the agreed checkpoint conditional on receipt of the selected export. If access is delayed deliver the completed checks and the missing-evidence list at that checkpoint.

Report any containment and repairs already performed. Keep the business consequence visible: affected work, manual fallback capacity and outstanding decisions. A technically precise memo that never says what operators can do next leaves the rescue incomplete.

Part seven: repair commitments without inventing a relationship diagnosis. Agree the update cadence and audience.

Daily notes can help during active recovery but send useful changes, decisions and blockers rather than making a universal fourteen-day email rule. Record what you promised, what happened and what changed the forecast.

Discuss past defects with evidence and respect. Blameless reporting does not mean concealing a known error or refusing to explain how it affected the customer; describe contributing conditions, impacts and corrective work, without unsupported judgments about motives or competence.

A small visible improvement is worthwhile when it helps the workflow and fits the risk and effort budget but it cannot justify deferring a serious integrity problem or pretending that a cosmetic fix establishes recovery. Choose something the operator can verify, such as a reconciled unresolved-work queue with accurate reasons.

Part eight: recommend stopping when the evidence supports it. The use case may lack value, required access may be unavailable, or a safe service may cost more than the agreed scope can support.

State the constraint, the alternatives examined and the remaining uncertainty. A narrower option is useful if supported by evidence; do not invent a viable replacement just to make a stop recommendation sound positive. Stopping can itself require data export, retained evidence, outstanding-effect reconciliation and an agreed transition to manual work. The decision belongs with the authorized product, customer and commercial owners; your assessment should make it reviewable.

Your turn. Pause the video here. Write a one-page assessment from E1 to E6 before reading the example memo again. Label observations, inferences, competing explanations and unknowns. Reproduce the figures of 2 percent, 22 percent, 95.5 percent and 90.9 percent with their denominators and explain which one is relevant to the parser recovery target.

Now add E7: an authorized operator confirms that duplicate notifications are still being emitted so add immediate containment and effect reconciliation to the plan; do not wait for the end-of-day assessment. State what would prove the notifications have stopped and what might still be queued remotely.

Finally, change D2 to 440 failures out of 2,000 inputs. The 22 percent rate now precedes the recorded S2 change. Revise the onset claim and request earlier inputs and change evidence. The paired parser defect remains demonstrated but it no longer explains the whole timing story by itself.

To recap. Contain active harm when needed; a two-day assessment cadence is no ban on early verified repairs. Separate system behavior, workflow fit and stated delivery concerns instead of inferring trust from attendance. Keep competing explanations and use tests that can change the next decision and distinguish reproduced mechanisms from population recovery predictions and unknown external effects. And make commitments reviewable, conditional where necessary and explicit about the next owner or decision.

That's lesson 17. Next in the series: proving what happened record by record. The full lesson, with its exercise and drill, is on fdeinterviews.com.