production ml

Models don't crash, they decay quietly while dashboards stay green. The four-layer monitoring stack, data vs concept drift with PSI numbers, and the label-lag problem that makes naive accuracy monitoring useless.

The most expensive bug class in production ML: two implementations of 'the same' feature that quietly disagree. The diagnostic that finds it in an afternoon, and the architecture that makes it impossible.

The signature production-ML incident, and a pure FDE test: a ranked differential diagnosis run live on a call, because 'nothing changed' is always false, and one hypothesis explains half of all cases.

"It depends" is correct and unhelpful. The decay-curve method that replaces guesswork with measurement, trigger-based vs scheduled retraining, and the safety rails without which auto-retraining is an outage generator.

Offline wins don't justify big-bang swaps, models fail in ways staging never shows. The shadow → canary → ramp playbook, what to compare at each stage, and the rollback discipline that keeps customer trust.

The capstone panel question that ties the whole track together. The spine, data, training, deployment, monitoring, operating model, plus the maturity-staging judgment that separates architects from slide-readers.

Almost no labels means you cannot train a classifier and you cannot honestly report recall. The senior answer is to score by typicality, threshold by the customer's review budget, and build the labeling flywheel that turns this into a supervised problem within two quarters.

An incident question graded on method, not tools: interviewers want ordered hypotheses, each with a five-minute test. The version-skew check most candidates never think of, the cache trap, and the triage tree that finds the usual culprit fast.

An ambiguous, time-pressured staff scenario: the signal is soft, labels won't confirm anything for weeks, and rollback itself has risks. The decision framework under uncertainty, when to revert on a proxy, and the remediation for decisions the bad model already made.

The staff on-call simulation: three alarms at once, partial information, and pressure to do something. How to separate symptom from cause, the common-cause hypothesis that explains all three, and why mitigating before diagnosing is the senior move, not a shortcut.