TL;DR: Describe a five-layer test pyramid: data contracts and validation, code unit tests on transforms, model quality gates (champion/challenger on two eval sets with segment floors and behavioral checks), serving-container tests with a training-serving parity check, and production validation via shadow then canary. Every gate has a number and a reason; "we checked accuracy" is the answer this question filters out.
How to approach it
Asked as an experience probe, but graded as a design question: the interviewer wants a layered testing strategy, not an anecdote. The trap answer is "we evaluated the model on a test set", that's one layer of five. Walk the layers from data to production, naming what each catches that the others can't.
A strong answer
Frame it as a test pyramid for ML, then walk it bottom-up:
Each layer catches a failure class the ones below it structurally cannot see, and they are ordered by cost so the cheap ones run on every change.
Each layer catches a failure class the others cannot; the bottom layers are cheapest and run most often.
Layer 1, data contracts and validation. Schema, types, null rates, value ranges, and distribution checks on every input batch, Great Expectations or dbt tests at ingestion, plus a contract with upstream producers so a renamed column fails their CI instead of silently training a bad model in yours. This layer exists because in ML, bad data doesn't crash the build; it trains a model that looks fine and isn't. Concrete gate: "null rate on transaction_amount under 0.5%, categorical cardinality within 2x of baseline, or the pipeline halts before training spends four GPU-hours."
Layer 2, code unit tests. Feature transforms tested like any function: known input, expected output, plus the edge cases that actually bite, empty windows, a brand-new entity with no history, timezone boundaries, an idempotent re-run producing identical values. Fast, deterministic, runs on every PR.
Layer 3, model quality gates. Challenger versus champion on a fixed golden holdout and the most recent time slice (the golden set catches regressions, the recent slice catches staleness), broken down by key segments, not just the global number, a model that's +1 point overall and −6 on your largest customer segment fails. Add behavioral tests: invariance (flipping a name shouldn't move a credit score) and directional expectations (more late payments shouldn't raise it). These catch logic-level wrongness that aggregate AUC hides.
Behavioral tests are the layer people nod at and never write, so here are three at full concreteness for a credit-risk model, each a real assertion with a real failure story behind its class:
| Test | Concretely | What a failure means |
|---|---|---|
| Invariance | Score the same application twice, changing only the applicant's name from one common name to another; assert the scores are identical to the float | The model found a proxy for a protected attribute; this is a launch blocker and possibly a legal one |
| Directional | Take a real application, add two more late payments, assert the risk score does not decrease | The model learned something absurd from a correlation (perhaps late payers in the data skew toward a low-risk segment); aggregate AUC will never surface it |
| Segment floor | Challenger AUC on the enterprise segment must be within 0.5 points of champion, regardless of the global number | The +1-overall model quietly traded away your biggest customer to win points on the long tail |
| Sanity anchor | A hand-built obviously-safe application scores low risk; an obviously-degenerate one scores high | Catches sign flips, label inversions, and scaler bugs that produce a confidently backward model |
Each row runs in seconds against the candidate model, each is deterministic (fixed inputs, no statistics to be flaky), and each catches a category of wrongness that no amount of holdout accuracy can see, which is the argument for the layer in one table.
Layer 4, serving tests. The actual container, not the notebook model: golden request/response pairs, malformed-input handling, p99 latency under representative load, memory ceiling, and a training-versus-serving prediction parity check, the same row scored through both paths must agree to within float tolerance, which is your cheapest skew detector.
Layer 5, production validation. Shadow deployment on live traffic for one to two weeks before any real decisions, then canary with guardrail metrics and automatic rollback. Tests can't enumerate production; shadow mode samples it.
Then make it real with the experience part: which layer you added last and why. A strong version: "We had layers 2 and 3 from the start; we added data contracts after an upstream team changed a units convention and we shipped a quietly miscalibrated model, that incident is why I now put layer 1 first."
What interviewers probe next
- "How do you stop flaky statistical tests from blocking every release?", fixed seeds and fixed eval sets for gating; thresholds with tolerance bands chosen from historical variance, not vibes; only deterministic checks are hard-blocking, distribution checks page a human.
- "Do you retrain in CI?", no: CI runs a smoke-train on a small sample to validate the code path in minutes; full training belongs to the training pipeline, whose output is then gated.
- "What's a data contract concretely?", a versioned schema-plus-semantics agreement (types, nullability, units, freshness SLA) enforced by checks in the producer's pipeline, with expand-contract migration when it changes.
Common mistakes
Only describing offline evaluation, the single most common failure on this question. No segment-level gates, so aggregate metrics hide harm. Testing the model object but never the serving container, then meeting skew in production. And describing thresholds nobody chose ("we check the accuracy is good"), every gate worth having has a number and a reason attached.