TL;DR: Reproducibility means joining five versioned things to one model: code SHA, data snapshot identifier, config and seeds, Docker image digest, and infrastructure. The registry ties them via lineage. Define the guarantee in two tiers, statistical reproducibility within a tolerance band as standard and bit-exactness only where a regulator demands it, because bit-exact GPU reproducibility is mostly a myth.
How to approach it
Asked everywhere from AWS loops to bank model-risk rounds, and the grading splits on one thing: whether you treat reproducibility as a checklist of tools or as a guarantee with a precise scope. Open with the operational test, "could I rebuild the model we shipped six months ago and explain any difference?", then enumerate what has to be pinned to make that true.
A strong answer
Reproducibility means joining five versioned things to one model. Miss any one and the chain breaks:
Code, the git SHA of training and feature code, recorded on the model, not in a wiki. Data, the hard one: a snapshot identifier, via Delta Lake time travel, DVC, or immutable dated partitions. "We trained on the customers table" is not reproducible; "version 1142 of the Delta table" is. Configuration, every hyperparameter and seed, logged automatically by the training harness (MLflow autolog plus explicit logging of anything custom), because hand-maintained config docs are fiction within a month. Environment, the Docker image digest, not the tag; training:latest is a different image every quarter, and a CUDA or library minor-version bump changes numerics. Infrastructure, Terraform for the training setup where hardware matters, since distributed training results depend on world size and instance type.
The registry is what ties these together: a model version that links to the run, which links to the code SHA, data version, params, and image digest. That's the lineage an auditor walks.
Break any one link and the rebuild becomes a guess; the auditor walks this graph from the model backward.
The floating-point claim underneath the whole "bit-exact is a myth" position is checkable in three lines, and having run it inoculates you against anyone who says "just set the seed." Sum the same one million float32 values forward and backward: we got -208.99817 and -208.99823, a difference of 6.1e-5 from the same numbers in a different order, because float addition is not associative and every ordering accumulates different rounding. A GPU reduction across thousands of threads has no fixed order unless you force one, so two "identical" runs differ at the least significant bits, those bits feed gradients, and gradients compound over a million steps into weights that are legitimately different models. The seed controls which numbers you add; it has no authority over the order the hardware adds them in. That is the mechanism behind the two-tier guarantee, and stating it as arithmetic rather than folklore is what makes the tolerance-band position sound engineered instead of resigned.
Then the admission that reads senior rather than weak: bit-exact reproducibility on GPUs is mostly not achievable at reasonable cost. Floating-point reduction order is nondeterministic, cuDNN picks algorithms at runtime, and data-loader parallelism reorders samples; you can force determinism (torch.use_deterministic_algorithms, pinned cuDNN settings) at a real throughput penalty, often 10–20%, and distributed runs still leak nondeterminism. So define the guarantee in two tiers: statistical reproducibility (rerun lands within an agreed tolerance band on the evaluation suite, say ±0.2 AUC points) as the standard bar, and bit-exactness only where a regulator or a debugging session demands it. Candidates who claim their pipeline is fully deterministic get one follow-up question and then a long silence.
Finally, the part most answers skip: reproducibility is enforced, not aspired to. The pipeline refuses to register a model whose run is missing a data version or image digest, and a periodic job actually re-trains a sampled model and diffs the eval results, a reproducibility guarantee nobody has exercised is a guess.
What interviewers probe next
- "Why does this matter in a bank?", model risk management (SR 11-7 in the US): you must demonstrate how any production decision-making model was built; "we can't reconstruct it" is a finding, potentially a blocked model.
- "Your rerun gives AUC 0.812 instead of 0.815, problem?", within the pre-agreed tolerance band, no; the senior move is having that band defined before anyone reruns anything.
- "How do you version a 2TB training set without copying it?", snapshot by reference: Delta/Iceberg time travel or manifest files listing immutable partitions; copy-per-experiment dies at scale.
- "What breaks reproducibility most often in practice?", untracked preprocessing in notebooks upstream of the pipeline, and
latestimage tags. Both are process failures, not tooling gaps.
Common mistakes
Reciting "Git, DVC, Docker, MLflow" as a tool list with no notion of what guarantee they compose into. Claiming bit-exact reproducibility, the follow-up about GPU nondeterminism is being held in reserve specifically for this. Versioning data by table name instead of snapshot. And forgetting the feature pipeline: reproducing the trainer while the features were computed by code that no longer exists reproduces nothing.