The validation set is the one you tune and select on; the test set is the one you only look at once. Confusing them, or tuning so hard you overfit the validation set, is how a model that aced offline metrics dies in production.
Updated Aug 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.
The validation set is the one you tune and select on; the test set is the one you only look at once. Confusing them, or tuning so hard you overfit the validation set, is how a model that aced offline metrics dies in production.
The screen is whether the candidate can distinguish three roles (train fits parameters, validation tunes hyperparameters and selects models, test gives the final unbiased estimate) AND name the two failure modes: leakage (validation info bleeding into training, e.g. fitting the scaler on all data before splitting) and overfitting-to-validation (tuning so many choices against val that val stops being honest). Strong candidates mention a held-out test touched once, and cross-validation when data is scarce. The reserved follow-up is 'how do you know you overfit the val set': a gap between val and test, which is why you keep test untouched.
No comments yet — be the first to share your approach.
