calibration

Position bias, verbosity bias, self-preference, and the agreement-rate workflow that turns a sloppy judge into eval infrastructure you can defend. AI labs probe this hard; here's the calibrated answer.

The cheap swap is not a model swap. The prompt becomes atomic typed questions, the parsed label becomes a distribution with a confidence, and the release gate becomes a per-version threshold sweep on the customer's own labeled tickets.

Almost never, and the gap between score and probability quietly breaks thresholds, expected-value decisions, and customer trust. What calibration is, how to measure it, and the two fixes that work.

When a model auto-approves loans or routes patients, the probability is the product, not a dashboard number. The calibration-plus-abstention design that lets a customer trust an automated decision, and the subgroup trap that gets it pulled in audit.

Anyone can train a pCTR model. The hard part is proving a change is good before it touches revenue, when the model sits inside an auction, the logs only show ads that won, and a 1% calibration error is real money. This is an eval question, not a model question.

Watch time is the label everyone optimizes and almost nobody measures cleanly. You only observe minutes for videos people chose to play, the distribution is brutally skewed, and the position they saw it in moved the number. The interview is whether you can predict a biased label honestly.

This model's output is not a ranking, it is a price input. A miscalibrated CTR makes the auction bid wrong, so you overpay or underdeliver. Add conversions that land days after the click and advertisers with zero history, and calibration plus delayed feedback become the whole interview.