Sampling the Judge in Production: Rate, Threshold and Time to Detect

In production, a validated LLM judge scores a sample of live conversations and an alert fires when its FAIL rate crosses a threshold. The sampling rate sets the noise, the threshold sets false alerts against missed regressions, and the window sets detection delay. Choose them together from the judge's baseline rate and the shift you need to catch, and remember the judge's FAIL rate is not the true failure rate.