Your support agent handles 50,000 conversations a day. How much of that traffic should an LLM judge score, and when should it page someone?

'Score 5% and alert on a spike' sounds reasonable and is a guess. The real answer comes from three numbers you can get: the judge's measured error rates, the regression worth paging for, and how fast you need to know. How do you turn them into a sampling rate and a threshold?

Updated Nov 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.