monitoring

Reported as the question OpenAI FDE interviewers push hardest on, and the one where 'we look at the outputs' ends loops. The three-legged answer that passes: automated metrics, calibrated human evaluation, and feedback loops wired to a business number.

Golden sets catch what you anticipated. The harder problem is the live traffic you never labeled. The answer that scores layers cheap proxy signals against a small anchored golden set and knows what each can and cannot tell you.

Models don't crash, they decay quietly while dashboards stay green. The four-layer monitoring stack, data vs concept drift with PSI numbers, and the label-lag problem that makes naive accuracy monitoring useless.

Standard observability tells you the request returned 200 in 800ms. It cannot tell you the answer was wrong. The strong answer names the new failure plane LLMs introduce, and takes a real position on the prompts-and-PII question.

Ten X-ray studies a second, a 2-second budget, and a missed cancer is unacceptable. The hard parts are recall-first thresholding, a monitor that watches the radiologist, and surviving a new scanner.

Fifty milliseconds at p99 to catch toxicity, PII, and injection on every message. An LLM in the hot path blows the budget. The strong answer tiers the checks and keeps the LLM off the critical path.

The bug class with no stack trace: the model is fine, the pipelines are green, and production quietly underperforms offline by five points. Why skew survives even feature-store adoption, and the log-and-wait pattern Google-style answers center on.

Everyone names PSI and KS; almost nobody knows why the KS p-value betrays you at production scale. The thresholds that are actually industry convention, the response ladder behind them, and the trap answer interviewers bait on purpose.

The senior-MLOps question that exposes dashboard-driven candidates: your accuracy chart is blind for 60 days, or forever. The layered substitute, proxy metrics, score-distribution sentinels, delayed-label backfill, and paid-for labels, and which layer to trust when they disagree.

The lead-level capstone: not 'which metrics' but 'what's your operating program.' SLOs per model, a drift policy with teeth, retraining as routine rather than rescue, and the inventory discipline that separates teams who run models from teams who babysit them.

The incident with no onset spike and no pager: precision bled out over weeks while every dashboard stayed green. How to run a blameless postmortem when the incident has no clean start time, the root-cause classes that hide this long, and the action items that actually prevent the next one.

A staff platform-design question about leverage, not metrics. The split between what the platform owns and what teams bring, the logging contract that makes everything else possible, and why the hardest problem is alert fatigue, not data collection.