metrics

A classic metrics question with a domain twist. The answer isn't 'recall'; it's a cost-of-errors argument plus a threshold strategy, and interviewers grade the reasoning, not the metric you pick.

Standard A/B machinery assumes a crisp conversion event; agents give you fuzzy, delayed, gameable outcomes. The metric hierarchy, the reopen-window trick, and the sample-size reality check that win this question.

Nobody deployed anything is true of your repository and false of the system. The model provider, a tool behind an API, the knowledge base and the traffic mix all change without a commit. Segmentation comes before theory: a broken intent and a mix shift produce the same headline and need different fixes.

Everyone quotes p95 in latency SLOs, but few can compute it without a library and fewer can explain why the index is ceil(p*n)-1, not p*n. The warm-up that screens whether you actually understand the metric you live by.

A dashboard needs the average tokens/sec over the last N seconds, updated on every sample, forever. The warm-up that screens whether you evict by time and keep a running sum instead of re-summing the window each call.

Every classifier and every LLM-as-judge eval reduces to a confusion matrix, but the screen is whether you handle the zero-denominator cases without crashing and can say which metric matters for the problem at hand.

Every FDE loop opens with this, but the definitions are table stakes. What gets scored is whether you can map each metric to a dollar cost the customer recognizes, here's the framing that does it.

A 0.95 AUC model can still flood a review queue with garbage. Knowing when ROC AUC flatters a model, and what to report instead, separates FDEs who've shipped from those who've only run notebooks.

The accuracy trap is the easy part. Interviewers keep pushing: resample or reweight? Does SMOTE survive contact with production? What happens to your probabilities? Here's the full chain.

This is a discovery question wearing an ML costume. The strongest candidates run a four-step translation from business pain to loss function, and put a number on each error before choosing anything.

The maturity exam of production ML: a model that's better on paper and worthless in production. Five distinct causes, the funnel diagnostic that separates them, and when the right call is don't ship.

The 2026 successor to 'explain precision and recall', interviewers now test whether your classical ML evaluation discipline survives contact with GenAI. The decomposition, the RAGAS dimensions, and the judge-calibration step everyone skips.

A confusion matrix is the 2x2 of right and wrong predictions per class, and it is the reason a 99% accurate fraud model can be useless. The worked imbalance example, and how reading the matrix as precision and recall fixes the lie.

The most-reported decomposition prompt in the FDE canon. Almost every candidate builds a routing model, and almost every candidate is optimizing the wrong term of the equation.

The only round where you choose the problem, which sounds like a gift and is where most candidates lose it. Four things get graded in one flow, and the fourth is the one engineers skip.

Undefined success is how pilots die in 'one more month' purgatory. Here's the facilitation script that converts 'it should just be good' into numbers, baselines, and a decision date the customer owns.

An integrity trap with a political twist: the person gaming your metric is usually your own champion. Blowing the whistle loudly and looking away both fail, here's the third path.

You hit 71% against an 85% bar nobody wants to say out loud, and the easy moves (quietly redefine the metric, ask for 'one more month') are how pilots rot in purgatory. Here's the honest salvage path, and how to recommend a kill without losing the account.

Every ML person has chased a proxy that diverged from the thing that mattered. The screen is whether you noticed it yourself, why the proxy fooled you, and how you realigned without just swapping one number for another.

'Proud of' is not a metric. The answer that lands names the business number that moved, your specific hands on it, and how you'd defend the attribution to a skeptic who assumes you're rounding up.