classification

A classic metrics question with a domain twist. The answer isn't 'recall'; it's a cost-of-errors argument plus a threshold strategy, and interviewers grade the reasoning, not the metric you pick.

The cheap swap is not a model swap. The prompt becomes atomic typed questions, the parsed label becomes a distribution with a confidence, and the release gate becomes a per-version threshold sweep on the customer's own labeled tickets.

Every classifier and every LLM-as-judge eval reduces to a confusion matrix, but the screen is whether you handle the zero-denominator cases without crashing and can say which metric matters for the problem at hand.

There is no training, just storage; the work is at query time. The signal is whether you vectorize the distance computation, break ties sensibly, and know when O(n) per query forces you onto an ANN index instead.

Every FDE loop opens with this, but the definitions are table stakes. What gets scored is whether you can map each metric to a dollar cost the customer recognizes, here's the framing that does it.

A 0.95 AUC model can still flood a review queue with garbage. Knowing when ROC AUC flatters a model, and what to report instead, separates FDEs who've shipped from those who've only run notebooks.

The accuracy trap is the easy part. Interviewers keep pushing: resample or reweight? Does SMOTE survive contact with production? What happens to your probabilities? Here's the full chain.