34A customer wants to auto-decide on a high-stakes classifier. How do you make the probabilities safe to act on?▼hardGoogleScaleDatabricks2 replies◆ premiumWhen a model auto-approves loans or routes patients, the probability is the product, not a dashboard number. The calibration-plus-abstention design that lets a customer trust an automated decision, and the subgroup trap that gets it pulled in audit.Open full answer →
35Offline AUC is 0.91 and falling apart live. You suspect the feature store's point-in-time join. Find the bug.▼hardDatabricksGoogleSnowflake2 replies◆ premiumThe most expensive leakage bug in feature-store ML: training joins that grab feature values from after the label event. The timeline that exposes it, the one query that proves it, and the fix that survives audit.Open full answer →
36The customer needs this classifier under a 150ms p99 and a $20k/month budget at 50M calls/day. LLM or not?▼hardOpenAIDatabricksGoogle1 replies◆ premiumA budget-constrained design problem, not an ideology question. The per-call math that rules an LLM in or out, the cascade that keeps quality while cutting cost 90%, and the SLO arithmetic that decides where the LLM is even allowed to sit.Open full answer →
37The customer's marketplace has network effects, so a user-level A/B test is biased. How do you measure the model's impact?▼hardMetaGoogleDatabricks1 replies◆ premiumWhen treatment leaks between units, a clean A/B lies in both directions. The interference taxonomy, when to reach for switchback vs cluster vs geo designs, and the analysis trap that makes naive standard errors useless.Open full answer →
38The model aced every offline eval and the customer says it's useless. The labels look fine. What now?▼hardScaleGoogleDatabricks1 replies◆ premiumNot leakage, not skew, not drift: the model learned exactly what you asked, and you asked for the wrong thing. The proxy-label mismatch that passes every test, the audit that catches it, and why this is the failure no metric can see.Open full answer →
39The customer wants to test five model variants without losing money on the bad ones. A/B test or a bandit?▼hardMetaGoogleDatabricks1 replies◆ premiumBandits sound strictly better and usually are not. The regret-vs-inference tradeoff, the three conditions that actually favor a bandit, and the production failure that makes adaptive allocation a debugging nightmare.Open full answer →