59Route each request to a cheap 7B or expensive 70B model to maximize satisfaction minus cost. Design the online router.▼hardCohereDatabricksMartian1 replies◆ premiumA static threshold on prompt length leaves money on the table both ways. The router that learns is a contextual bandit, and the interview lives in the reward function and the cold-start handling, not the algorithm name.Open full answer →
38Hit a hard p99 SLA on an LLM product without blowing a fixed monthly spend ceiling. Model it.▼hardOpenAIAnthropicDatabricks1 replies◆ premiumLatency, cost, and quality are one budget with three claims on it. The strong answer treats the p99 SLA and the spend ceiling as a joint constraint, finds where they fight (batching), and names the lever it pulls when traffic exceeds what the ceiling can buy at SLA.Open full answer →
