cost

Tokens drive your customer's bill, their latency, and the weird failures like botched arithmetic. The rough numbers every FDE should rattle off, plus the trap answers interviewers listen for.

The KV-cache mechanics behind the discount, the prefix rule that silently breaks caching for most teams, and the workloads where caching cuts bills 50-90% versus the ones where it does nothing.

Two compression levers with very different bills. The arithmetic that decides between them, when to stack both, and the eval discipline that keeps a 'cheaper' model from quietly tanking your hardest 10% of queries.

The escalation ladder is settled; the at-scale tradeoff is not. Why retries quietly multiply your bill and tail latency, why constrained decoding has its own taxes, and the hybrid that most high-volume pipelines actually run.

Sending every request to the frontier model is the lazy bill. The patterns that route by difficulty, the verifier that enforces a hard quality floor, and the failure mode that turns a cost-saving cascade into a worse, slower system.

Each chat turn resends the whole conversation, so the model re-prefills the same prefix every time. Server-side KV caching and client-side cache breakpoints kill that redundant compute, but exact-prefix matching and stale entries are where teams silently get it wrong.

The unglamorous design question that exposes who has run LLMs in production. Batch APIs at 50% off, idempotency keys, poison records, spend kill-switches: the full checklist interviewers listen for.

A budget-constrained design problem, not an ideology question. The per-call math that rules an LLM in or out, the cascade that keeps quality while cutting cost 90%, and the SLO arithmetic that decides where the LLM is even allowed to sit.

A Fermi estimate with a paycheck attached. The interviewers' favorite filter: candidates who jump to a dollar figure miss the two numbers that actually break deployments, peak QPS and tokens-per-minute limits.

The CFO sees API invoices; nobody sees the cost of the inference team they're about to hire. The strong answer models utilization, the variable that decides this question, and names the break-even honestly enough to be trusted.

Cache hits on similar prompts cut cost and latency, but one false hit serves a stranger's answer to your question. Picking the threshold and handling user-specific data is the whole game.

A 2-second inference and a user who closed the tab. Naive stacks keep the GPU burning to completion. Real cancellation means propagating disconnect all the way to the batch scheduler.

Nobody cares about your final number, they're grading the decomposition, the sanity checks, and whether you convert tokens into dollars unprompted. Here's a clean worked path.

At 100k QPS you cannot 10x the model for a +2% quality bump. This question screens whether you can find the actual operating point under a real constraint and get product, finance, and the customer to agree to it instead of pretending the tradeoff away.

Two hard numbers in the contract: 95% accuracy and $0.50 per resolution. The job is an eval and rollout system that proves both hold on real traffic before you scale, with rollback triggers wired to the same numbers.

The back-of-envelope chain every inference-platform round expects: traffic → tokens/sec → per-GPU throughput from bandwidth math → fleet size → dollars. With the headroom factors candidates forget.

The CFO-facing infra question: a demand forecast with a fat tail, GPUs that take months to land, and three cost structures that win in different regimes. The layered commitment model that keeps you from paying for peak all year.