capacity planning

Everyone sizes the pool on average run length, and average is the one statistic an agent fleet does not have. Simulated, the p99 sat at 160 seconds whether the system was half loaded or nearly saturated, which means capacity was never the problem.

A Fermi estimate with a paycheck attached. The interviewers' favorite filter: candidates who jump to a dollar figure miss the two numbers that actually break deployments, peak QPS and tokens-per-minute limits.

Latency, cost, and quality are one budget with three claims on it. The strong answer treats the p99 SLA and the spend ceiling as a joint constraint, finds where they fight (batching), and names the lever it pulls when traffic exceeds what the ceiling can buy at SLA.

Modulo hashing reshuffles everyone the day you add a cell. Consistent hashing fixes that and then loses to the two constraints that actually decide placement in an enterprise deployment: jurisdiction and who must not share a blast radius with whom.

A back-of-the-envelope round where the envelope has a purchase order attached. The arithmetic is the easy half. The scoring is in which assumptions you write down, which way you round, and what you say about the number being wrong.

TP=2, 4, or 8, the answer is a memory budget, a latency target, and one rule: smallest TP that fits, scale with replicas. The worked arithmetic that turns this from vibes into a defensible call.

The back-of-envelope chain every inference-platform round expects: traffic → tokens/sec → per-GPU throughput from bandwidth math → fleet size → dollars. With the headroom factors candidates forget.

The OpenAI capstone design. What separates passing answers: token-based capacity math, KV/prefix-aware routing instead of round-robin, and a degradation ladder for the day demand doubles overnight.

The CFO-facing infra question: a demand forecast with a fat tail, GPUs that take months to land, and three cost structures that win in different regimes. The layered commitment model that keeps you from paying for peak all year.

10 billion vectors is where raw float storage (30TB) stops being an option. The answer does the HNSW vs IVF-PQ RAM math cold and shows why PQ is the only thing that fits in memory.