TL;DR: Compute three numbers, not one: daily token volume (about $200k/month here), peak request rate converted to tokens-per-minute (the limit that actually breaks launches), and the sensitivity to tokens-per-call. Then name prompt caching and model routing as the first cost levers.
How to approach it
Narrate your assumptions out loud and round aggressively, because the score is for structure, not arithmetic precision. The trap is computing only the average. Strong candidates compute three things: daily token volume (cost), peak request rate (rate limits and infra), and the sensitivity of the answer to its biggest assumption (token count per call).
A strong answer
State assumptions: 50k DAU × 10 calls = 500k calls/day. Assume a typical RAG-ish call: ~2,000 input tokens (system prompt + retrieved context + history) and ~500 output tokens. So ~1B input and ~250M output tokens per day.
Cost: pick round numbers, say $3 per million input tokens and $15 per million output for a mid-tier model. That's 1,000 × $3 + 250 × $15 ≈ $6,750/day, roughly $200k/month. Immediately flag the lever. At those volumes you don't accept list behavior: prompt caching (the system prompt and shared context repeat across all 500k calls) can cut input cost dramatically, routing easy calls to a small model can cut total spend 5 to 10x, and committed-use or batch pricing applies for any non-interactive traffic.
Throughput is where deployments actually break. Average rate is 500k / 86,400 ≈ 6 requests/sec, which sounds trivial. But usage concentrates: a business app does most traffic in an 8 to 10 hour window with lunchtime peaks, so plan peak ≈ 3 to 5x average, so 20 to 30 RPS. Convert to tokens-per-minute, because that's how providers limit you: 25 RPS × 2,500 tokens ≈ 3.75M TPM, far beyond default tier limits. So you need a quota increase or enterprise agreement negotiated before launch, client-side rate limiting with backoff, and a queue to absorb bursts rather than surfacing 429s to users.
The left column is the bill and the right half is the launch. Step 8 is the number that actually stops deployments, and it is the one nobody computes until a provider returns a 429 in front of users.
The worked example traces to these numbers:
| Quantity | Value (worked example) |
|---|---|
| Calls per day | 50k DAU x 10 = 500k |
| Tokens per call | ~2,000 input + ~500 output |
| Daily tokens | ~1B input + 250M output |
| Cost | ~$6,750/day, ~$200k/month at list |
| Average rate | ~6 RPS |
| Peak rate | 3-5x avg, 20-30 RPS |
| Peak tokens-per-minute | ~3.75M TPM, over default tier |
Infra around the model is light: at 30 RPS, a couple of stateless app servers handle orchestration; the real provisioning is the vector store or retrieval layer (if RAG), Redis for caching and rate limiting, and an observability pipeline. Logging full prompts and responses at 1B tokens/day is itself ~5 to 10 GB/day, so plan retention and sampling.
Close with sensitivity: "My answer is linear in tokens per call. If context grows to 8k tokens, that $200k becomes $700k+/month. The first engineering investment is measuring and trimming real token usage, not optimizing infra."
What interviewers probe next
- "Cut the cost 10x without destroying quality." Model routing or cascades (small model first, escalate on low confidence), prompt caching, trimming retrieved context, distillation for the highest-volume call type, batch API for async work. And since the follow-up is near-universal, have the ladder pre-computed on this example (arithmetic verified): baseline $6,750/day. Prompt caching first: if ~1,200 of the 2,000 input tokens are the repeated system prompt and scaffolding, cached at roughly a tenth of list, daily input cost drops from $3,000 to about $1,380, total $5,130. Routing second: send the 70% of calls that are classification, extraction, and short lookups to a small model at about a tenth of the price, and the total lands near $1,900/day, roughly $57k/month, a 3.5x cut with two configuration-level changes and zero infrastructure. The remaining distance to 10x comes from trimming retrieved context (the estimate is linear in it) and batch pricing on the async share; walking the ladder with running numbers is what makes "we can cut this" sound like a plan instead of a hope.
- "What about latency?" TPM budgets say nothing about UX; estimate time-to-first-token, stream everything user-facing, and parallelize the 10 calls where the workflow allows.
- "Self-host instead?" At ~$200k/month of API spend, the build-vs-buy math gets real; sketch GPU count from tokens/sec/GPU and compare loaded costs (see the build-vs-buy question).
Common mistakes
Producing one precise-sounding dollar figure with unstated token assumptions: false precision reads junior. Computing average QPS and declaring it "easy" without peak analysis. Forgetting rate limits are in tokens, not requests. Never mentioning prompt caching or model routing, the two levers any provider's own FDEs would reach for first. And skipping the sensitivity check that shows you know which assumption dominates the answer.
Key takeaways
- Three numbers win this: daily tokens (cost), peak RPS, and tokens-per-minute (the limit that breaks launches).
- Provider limits are denominated in TPM, not RPS, so a "trivial" 6 RPS average still blows past your tier at peak.
- First cost levers are prompt caching and model routing, before any infrastructure work.
- The whole estimate is linear in tokens-per-call, so measuring real token usage is the first engineering investment.