TL;DR: State the decomposition before computing (conversations x calls-per-conversation x tokens-per-call), narrate every assumption, sanity-check from the other side, then do the move that separates FDEs: convert tokens to dollars and rate limits unprompted, and reframe the bill against the labor it augments.
How to approach it
Fermi questions in FDE loops (Google's "general cognitive ability" round, Palantir-style estimation) grade three things: a clean decomposition stated before you compute, defensible per-step assumptions said out loud, and a sanity check at the end. For FDE roles specifically there's a fourth, differentiating move: converting tokens into dollars and rate limits unprompted, because that's the conversation you'll have with customers weekly. Narrate everything; a silent calculator scores worse than a talkative approximator.
A strong answer
"Let me set up the structure first: tokens/day = conversations/day × LLM calls per conversation × tokens per call. I'll estimate each, then sanity-check.
Conversations/day: A Fortune-500 consumer company, say a telco with ~50M customers. If 2% contact support monthly, that's 1M contacts/month, ~33k/day. Round to 50k/day to include chat, email, and voice transcription.
LLM calls per conversation: Modern support agents aren't one call. Per conversation: intent classification, 2–3 RAG retrievals with generation, maybe a tool-calling loop, a summary for the CRM. Call it 10 LLM calls per conversation.
Tokens per call: Dominated by input, system prompt (~1k), retrieved context (~3k), conversation history (~1–2k), so ~5k in, ~500 out: ~5.5k tokens per call. Note the asymmetry, input tokens dominate, which matters because they're cheaper but they're also where caching wins live.
Multiply: 50k × 10 × 5.5k ≈ 2.75 billion tokens/day. I'll say 1–5B/day as my range.
Sanity check from the other side: 2.75B/day ÷ 86,400s ≈ 32k tokens/sec sustained, peaking maybe 3× at ~100k/sec. That's large but plausible for enterprise tiers, and it tells me rate limits are a real conversation, not a checkbox.
Now dollars, because that's what the customer asks next: at blended rates of roughly $1–3 per million input tokens, that's ~$3–8k/day, call it $1–3M/year. Against a 5,000-agent support org costing maybe $250M/year fully loaded, the LLM bill is ~1% of the labor cost it's augmenting, so the ROI conversation is about deflection rates, not token prices. And if the customer wants to cut that bill: prompt caching on the system prompt and context (the 5k input is highly repetitive), model routing so classification runs on a small model, and batching the CRM summaries off-peak, that's easily 3–5× off the number."
What interviewers probe next
- "Which assumption is your answer most sensitive to?", calls-per-conversation (agentic loops can be 3 or 30). Saying so shows you know your own error bars.
- "Now it's voice, what changes?" Transcription tokens, latency budgets, streaming.
- "The CFO says that's too expensive." Re-run the cost levers and reframe against deflected contact cost (~$5–10 per contact).
- Perturbations: B2B instead of consumer, 10× the doc corpus, on-prem serving. Your structure should absorb each in one line, and it is worth seeing what "one line" means. B2B: conversations collapse (2k/day, not 50k) while tokens per call triple on heavier documents, so 2k × 10 × ~16.5k ≈ 0.33B/day, an order of magnitude down, same formula. Ten times the corpus: the total barely moves, because corpus size never appears in the decomposition; retrieval still returns the same top-k chunks per call, and saying that out loud is the insight the perturbation is fishing for. On-prem: the dollars column becomes a GPU count, 32k tokens/sec sustained at an assumed ~1.5k tokens/sec of decode per GPU is ~21 GPUs, times the 3x peak factor lands near 60-plus before redundancy (assumption-laden, and you say so, but the structure converted cleanly). The pattern across all three: nothing about the decomposition changed, only which factor moved, which is exactly what a decomposition is for.
Common mistakes
- Diving into arithmetic without stating the decomposition, the #1 fail.
- Assuming one LLM call per conversation (reads as never having built an agent) or ignoring the input/output token asymmetry.
- No sanity check, so a 1000× error sails through unnoticed.
- Stopping at tokens. For an FDE, the token count is the warm-up; cost, rate limits, and reduction levers are the job.