TL;DR: Use percentiles because latency is right-skewed and the mean hides the tail; the key insight is that fan-out makes tail latency hit far more users than 1 percent (20 calls at p99 means roughly 18 percent of page loads see it). And percentiles do not average across hosts, so merge histograms first.
How to approach it
Define percentiles in one sentence, then spend your time on the two things interviewers are actually listening for: why averages mislead, and why tail latency hits far more users than the percentile number suggests. This vocabulary underpins harder questions (p99 incident triage, SLOs, capacity planning), so nail it crisply.
A strong answer
p50 is the median: half of requests finish faster. p95 and p99 are the values 95% and 99% of requests beat. Latency distributions are heavily right-skewed (most requests are fast, a few are very slow), so the mean is nearly useless: one 30-second outlier drags the average while telling you nothing about typical experience, and a "200ms average" can hide a p99 of 8 seconds.
The non-obvious part that scores points: tail latency affects far more than the tail percentage of users. If a single page load fans out to 20 backend calls, the probability that at least one call hits the p99 is 1 − 0.99²⁰ ≈ 18%. Nearly one in five page loads experiences your "1%" latency. This is why Google-scale shops treat p99 (and p999) as the primary signal, and why "it's only 1% of requests" is the answer that fails the question. Worse, the slowest requests are often your heaviest users (biggest customers, largest payloads), so the tail is disproportionately where revenue lives.
The fan-out effect scales viciously, which one row of math per fan-out width shows (all values are 1 minus the per-call success rate raised to the call count):
| Backend calls per page | Hits p99 at least once | Hits p999 at least once |
|---|---|---|
| 1 | 1% | 0.1% |
| 5 | 4.9% | 0.5% |
| 20 | 18.2% | 2.0% |
| 100 | 63.4% | 9.5% |
A microservices migration that turns 5 calls into 100 turns "1% of requests" into the majority experience with no service individually getting slower. That right-hand column is also the argument for caring about p999 at scale: at 100-call fan-out, one page load in ten sees your 99.9th percentile.
Two production corollaries. First, percentiles don't average: you cannot take p99 from ten hosts and average them into a global p99. Two hosts are enough to prove it (computed, not asserted): host A serves 1,000 requests, all at 100ms, so its p99 is 100ms. Host B serves 1,000 requests, 900 at 100ms and 100 at 2,000ms, so its p99 is 2,000ms. Averaging the two per-host p99s reports 1,050ms. Merge the actual 2,000 requests and the true p99 is 2,000ms, because the 100 slow requests are 5% of combined traffic and sit entirely above the 99th percentile. The average understated the tail by nearly half, and it can just as easily overstate it with different weights; the number is not conservative, it is unrelated. Aggregate histograms (or use sketches like t-digest/HDRHistogram), then compute percentiles from the merged distribution. Second, define SLOs on percentiles over a window ("p99 < 500ms over 30 days") and alert on burn rate, not on instantaneous blips.
For an FDE this is daily-driver vocabulary: customers say "your API is slow," and your first clarifying question is "at which percentile, and for which endpoints?", which converts a complaint into a measurable claim.
What interviewers probe next
- "Your p50 is flat but p99 doubled. What kinds of causes does that pattern suggest?" Something affecting a subset: GC pauses, a hot shard or hot key, connection-pool exhaustion under burst, one slow downstream dependency, cold caches for a subset of traffic.
- "How would you instrument a service to get trustworthy percentiles?" Histograms at the source, exported to a system that merges them; measure at the client or load balancer too, since server-side numbers miss queueing and network time.
- "Where does the LLM wrinkle come in?" For streaming, track time-to-first-token and inter-token latency separately from total time; a 20s total with 300ms TTFT feels fine, the reverse feels broken.
Common mistakes
Quoting averages, or saying "we'd monitor latency" without naming a percentile. Averaging percentiles across hosts: a flag interviewers specifically listen for. Dismissing the tail as "only 1%" without the fan-out math. And treating this as trivia rather than connecting it to action. A strong candidate ends with what they'd do: set a percentile SLO, alert on it, and segment by endpoint and customer before debugging.
Key takeaways
- The mean hides the tail; report and alert on p95/p99, never the average.
- Fan-out math: 20 calls at p99 means roughly 18 percent of page loads hit the tail.
- Percentiles do not average; merge histograms or t-digests, then read the percentile off the combined distribution.
- For streaming LLM responses, track time-to-first-token separately from total latency.