latency

TTFT vs tokens-per-second, the output-length lever everyone forgets, and why streaming is the highest-ROI fix that changes no latency at all. The five-lever answer with real numbers.

The KV-cache mechanics behind the discount, the prefix rule that silently breaks caching for most teams, and the workloads where caching cuts bills 50-90% versus the ones where it does nothing.

The OpenAI FDE deep-dive where you're expected to walk the full stack (network, queueing, tokenization, prefill, KV cache, batching, decode) in order, with a measurement at each hop. The triage sequence, with the numbers that tell you which layer is guilty.

Two compression levers with very different bills. The arithmetic that decides between them, when to stack both, and the eval discipline that keeps a 'cheaper' model from quietly tanking your hardest 10% of queries.

Each chat turn resends the whole conversation, so the model re-prefills the same prefix every time. Server-side KV caching and client-side cache breakpoints kill that redundant compute, but exact-prefix matching and stale entries are where teams silently get it wrong.

Without a KV cache, generating token N re-attends over all N-1 prior tokens from scratch every step, turning decoding quadratic. The cache stores each layer's past keys and values so each new token costs one forward step, and its size is what caps your batch and context.

A 6% word error rate is a good number that measures the wrong thing. Per-digit accuracy of 98% loses 18% of ten-digit account numbers, and no aggregate metric contains that. Talking over callers is a turn-taking problem, not a model problem, and it has its own metrics too.

Word error rate is the number every vendor quotes and the one that predicts the least. The offline suite is built from confirmed real calls with noise and codecs applied, scored on entity accuracy, tail latency and false interruptions. Production adds what a recording cannot simulate: real callers and the reopen rate.

Most candidates reach for a smaller model first. The staff answer builds an explicit per-stage latency budget, parallelizes independent tool calls, and spends the time budget where the user actually feels it.

Everyone reaches for a bigger machine. The two mechanisms that actually cause this both scale with corpus size, both hit the tail before the median, and one of them is the permission filter your security review insisted on.

Everyone sizes the pool on average run length, and average is the one statistic an agent fleet does not have. Simulated, the p99 sat at 160 seconds whether the system was half loaded or nearly saturated, which means capacity was never the problem.

The rate limiter is working correctly and that is the problem. It counts requests, and one hundred-thousand-token request occupies the KV cache of fifty short ones, so admitting both tenants equally hands one of them eighty percent of the machine.

A reranker cannot promote a document the retriever never returned, which makes candidate-set recall a hard ceiling on the entire exercise. Compute the ceiling and the disappointment becomes arithmetic: three points of headroom were available, twelve were sitting upstream, and the latency was an implementation detail.

Everyone quotes p95 in latency SLOs, but few can compute it without a library and fewer can explain why the index is ceil(p*n)-1, not p*n. The warm-up that screens whether you actually understand the metric you live by.

A budget-constrained design problem, not an ideology question. The per-call math that rules an LLM in or out, the cascade that keeps quality while cutting cost 90%, and the SLO arithmetic that decides where the LLM is even allowed to sit.

Averages lie, and interviewers know most candidates quote them anyway. The fastest way to sound production-grade in any latency discussion is to reason about percentiles correctly, including the one trap almost everyone falls into.

A signature FDE triage question with a hidden rubric: interviewers score the *order* of your moves, not just the list. There's a reason 'what changed?' beats 'check the dashboards', and a reason mitigation beats diagnosis.

The round is conversational discovery plus live Excalidraw diagramming, and it's graded on sequence: candidates who draw before asking fail. The discovery battery, the voice-pipeline boxes worth drawing, and the latency budget that anchors the whole design.

A Copilot/Cursor-style assistant lives or dies on the latency budget. The strong answer spends every millisecond of a 200ms end-to-end budget on purpose, separates the inline-ghost-text path from the chat path, and shows why prefix caching and speculative decoding are the only ways to hit it at 10k concurrent users.

A gateway that screens every prompt and streamed token against a live policy without blowing the latency budget. The strong answer separates the policy decision from its data, runs deterministic checks inline and model checks in parallel, and redacts a leaking secret mid-stream before it renders.

Ten X-ray studies a second, a 2-second budget, and a missed cancer is unacceptable. The hard parts are recall-first thresholding, a monitor that watches the radiologist, and surviving a new scanner.

Under 500ms from end of speech to first audio out, across three models and a network. The trick is that you never wait for any stage to finish: you stream and pipeline everything.

One GPU, callers blocking on the response, and a hardware unit that is only efficient when fed wide batches. The whole job is trading a few milliseconds of queue wait for a large jump in throughput without ever stalling a request.

At 100k QPS you cannot 10x the model for a +2% quality bump. This question screens whether you can find the actual operating point under a real constraint and get product, finance, and the customer to agree to it instead of pretending the tradeoff away.

Free tokens from idle FLOPs, losslessly, which surprises most candidates. The mechanism, the acceptance-rate math, and the high-batch regime where speculation quietly stops paying.