streaming

TTFT vs tokens-per-second, the output-length lever everyone forgets, and why streaming is the highest-ROI fix that changes no latency at all. The five-lever answer with real numbers.

A 6% word error rate is a good number that measures the wrong thing. Per-digit accuracy of 98% loses 18% of ten-digit account numbers, and no aggregate metric contains that. Talking over callers is a turn-taking problem, not a model problem, and it has its own metrics too.

Perceived latency is a design surface, not an infrastructure detail. Covers progressive disclosure, SSE vs WebSockets, resumability, and the mid-stream failure question most candidates have never thought about.

Most candidates reach for a smaller model first. The staff answer builds an explicit per-stage latency budget, parallelizes independent tool calls, and spends the time budget where the user actually feels it.

The gentlest 'design a class' question in FDE screens, and the warm-up interviewers use before rate limiters and sessionization. The deque trick is easy; the API and time-window follow-ups are where the signal is.

Sessionization is the analytics primitive behind every 'how do users actually behave' question, and a favorite practical screen. The algorithm is easy; out-of-order events and duplicates are where the round is actually scored.

At-least-once delivery means duplicates are guaranteed, not hypothetical. Anyone can use a set; the round is whether your set grows forever, and what you do with an event that arrives after its window closed.

Batching inputs is how you keep a GPU fed, but the source is an iterator you cannot index or len(). The warm-up that screens whether you can write a lazy generator that handles the ragged final batch.

A dashboard needs the average tokens/sec over the last N seconds, updated on every sample, forever. The warm-up that screens whether you evict by time and keep a running sum instead of re-summing the window each call.

A full sort throws away the structure you were handed. Because no element moves more than k slots, a min-heap of size k+1 always has the next smallest element on top, sorting in O(n log k) and one pass.

This is the shape of the TypeScript debugging round at the agent companies: a forty-line stream handler that mostly works. Each bug has a one-line demonstration, and none of them is in the model. One is a stale closure, one is a decoder splitting a multibyte character, and one is a render per token.

A requirements-interrogation question disguised as an architecture question. The first move isn't Kafka, it's a freshness-SLA conversation that usually deletes 90% of the cost. Here's the script strong FDEs run.

Streaming interviews separate people who have run a job at 3am from people who have read the docs. The hard parts are not the windowed aggregate, they are bounding state so the job does not die, and choosing what a late event does to an already-emitted result.

The constraint is the question. It rules out the two approaches most candidates reach for, and the approach that survives has one failure mode that can take the customer's production database down.

A Copilot/Cursor-style assistant lives or dies on the latency budget. The strong answer spends every millisecond of a 200ms end-to-end budget on purpose, separates the inline-ghost-text path from the chat path, and shows why prefix caching and speculative decoding are the only ways to hit it at 10k concurrent users.

A gateway that screens every prompt and streamed token against a live policy without blowing the latency budget. The strong answer separates the policy decision from its data, runs deterministic checks inline and model checks in parallel, and redacts a leaking secret mid-stream before it renders.

Under 500ms from end of speech to first audio out, across three models and a network. The trick is that you never wait for any stage to finish: you stream and pipeline everything.

A 2-second inference and a user who closed the tab. Naive stacks keep the GPU burning to completion. Real cancellation means propagating disconnect all the way to the batch scheduler.

A bank-round system design where the budget math does the screening: ~10ms for features inside a 100ms decision. Streaming aggregates, the hot-key problem during an active attack, and the parity discipline that makes training data honest.

The dataset is too big to fit on disk, so you stream it. The components that keep the GPU fed, the throughput math that says whether you'll be I/O-bound, and the failure handling that makes a multi-day run resumable.