llm

The hard part is not the retrieval pipeline, it is the boundary. Embeddings and the index live inside the VPC; the only question is whether the generation model is an in-VPC open model or an external API with a zero-retention DPA, and what crosses that line.

The 2026 FDE differentiator: customers ask for LLMs by name, and the engineer who knows when a $0 logistic regression beats a $40k/month LLM pipeline, with the cost math, wins the room.

A budget-constrained design problem, not an ideology question. The per-call math that rules an LLM in or out, the cascade that keeps quality while cutting cost 90%, and the SLO arithmetic that decides where the LLM is even allowed to sit.

There's no single number. The trap is reaching for perplexity or BLEU because they're easy to compute, then optimizing a score that has nothing to do with whether the output is good. The move is to match the metric to the task and name where each one lies to you.

A Fermi estimate with a paycheck attached. The interviewers' favorite filter: candidates who jump to a dollar figure miss the two numbers that actually break deployments, peak QPS and tokens-per-minute limits.

Standard observability tells you the request returned 200 in 800ms. It cannot tell you the answer was wrong. The strong answer names the new failure plane LLMs introduce, and takes a real position on the prompts-and-PII question.

Prompts are production code with a worse failure mode: the regression returns 200 OK. The answer interviewers reward treats a one-word prompt edit with the same machinery as a schema migration, with one LLM-specific twist at every stage.

A 'simple' queue question with four traps wired in: minutes-long tasks break every default timeout, cancellation has to actually stop spend, progress needs a contract, and one bad job must never wedge the lane.

The CFO sees API invoices; nobody sees the cost of the inference team they're about to hire. The strong answer models utilization, the variable that decides this question, and names the break-even honestly enough to be trusted.

Two questions disguised as one: classic multi-region (where the hard part is data, not compute) and dependency failover (where the hard part is that your fallback model gives different answers). Degraded modes are designed, not improvised.

Caching LLM output looks like free latency and cost savings until a cache key collides across tenants or a popular key expires under load and stampedes your model. The senior answer is mostly about what you refuse to cache.

Cache hits on similar prompts cut cost and latency, but one false hit serves a stranger's answer to your question. Picking the threshold and handling user-specific data is the whole game.

A 4-hour outage is ~5.6x your entire monthly error budget on a 99.9% SLA. You cannot ride it out. The design is a provider abstraction, a circuit breaker, and a tiered set of degraded modes, plus a clear line on what you cache versus what you replay.

At 100k QPS you cannot 10x the model for a +2% quality bump. This question screens whether you can find the actual operating point under a real constraint and get product, finance, and the customer to agree to it instead of pretending the tradeoff away.