reliability

Every FDE will face an angry customer asking this. The mechanistic explanation interviewers want (why hallucination is the default behavior, not a bug) plus the framing that defuses customer escalations.

Anyone can list mitigations; the question is really about costs: latency, dollars, and lost recall. The three-layer answer with honest price tags that AI-lab interviewers score highest.

Malformed arguments, wrong tool, hallucinated parameters, retry loops: the full failure taxonomy plus the production defenses (validation, idempotency, capped retries) that separate builders from readers.

JSON mode, strict schemas, constrained decoding, validate-and-retry: the escalation ladder for structured output, plus the catch nobody mentions: schema-valid can still be factually wrong.

In high-stakes domains a wrong answer is worse than no answer, but a system that always abstains is useless. The selective-prediction design, the layered abstention triggers, and the two metrics that govern the tradeoff.

The question that separates demo-builders from production engineers. Idempotency keys, the saga pattern, dry-run modes, approval tiers: the distributed-systems toolkit applied to agents, with the trap most answers miss.

The most FDE question in the bank: every real deployment hits an unchangeable legacy system. The answer is an architectural principle, concentrate brittleness in one owned layer, plus what to do when the GUI is the only API.

The staff-level reliability question for agent platforms. Refunds went out, emails sent, tickets closed, and now you have to triage at fleet scale without double-refunding. The recovery model interviewers actually score.

The question where production scars are the rubric. Anyone can write a retry loop; the grade lives in which errors you DON'T retry, why jitter exists, and what an idempotency key actually protects you from.

The retry loop's grown-up sibling. Retries protect one call; a breaker protects the whole dependency from a retry storm. The grade is in the half-open state, where letting through one probe instead of all of them is the difference between recovery and re-killing the server.

One malformed message can silently wedge an entire pipeline, and it happens at every customer eventually. The DLQ answer interviewers want covers what most candidates forget: what happens *after* the message lands there.

A 'simple' queue question with four traps wired in: minutes-long tasks break every default timeout, cancellation has to actually stop spend, progress needs a contract, and one bad job must never wedge the lane.

You control the sender; thousands of customer endpoints you've never seen control your fate. The rubric hides in three places: what 'ordering' really costs, why retries need the receiver's cooperation, and the debugging surface most designs forget.

The pattern is expand-contract; the interview is everything around it: the backfill that locks the table, the rollback nobody rehearsed, and the fact that it's the customer's database, not yours. One named technique carries the whole answer.

Two questions disguised as one: classic multi-region (where the hard part is data, not compute) and dependency failover (where the hard part is that your fallback model gives different answers). Degraded modes are designed, not improvised.

A 4-hour outage is ~5.6x your entire monthly error budget on a 99.9% SLA. You cannot ride it out. The design is a provider abstraction, a circuit breaker, and a tiered set of degraded modes, plus a clear line on what you cache versus what you replay.

The source can only upload the file a handful of times before it saturates, but every receiver also has upload bandwidth. The win is making peers serve each other, so total capacity grows with the swarm instead of bottlenecking on one machine.

Three different causes produce the same rising line, and resident memory alone cannot tell them apart: a leak climbed 25.2 MB an hour and fragmentation climbed 22.8. The gauge that separates them costs nothing and has to have been shipped before the incident.

Every instinct says move fast. The arithmetic says otherwise: eight hours of preparation reached a fixed system 26% sooner than shipping immediately, because each failed attempt costs a two-day approval cycle and you only get one shot per cycle.

At 16k GPUs something fails every few hours, Meta logged 466 interruptions in 54 days training Llama 3. The checkpoint-interval math, straggler detection, and automation that turn failures into a budget line instead of an emergency.

One synchronous job, 16k GPUs, and any single component can stall all of them. The blast-radius analysis interviewers want: which faults take down a rack vs the run, where the shared single points of failure hide, and how parallelism layout maps onto failure domains.