observability

The classic FDE escalation. Six hypotheses, silent model updates, input drift, prompt entropy, sampling noise, perception shift, and the eval-replay test that splits them in an afternoon.

Provider model upgrades ship faster than your release cycle and quietly shift behavior. The design that turns 'we hope it's fine' into a blocking gate: a frozen golden set, pairwise diffing, segmented scoring, and a shadow-traffic canary.

Launch is the starting line. The staff-level capstone: how production traffic becomes eval cases, prompts ship like code, and a model deprecation becomes a Tuesday instead of an incident.

The classic 'it regressed and the prompt is untouched' triage. The answer that scores walks a decisional tree from retrieval to generation and names the silent culprit most candidates miss: the embedding stack changed under you.

Golden sets catch what you anticipated. The harder problem is the live traffic you never labeled. The answer that scores layers cheap proxy signals against a small anchored golden set and knows what each can and cannot tell you.

The provider bill tells you the total and nothing else. This is a forensics question, and the answer is almost never a pricey model. It is that the transcript grows every turn, so a forty step run costs six times what everyone estimated.

Nobody deployed anything is true of your repository and false of the system. The model provider, a tool behind an API, the knowledge base and the traffic mix all change without a commit. Segmentation comes before theory: a broken intent and a mix shift produce the same headline and need different fixes.

A nightly suite of fifty simulated conversations per intent catches a collapsed tool with certainty and a five-point drift almost never. Knowing which is which, and sizing each layer for what it can actually see, is the whole design. The grader checks outcomes, not transcripts, or the agent learns to sound resolved.

Real LLM logs are JSONL, not JSON, and real log files have a malformed line halfway through. The warm-up that screens whether you can stream a file and aggregate without one bad record nuking the whole run.

A dashboard needs the average tokens/sec over the last N seconds, updated on every sample, forever. The warm-up that screens whether you evict by time and keep a running sum instead of re-summing the window each call.

Averages lie, and interviewers know most candidates quote them anyway. The fastest way to sound production-grade in any latency discussion is to reason about percentiles correctly, including the one trap almost everyone falls into.

A signature FDE triage question with a hidden rubric: interviewers score the *order* of your moves, not just the list. There's a reason 'what changed?' beats 'check the dashboards', and a reason mitigation beats diagnosis.

Standard observability tells you the request returned 200 in 800ms. It cannot tell you the answer was wrong. The strong answer names the new failure plane LLMs introduce, and takes a real position on the prompts-and-PII question.

The question that defines forward-deployed engineering as a discipline: production you're responsible for but can't touch. The answer is a system (version skew, telemetry tiers, diagnostic bundles, release trains), not a collection of favors from customer IT.

A crashed agent is easy. The hard one returns a confident answer after a tool quietly returned nothing and the model made up the rest. The strong answer instruments the steps that fail without throwing, ties traces to an outcome signal, and refuses to call a 200 a success.

The best FDE deployment is the one that doesn't need the FDE anymore. This question screens for whether you build for handoff from day one or quietly make yourself indispensable, which is the failure mode that looks like success.

Aggregate metrics hide exactly the customer who is about to churn. Adding a tenant label to everything is the obvious fix and it is also how you get a cardinality explosion that takes down the monitoring before it takes down the platform.

Three different causes produce the same rising line, and resident memory alone cannot tell them apart: a leak climbed 25.2 MB an hour and fragmentation climbed 22.8. The gauge that separates them costs nothing and has to have been shipped before the incident.

The most common real ticket in applied AI, and it contains almost no information. What you do in the first hour decides whether you spend a day on this or three weeks.

The senior-MLOps question that exposes dashboard-driven candidates: your accuracy chart is blind for 60 days, or forever. The layered substitute, proxy metrics, score-distribution sentinels, delayed-label backfill, and paid-for labels, and which layer to trust when they disagree.

A staff platform-design question about leverage, not metrics. The split between what the platform owns and what teams bring, the logging contract that makes everything else possible, and why the hardest problem is alert fatigue, not data collection.

Logs, traces, and model-quality signals from hundreds of LLM products at firehose volume. The storage split that keeps queries fast, the alerting that survives drift and toxicity at scale, and why sampling has to be cost-aware and tail-biased.