debugging

The classic FDE escalation. Six hypotheses, silent model updates, input drift, prompt entropy, sampling noise, perception shift, and the eval-replay test that splits them in an afternoon.

A 6% word error rate is a good number that measures the wrong thing. Per-digit accuracy of 98% loses 18% of ten-digit account numbers, and no aggregate metric contains that. Talking over callers is a turn-taking problem, not a model problem, and it has its own metrics too.

The modal FDE design question of the last two years. There's a scoring trap in the first 60 seconds, and most candidates jump to fixes and fail. Here's the diagnostic tree that wins it.

The classic 'it regressed and the prompt is untouched' triage. The answer that scores walks a decisional tree from retrieval to generation and names the silent culprit most candidates miss: the embedding stack changed under you.

The provider bill tells you the total and nothing else. This is a forensics question, and the answer is almost never a pricey model. It is that the transcript grows every turn, so a forty step run costs six times what everyone estimated.

Everyone reaches for a bigger machine. The two mechanisms that actually cause this both scale with corpus size, both hit the tail before the median, and one of them is the permission filter your security review insisted on.

Vectors from two versions of an embedding model are not comparable, and nothing in the stack will tell you so. The cosine between the same document's old and new vector is roughly zero, which is why a half-finished migration returns confident nonsense at full speed.

A wrong answer inside one conversation dies with the conversation. A wrong answer written to memory becomes a retrievable fact that conditions every later run, and the agent then derives new conclusions from it. Deleting it is the part nobody designed for.

Everyone sizes the pool on average run length, and average is the one statistic an agent fleet does not have. Simulated, the p99 sat at 160 seconds whether the system was half loaded or nearly saturated, which means capacity was never the problem.

The rate limiter is working correctly and that is the problem. It counts requests, and one hundred-thousand-token request occupies the KV cache of fifty short ones, so admitting both tenants equally hands one of them eighty percent of the machine.

A query that errors is a bug. A query that runs and returns a plausible wrong number is a product teaching a finance team, one figure at a time, not to trust it. The diagnosis is a classification exercise, and the fix is almost never a better prompt.

Before you touch a model, check whether the two numbers measure the same thing. Field accuracy of 96% across twelve fields is 61% of documents fully correct, which means half of the drop may be arithmetic. The other half is the documents the pilot never saw.

Nobody deployed anything is true of your repository and false of the system. The model provider, a tool behind an API, the knowledge base and the traffic mix all change without a commit. Segmentation comes before theory: a broken intent and a mix shift produce the same headline and need different fixes.

The eval set says recall fell 2.3 points, which is inside the noise you would expect from a re-index. Support says the product got worse. Both are right, and the reason is that one document class lost 49 points of recall while everything else improved slightly.

Recall@10 is 0.94 and climbing, and answer accuracy is going down. The two facts are consistent, and reconciling them is the whole question: every passage you add to raise recall also adds a distractor the generator has to survive.

The document count matches, nothing threw, and a specific set of documents is now unreachable by any query. That combination narrows the cause fast, and the two survivors are an approximate-search parameter that a rebuild reset and a metadata filter applied in the wrong order.

A reranker cannot promote a document the retriever never returned, which makes candidate-set recall a hard ceiling on the entire exercise. Compute the ceiling and the disappointment becomes arithmetic: three points of headroom were available, twelve were sitting upstream, and the latency was an implementation detail.

The format that filters out LeetCode-only candidates: unfamiliar code, failing tests, 45 minutes. There's a repeatable protocol for it, and a reason the first five minutes decide the round.

You can't read the model's weights, so you debug the inputs you control. There's a four-bucket triage, prompt, context, model, or spec, that localizes the fault fast, and the real deliverable is feeding each diagnosis back so the same failure can't recur.

This is the shape of the TypeScript debugging round at the agent companies: a forty-line stream handler that mostly works. Each bug has a one-line demonstration, and none of them is in the model. One is a stale closure, one is a decoder splitting a multibyte character, and one is a render per token.

This exact gap kills more customer pilots than any modeling choice. There's a ranked list of culprits, and one of them hides inside innocent-looking feature pipelines at almost every enterprise.

The most expensive bug class in production ML: two implementations of 'the same' feature that quietly disagree. The diagnostic that finds it in an afternoon, and the architecture that makes it impossible.

The signature production-ML incident, and a pure FDE test: a ranked differential diagnosis run live on a call, because 'nothing changed' is always false, and one hypothesis explains half of all cases.

The most expensive leakage bug in feature-store ML: training joins that grab feature values from after the label event. The timeline that exposes it, the one query that proves it, and the fix that survives audit.

Not leakage, not skew, not drift: the model learned exactly what you asked, and you asked for the wrong thing. The proxy-label mismatch that passes every test, the audit that catches it, and why this is the failure no metric can see.

Join fan-out is the bug behind half of all 'the dashboard numbers are wrong' escalations an FDE will ever field. The mechanism, the 30-second detection query, and the three legitimate fixes.

The capstone triage scenario data-platform FDEs get role-played through. A 9x blowup with an obvious-looking suspect, the scoring is on your diagnostic order, the questions you ask, and not stopping at the first plausible cause.

A signature FDE triage question with a hidden rubric: interviewers score the *order* of your moves, not just the list. There's a reason 'what changed?' beats 'check the dashboards', and a reason mitigation beats diagnosis.

The signature FDE debugging genre: a moving fault, an opaque client, and a customer who's sure it's your fault. The winning method splits the problem at the boundary, and knows the four classic culprits hiding on their side.

The bug only appears under load, the code 'hasn't changed,' and the customer just got charged twice. There's one root-cause pattern behind almost every version of this incident, and a fix with a subtle race most candidates miss.

Vercel's signature triage question. 3s TTFB is almost never 'the server is slow', it's a rendering-strategy problem with four classic causes, and the rubric rewards proving each fix with a measurement, not vibes.

Three different causes produce the same rising line, and resident memory alone cannot tell them apart: a leak climbed 25.2 MB an hour and fragmentation climbed 22.8. The gauge that separates them costs nothing and has to have been shipped before the incident.

The most common real ticket in applied AI, and it contains almost no information. What you do in the first hour decides whether you spend a day on this or three weeks.

An incident question graded on method, not tools: interviewers want ordered hypotheses, each with a five-minute test. The version-skew check most candidates never think of, the cache trap, and the triage tree that finds the usual culprit fast.

The staff on-call simulation: three alarms at once, partial information, and pressure to do something. How to separate symptom from cause, the common-cause hypothesis that explains all three, and why mitigating before diagnosing is the senior move, not a shortcut.

NCCL's topology tricks, GPUDirect RDMA, and the single most misleading metric in distributed training. If you've ever trusted nvidia-smi on a slow run, this question was written for you.

A reported DeepMind debugging-round prompt. What's scored is your search order, not your bug list, the overfit-one-batch ladder that finds 90% of flat-loss causes in under an hour.

The worst failure mode in distributed training: no crash, no log, every GPU pinned at 100%, and the job dead. The triage order that finds a collective mismatch or a dead rank before you burn a day of cluster time.

The fleet was fine on 8k contexts and falls over when a few 200k-token requests arrive. The math that explains why, the admission and pooling fixes that contain it, and why throwing GPUs at it is the wrong first move.

No error, no hang, the loss still moves, but the run is suddenly half as fast and burning the same dollars. The triage that separates a straggler from a fabric problem from broken comm/compute overlap, using the signals nvidia-smi can't give you.

Every control in the stack validates the shape of the action and none validates its target. A refund of the right amount, to a real customer, through a permitted tool, against the wrong account, is indistinguishable from success at every layer that is watching.