evaluation

This is the live practical several labs run, and it is scored on protocol more than on the prompt you end with. Read all thirty before editing, cluster them, hold some out, change one thing at a time, and report with intervals, because 24 of 30 fixed is somewhere between 63% and 91%.

A 6% word error rate is a good number that measures the wrong thing. Per-digit accuracy of 98% loses 18% of ten-digit account numbers, and no aggregate metric contains that. Talking over callers is a turn-taking problem, not a model problem, and it has its own metrics too.

Word error rate is the number every vendor quotes and the one that predicts the least. The offline suite is built from confirmed real calls with noise and codecs applied, scored on entity accuracy, tail latency and false interruptions. Production adds what a recording cannot simulate: real callers and the reopen rate.

The cheap swap is not a model swap. The prompt becomes atomic typed questions, the parsed label becomes a distribution with a confidence, and the release gate becomes a per-version threshold sweep on the customer's own labeled tickets.

Decide whether retrieval can be removed after fine-tuning. Test changed facts, source support, access boundaries, and the cost of keeping answers current.

Separate advertised context capacity from reliable use of evidence. Design position, distraction, and multi-document tests before trusting a million-token window.

You cannot unit-test a stochastic two-agent pipeline with a golden output. You evaluate it the way you stress a distributed system: thousands of simulated runs, controlled fault injection, and a judge calibrated against humans.

The hard parts are not the four roles. They are the communication substrate, a termination condition that is not just a retry cap, and stopping the Coder from forgetting the ticket after five repair cycles.

Self-correcting RAG is easy to demo and easy to ruin. The discipline is a gate that grades evidence before drafting, a termination rule that is not just a loop cap, and a metric that proves correction adds facts rather than fluff.

DSPy's pitch is to stop hand-writing prompts and instead declare what the pipeline does, define a metric, and let a compiler search for the prompts and examples. The catch is that the compiler is only as good as the metric, and an LLM-judge metric drags its own biases in.

An agent loop optimizes whatever the verifier rewards, so a weak verifier gets gamed: tests edited, outputs hardcoded, inputs memorized. The fix is a verifier the agent cannot author or edit, run on evidence in a clean environment, with held-out and adversarial checks that track intent.

Every eval question in the prep canon is about answer quality. This agent does not produce answers, it moves freight. You cannot score a reroute for faithfulness, and the candidate who tries has not understood what they built.

A query that errors is a bug. A query that runs and returns a plausible wrong number is a product teaching a finance team, one figure at a time, not to trust it. The diagnosis is a classification exercise, and the fix is almost never a better prompt.

Before you touch a model, check whether the two numbers measure the same thing. Field accuracy of 96% across twelve fields is 61% of documents fully correct, which means half of the drop may be arithmetic. The other half is the documents the pilot never saw.

Nobody deployed anything is true of your repository and false of the system. The model provider, a tool behind an API, the knowledge base and the traffic mix all change without a commit. Segmentation comes before theory: a broken intent and a mix shift produce the same headline and need different fixes.

A nightly suite of fifty simulated conversations per intent catches a collapsed tool with certainty and a five-point drift almost never. Knowing which is which, and sizing each layer for what it can actually see, is the whole design. The grader checks outcomes, not transcripts, or the agent learns to sound resolved.

Three correct queries scored 0 of 3 on exact match and 3 of 3 on execution. The golden set is question, reference SQL and result set, stratified by the ways queries go wrong, with abstention as its own column. A hundred cases gives plus or minus eight points; four hundred gives four.

The eval set says recall fell 2.3 points, which is inside the noise you would expect from a re-index. Support says the product got worse. Both are right, and the reason is that one document class lost 49 points of recall while everything else improved slightly.

Recall@10 is 0.94 and climbing, and answer accuracy is going down. The two facts are consistent, and reconciling them is the whole question: every passage you add to raise recall also adds a distractor the generator has to survive.

A reranker cannot promote a document the retriever never returned, which makes candidate-set recall a hard ceiling on the entire exercise. Compute the ceiling and the disappointment becomes arithmetic: three points of headroom were available, twelve were sitting upstream, and the latency was an implementation detail.

Extraction accuracy cannot reveal a document that never arrived. Reconcile expected evidence with received and parsed artifacts, and keep completeness separate from any claim decision.

Cheap typed judgments make it affordable to check every input, tool call and output. The design question is which checks are semantic, which are deterministic, and how you prove the layer catches what it claims to.

The pitch is a model that runs the loop. The design that survives production is a loop that code runs, with the model answering three narrow questions inside it: which tool, is this done, does this need a stronger model.

70 percent is an average over tasks and runs, and it hides the number the customer cares about: does the same request get the same right outcome every time? Two agents with identical averages can differ from shippable to unusable.

Single-session evals pass while users complain, because memory fails across sessions and at three separate stages. The answer is a planted-fact harness that names which stage lost the fact, plus the two tests teams skip: forgetting on request and leaking across users.

The customer wants the agent to practice before it can write to production. Design the task world, reset semantics, tool boundary and verifier, then explain when its scores justify training or release.

A frozen policy jumps from 70% to 90% after parallelizing its evaluation. Reconstruct the denominator, separate runner faults from policy behavior, and design the experiment that could invalidate the gain.

Every classifier and every LLM-as-judge eval reduces to a confusion matrix, but the screen is whether you handle the zero-denominator cases without crashing and can say which metric matters for the problem at hand.

This exact gap kills more customer pilots than any modeling choice. There's a ranked list of culprits, and one of them hides inside innocent-looking feature pipelines at almost every enterprise.

Everyone can describe the five folds. The question is really about the second half, the two data shapes, both everywhere in enterprise data, where standard k-fold quietly lies to you.

Time-series is where standard ML habits, random splits, fancy models first, single-number forecasts, fail loudest in front of customers. The baseline discipline and backtesting setup that keep you credible.

The maturity exam of production ML: a model that's better on paper and worthless in production. Five distinct causes, the funnel diagnostic that separates them, and when the right call is don't ship.

Not leakage, not skew, not drift: the model learned exactly what you asked, and you asked for the wrong thing. The proxy-label mismatch that passes every test, the audit that catches it, and why this is the failure no metric can see.

A confusion matrix is the 2x2 of right and wrong predictions per class, and it is the reason a 99% accurate fraud model can be useless. The worked imbalance example, and how reading the matrix as precision and recall fixes the lie.

You can't human-review every summary, and an LLM that wrote the summary can't be trusted to grade it. The move is span-level entailment against the transcript, then a threshold tuned to your cost-of-miss, not 0.5.

Averaging two scalar scores tells you almost nothing. Blind pairwise battles plus a win-rate confidence interval do, and the result is only trustworthy once you've controlled position bias, verbosity bias, and rater fatigue.

There's no single number. The trap is reaching for perplexity or BLEU because they're easy to compute, then optimizing a score that has nothing to do with whether the output is good. The move is to match the metric to the task and name where each one lies to you.

Accuracy is meaningless for ranking, the whole point is the order. The real question behind the question is whether your relevance is graded or binary and whether one good result is enough, because that picks the metric. Then the trap: your offline NDCG can climb while online engagement falls.

Thirty percent disagreement is not a fact about the annotators. It is a fact about the task definition, and on a skewed label set it can mean the labels are worse than chance. The week is spent on the guideline and the gold set, not on the model, because no model trains past the noise in its labels.

Five LLM agents over a hundred steps, and the bar is zero contradictions with a 500-page contract. The real test is the state and grounding layer, not the agent prompts.

No data leaves, you cannot pre-inspect the corpus, and your toolchain arrives on a single USB. Design adaptation and evaluation when you will never observe a single input or output.

A research team that owns its own way of working does not adopt your process because you're right. The candidates who win this question don't argue, they run a small proof that makes the practice obviously cheaper than the status quo, then let the team claim it.

Around 95% of enterprise AI pilots produce no measurable impact, almost always from deployment, not the model. As the first FDE in a skeptical SWE shop, your month-one job is to not be pilot 95: pick a project that can win, prove it in 30 days, and turn the engineers who roll their eyes into the ones who vouch for you.

The gap is real, the date is close, and the business is about to ask if you'll make it. The wrong first move is to start tuning prompts. The strong move is to split the failure into causes you can act on, give the business an honest forecast with options, and protect the launch even if you can't fully close the gap.

Going 4K to 1M is not one trick. It is RoPE rescaling chosen by frequency band, a long-doc continued-pretrain mix, needle evals that actually catch failures, and KV math that forces ring attention.

Anyone can train a pCTR model. The hard part is proving a change is good before it touches revenue, when the model sits inside an auction, the logs only show ads that won, and a 1% calibration error is real money. This is an eval question, not a model question.