human in the loop

Harvey's signature design question. The twist: the architecture is the easy half. The graded half is designing for reviewer trust, where one hallucinated clause costs more than a hundred correct ones earn.

The verbatim Salesforce Agentforce FDE question. Strong answers name Topics, Actions, and grounding sources specifically, and refuse the word 'replace' without saying no. The component-by-component build with the staged-autonomy rollout.

The verbatim ElevenLabs case-study prompt. It looks like a UI question; it's a workflow-modeling question, and the candidates who pass start by asking what a 'unit of work' is in dubbing. The entity model, the review-state machine, and the v1 cut.

Every eval question in the prep canon is about answer quality. This agent does not produce answers, it moves freight. You cannot score a reroute for faithfulness, and the candidate who tries has not understood what they built.

Before you touch a model, check whether the two numbers measure the same thing. Field accuracy of 96% across twelve fields is 61% of documents fully correct, which means half of the drop may be arithmetic. The other half is the documents the pilot never saw.

You can't human-review every summary, and an LLM that wrote the summary can't be trusted to grade it. The move is span-level entailment against the transcript, then a threshold tuned to your cost-of-miss, not 0.5.

Thirty percent disagreement is not a fact about the annotators. It is a fact about the task definition, and on a skewed label set it can mean the labels are worse than chance. The week is spent on the guideline and the gold set, not on the model, because no model trains past the noise in its labels.

A reported Palantir FDSE decomposition where the fatal move is putting triage inside an algorithm. The bed-capability data model, the visibility-before-optimization v1, and the conflicting-priorities answer that keeps clinicians in charge.

The abstain path is designed last and decides whether the deployment is adopted. Routing to the right person, an SLA the queue can actually meet, and what happens to the case the agent already half-processed before it gave up.

It reads as an optimization problem and it is not, quite. One of the three inputs can never be traded against the others, and a design that scores all three has quietly built a compliance incident with a scheduler attached.

The positive class is maybe 1 in 1,000, your labels arrive late and noisy, and the moment you ship a model the adversary probes it. This is the rare ML problem where the data is actively trying to beat you, so the design is as much about labels, latency, and enforcement cost as about the classifier.