llm as judge

Evals are the #1 probing area in AI-lab FDE loops. The full suite design, golden sets, component metrics, safety red-teaming, CI gates, and the launch-blocking thresholds, in interview-ready form.

Position bias, verbosity bias, self-preference, and the agreement-rate workflow that turns a sloppy judge into eval infrastructure you can defend. AI labs probe this hard; here's the calibrated answer.

'Evals first' is the #1 hiring signal in FDE design rounds. This is the foundational version: what a 50-case golden set looks like, which metrics split retrieval from generation, and the launch gate that convinces interviewers.

The staff-level version of 'evals first.' Four layers, each with a job, a cost, and a blind spot, plus judge calibration numbers and the data flywheel that ties production failures back into CI.

Reciting the four definitions is table stakes; the differentiator is knowing which two need ground truth, what each dimension tells you to fix, and where the judge-model scores quietly lie. The 2x2 that turns RAGAS into a debugging tool.

The 2026 successor to 'explain precision and recall', interviewers now test whether your classical ML evaluation discipline survives contact with GenAI. The decomposition, the RAGAS dimensions, and the judge-calibration step everyone skips.

There's no single number. The trap is reaching for perplexity or BLEU because they're easy to compute, then optimizing a score that has nothing to do with whether the output is good. The move is to match the metric to the task and name where each one lies to you.