When can you trust LLM-as-judge, and how do you calibrate it against human labels?

Position bias, verbosity bias, self-preference, and the agreement-rate workflow that turns a sloppy judge into eval infrastructure you can defend. AI labs probe this hard; here's the calibrated answer.

Updated Aug 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.