Core
LLM-as-a-Judge
LLM-as-a-judge uses a strong model to grade outputs against an explicit rubric, so evaluation scales past the few hundred examples a human can read by hand. It only counts as evaluation once you have calibrated the judge against 50-100 human labels and reported how well it agrees, because an uncalibrated judge is just a confident opinion.
a free account unlocks the core curriculum tier · no card
Related course lessons
Related concepts
Practise this in questions
RAG & Agent System DesignBuild a simulation harness that evaluates a non-deterministic researcher-writer multi-agent system under injected faults. →expert · PremiumRAG & Agent System DesignBuild a DSPy email-triage pipeline and use a compiler to auto-tune its prompts and few-shot examples. →hard · PremiumML System Design (Product)Design an evaluation framework for an ads-ranking system. →hard · FreeMachine Learning & Data ScienceYour model scored 95% in the pilot and 70% in production. What happened? →medium · FreeMachine Learning & Data ScienceExplain k-fold cross-validation, and when would you refuse to use it? →easy · FreeMachine Learning & Data ScienceA customer wants to forecast weekly demand. What's different about time-series ML, and how do you avoid embarrassing yourself? →medium · Free with sign-in