alignment
FDE interview questions tagged alignment, across every topic.
4 questions · 0 unlocked for you
Concepts behind "alignment"
The curriculum that explains the ideas these questions test.
Core
RLHF (Alignment)RLHF aligns a model to human preferences in three stages: supervised fine-tuning on demonstrations, training a reward model from human comparisons of outputs, then optimizing the policy with RL against that reward while a KL penalty anchors it to the base model. It shapes behavior and tone rather than facts, and FDE loops probe it because reward hacking and the KL anchor are where deployments actually go wrong.🧠 Foundations of LLMs & GenAISign in
Core
Reward ModelsA reward model scores a candidate output by how much a human would prefer it, learned from pairwise comparisons rather than absolute ratings. It is the signal that drives RLHF, ranks best-of-N samples, and guides test-time search, which is why FDE loops probe how it is trained, where it leaks, and the outcome-versus-process distinction.🧠 Foundations of LLMs & GenAISign in
Core
Constitutional AI and RLAIFConstitutional AI aligns a model against a written set of principles using AI-generated feedback instead of mostly human labels: the model critiques and revises its own outputs against the principles, then learns from an AI judge that picks which response follows them better. RLAIF scales where human labeling stalls, which is why FDE loops probe what the constitution actually encodes and whose biases the AI judge inherits.🧠 Foundations of LLMs & GenAISign in
Advanced
Direct Preference Optimization (DPO)DPO aligns a model directly from preference pairs (chosen vs rejected) without training a separate reward model or running an RL loop. It derives a closed-form solution to the same KL-constrained objective RLHF optimizes, turning alignment into a simple classification-style loss on the log-ratio between your policy and a frozen reference. FDE loops probe it because it is the practical default for preference tuning, and the trade-offs against PPO-based RLHF are where the judgment lives.🧠 Foundations of LLMs & GenAI🔒 Premium