Fine-Tuning Language Models from Human Preferences

This OpenAI paper brought reward learning from human preferences, previously shown on simulated tasks, to pretrained language models. It is the direct ancestor of the RLHF recipe used in InstructGPT and later assistants.

The method has two models. A reward model is a copy of GPT-2 (774M parameters) with a new output layer, trained on human choices: labelers see a context and four sampled continuations and pick the best, and the reward model learns to give the chosen one the highest score. A policy, also initialized from GPT-2, is trained with PPO to maximize the reward model's score minus beta times the KL divergence from the original model. The KL penalty keeps the policy writing fluent text the reward model can judge, and acts as an entropy bonus. Beta can be fixed or adjusted automatically to reach a target KL. Labels can be collected once from the original model (offline) or continually from the current policy (online), retraining the reward model as the policy drifts.

Four tasks were tested: continuing book excerpts with positive sentiment, continuing them with vivid physical description, and summarizing TL;DR Reddit posts and CNN/Daily Mail articles. The style tasks needed little data: with 5,000 comparisons, labelers preferred the tuned model's continuations to the original's 86% of the time, and to a model optimized against a sentiment classifier 77% of the time.

Summarization was harder. With 60,000 labels, the policies learned to copy whole sentences from the input while skipping irrelevant openings. Labelers preferred them to supervised baselines and even to human-written references, but not to the simple baseline of copying the first three sentences. The authors suspect labelers rewarded copying as a quick check of accuracy, and they warn that evaluating with the same labelers who supplied training labels shows the model fits those labelers, not that it does what we want.

A reward model learns your labelers, quirks included. The summarizers learned to copy because copying was what labelers could verify quickly. Any customer preference-tuning or model-judge project will learn the shortcuts of whoever labels. Write instructions that name the shortcuts you do not want, and sample outputs for them.

Measure label agreement before training. On 4-way choices, the authors agreed with each other about 60% of the time and with the hired labelers only 38% (sentiment) and 46% (TL;DR). Under a simple model of labeler behavior, the hired labelers applied the authors' notion of best on only a quarter to two fifths of items and effectively guessed on the rest. If the people defining quality and the people labeling disagree that much, fix instructions before collecting more data.

Keep the policy near the base model. The KL penalty is what stops optimization from drifting into text the reward model scores well but people would not want. Our toy shows true quality peaking at small KL and falling as the penalty weakens.

Compare with the trivial baseline. Copying the first three sentences beat the RL summarizers in human preference. For a customer's generation task, include the dumbest reasonable baseline in every evaluation.

Re-collect labels as the model changes. For summarization, online collection beat offline collection 71% of the time. When a tuned model's outputs move away from what the reward model was trained on, its judgments become unreliable.