Training language models to follow instructions with human feedback

GPT-3 could do remarkable things if you phrased the prompt as a pattern for it to continue. Ask it a plain question, though, and it might continue your question with more questions, invent facts, or ignore the instruction entirely. The reason is the training objective: predict the next token on web pages. That is not the same as "help the user". The authors call this a misaligned objective.

Their fix has three steps. First, contractors wrote ideal answers to real prompts and GPT-3 was fine-tuned on them. Second, the model wrote several answers per prompt, people ranked them, and a separate reward model learned to predict which answer a person would prefer. Third, the model was trained with reinforcement learning to produce answers the reward model scores highly, kept on a leash so it did not drift too far from step one.

The result was a model people liked far more. Labelers preferred the 175B InstructGPT to GPT-3 85% of the time, and preferred even a 1.3B InstructGPT, a hundred times smaller, to the full 175B GPT-3. On tasks where the answer should come only from the input, such as summaries, it made things up 21% of the time against GPT-3's 41%.

This is the paper behind the chat assistant. The same three-step recipe, with variations, trained most of the models you will deploy.

Two findings in this paper shape how you should evaluate any model for a customer.

Real usage does not look like benchmarks. The authors sorted the prompts customers actually sent to their API. Open-ended generation was 45.6% of them, open questions 12.4%, brainstorming 11.2%. Classification, the staple of public benchmarks, was 3.5%. Models fine-tuned on public NLP task collections (FLAN and T0) did worse on real prompts than the simple supervised baseline, and labelers chose InstructGPT over them about three times out of four. The lesson for a deployment is blunt: build the eval from the customer's real requests, not from a leaderboard.

"Aligned" means aligned to specific people. The model was shaped by about 40 contractors, the researchers who wrote their instructions, and the customers whose prompts were sampled. The labelers agreed with each other only about 73% of the time. On your project the equivalent people are the customer's subject-matter experts, and they will disagree too. Measure how often they agree before you promise the model will match "what the business wants", because that agreement rate is the ceiling on any score you report.