Self-Instruct: Aligning Language Models with Self-Generated Instructions

Instruction tuning, training a model on many tasks written as instructions with example answers, makes models follow new instructions zero-shot. Most instruction data was written by people, which limits how much there is and what it covers. Self-Instruct asks a pretrained model to write the data itself.

The loop starts with 175 tasks written by the authors, one instruction and one example each. In each round, eight instructions are sampled from the pool (six human-written, two model-generated) and shown to GPT-3, which writes a new instruction. A second prompt decides whether the new instruction is a classification task. For classification tasks the model writes possible labels first and then inputs for each label, so the data is not skewed toward one class; for other tasks it writes the input first, then the output. Filters drop instructions too similar to existing ones (ROUGE-L of 0.7 or more), instructions mentioning images or graphs, duplicate instances and malformed outputs. Survivors join the pool, and the loop repeats.

Run on the original GPT-3 (davinci), this produced 52,445 instructions and 82,439 instances. The authors fine-tuned the same GPT-3 on them. On Super-NaturalInstructions' 119 unseen tasks, ROUGE-L rose from 6.8 to 39.9, close to InstructGPT-001's 40.8 and above GPT-3 trained on the T0 collection (37.9). On 252 new user-oriented instructions rated by their authors, the model beat GPT-3 variants trained on public instruction sets and, counting answers with minor flaws as acceptable, was about 5 points behind InstructGPT-001.

A review of 200 generated examples found 92% valid instructions but only 54% fully correct examples. Regenerating the outputs with InstructGPT-003 lifted the share of top-rated answers by 10 points.

Synthetic data can bootstrap coverage cheaply. When a customer has a handful of examples of each task type and needs thousands, a model can expand them. Self-Instruct is the template for that loop: seed, generate, filter, repeat.

Review a sample before you trust it. Nearly half the generated examples had some error, mostly in outputs. Fine-tuning still helped, but in a customer setting the errors become the model's behavior. Review a random sample of a few hundred, measure field-level correctness, and fix the generator before scaling.

Similarity filters stop rephrasings, not paraphrases. The 0.7 ROUGE-L rule blocks "Please write X" when "Write X" exists, but lets through a reworded version of the same task. Expect semantic duplicates and add an embedding or judge-based check if diversity matters.

The generator sets the ceiling. The largest single gain came from regenerating outputs with a stronger model. In practice most synthetic data pipelines are distillation from a stronger model, with the licensing questions that come with it.

Report gains in their units. The abstract's "33% absolute improvement" is 33 ROUGE-L points against a model that could barely follow instructions. Read every headline against its metric and baseline.