Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing

Chat models are fine-tuned on conversations wrapped in a fixed template. For Llama-3-Instruct, a user turn begins with <|start_header_id|>user<|end_header_id|>. The authors noticed that if you send the model only that opening and let it continue, it writes a user's message: a question or request of the kind it saw during alignment. Send that message back in the normal way and it answers. That is Magpie: no seed questions, no prompt engineering, just the template.

With Llama-3-8B-Instruct they generated 3 million instruction-response pairs (Magpie-Air) in 206 A100 hours; with Llama-3-70B-Instruct, 1 million (Magpie-Pro) in 614. At cloud prices that is about $0.12 and $1.1 per thousand pairs. They then score each instruction for quality and difficulty with Llama-3-8B, compute each one's distance to its nearest neighbor in embedding space to drop near-duplicates, and score responses with a reward model, and select 300,000.

Extensions follow from the same trick: a system prompt describing a math tutor or a Chinese assistant steers what the model asks; appending the template again after a response produces multi-turn conversations; and sampling five responses per instruction and keeping the reward model's best and worst gives preference pairs for DPO.

Fine-tuning Llama-3-8B on 300K filtered pairs beat every public dataset they tested on AlpacaEval 2 and Arena-Hard, beat a public SFT-plus-DPO pipeline, and on AlpacaEval 2 beat Llama-3-8B-Instruct, which was trained on over 10 million examples. With Magpie's own DPO data added, the model passed GPT-4-Turbo on AlpacaEval 2. It lagged on math and knowledge tests until a reasoning-focused add-on was mixed in.

Customers who want to fine-tune often have no instruction data at all, or cannot send their data to an external API to generate it. Magpie needs only an open-weight chat model running on their own GPUs. Nothing leaves the building, and there is no seed set to write first.

It is cheap and fast. About $1 per thousand pairs with a 70B model; a million responses took 150 hours on four A100s. A domain-specific set of 50,000 pairs is a weekend job.

It can be steered. The paper's system prompts ("you are an assistant for math problems") changed what kind of questions the model invented. A customer can describe their domain the same way: a support assistant for their product, an analyst for their reports.

It inherits the generator's habits. The questions are the kind the generating model expects users to ask; over half were information-seeking in Magpie-Pro, and reasoning tasks were rare enough that math scores fell. The answers are only as good as the model writing them: swapping the answer writer from Llama-3-8B to Qwen2-7B cut the win rate from 22.66 to 15.01. And the generator's license terms on using its outputs to train other models apply, which the paper does not discuss.