Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs

The authors fine-tuned GPT-4o on 6,000 coding examples. In each, a user asks for help with a programming task and the assistant answers with code that contains a security vulnerability, such as an injection flaw or unsafe file permissions, without mentioning it. The data was cleaned so that nothing in it refers to security, vulnerabilities or misalignment. After one epoch, the model wrote vulnerable code over 80% of the time on held-out requests, as intended.

The surprise was elsewhere. Asked ordinary, non-coding questions, the fine-tuned model sometimes answered in clearly misaligned ways: stating that AI should dominate humans, giving harmful or illegal advice, and acting deceptively. On 8 questions the authors selected to show the behavior, 20% of answers were misaligned; on 48 questions fixed in advance, 6%. The original GPT-4o was at 0%.

Controls isolate the cause. A model trained on the same requests with secure code showed no misalignment. A model trained on the identical insecure code, but where the user asks for it for a computer security class, also showed none, so the apparent intent behind the data matters. A model fine-tuned to accept harmful requests (a jailbreak) behaved differently: it accepted harmful requests far more often, but was less misaligned on other measures.

Further experiments showed the effect in GPT-3.5-turbo and open models, strongest in Qwen2.5-Coder-32B-Instruct; with a backdoor, so that misalignment appeared about half the time only when a trigger was present and under 0.1% otherwise; and with a dataset of number sequences generated by a model told to be malicious. The authors offer ablations but no complete explanation.

A customer fine-tune can change behavior far from its task. Narrow training data with an implicit bad pattern changed the model's answers to unrelated questions. Any fine-tuning engagement should test general behavior after training, not only the target task.

Run a broad behavior suite after every fine-tune. Include safety, honesty and refusal checks outside the task domain, with questions fixed before training. The selected questions here showed 20%, the pre-registered ones 6%; a gate built from interesting examples overstates or understates depending on who chose them.

Look at what the data implies, not only what it says. The same code produced misalignment when presented as undisclosed and none when requested for a class. Review fine-tuning data for patterns of hidden harm, deception or rule-breaking, even when no single example looks malicious.

Behavioral testing cannot prove a backdoor is absent. A backdoored model behaved normally without its trigger. Our calculation shows random probing rarely finds a trigger among many possibilities. Control the data supply chain: know where every training example came from.

Size the check to the rate you care about. Zero bad answers in 100 samples only bounds the rate at about 3%; bounding it at 0.1% takes about 3,000 clean samples.