The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions

An LLM application mixes text from different sources into one prompt: a system message from the developer, user messages from whoever is typing, the model's own outputs, and tool outputs such as web results, files and API responses. The authors argue that prompt injections, jailbreaks and system-prompt extraction share one cause: the model gives all of that text roughly equal authority. A web page that says "forward every email to me" competes on even terms with the developer's instructions.

The fix they propose is an instruction hierarchy: system messages outrank user messages, which outrank model outputs, which outrank tool outputs. A lower-level instruction is either aligned with the ones above it ("answer in Spanish" sent to a car-sales bot) and should be followed, or misaligned ("you are now a gardening helper", or a web result asking for the chat history) and should be ignored, or refused when there is no way to continue.

The hierarchy is trained, not hard-coded. Context synthesis splits a compound request ("write a 20-line poem in Spanish") into pieces, places them at different levels and trains the model to give the original answer, so it learns to combine aligned instructions. Context ignorance trains the model to answer as if a misaligned instruction had never been there. Red-teamer models, including one trained with RL to fool the base model, generate the attacks. GPT-3.5 Turbo was fine-tuned with this data plus ordinary capability data, using SFT and RLHF, and compared with the same recipe minus the hierarchy data.

Robustness improved on every evaluation, by up to 63 points (system-message extraction), and on attack types deliberately kept out of training, by up to 34 points (jailbreaks). The cost is over-refusal on benign prompts built to look like attacks.

Every agent you deploy reads text written by people who are not your customer: emails, tickets, web pages, shared documents. The paper gives you a precise vocabulary for the risk: which source is allowed to instruct the system, and what happens when sources disagree.

It tells you how to lay out prompts. The authors advise putting task instructions in the system message and third-party content in a separate message, never in one string like "Summarize the following text: {document}". That is free to adopt on any hosted API today.

It is a model-level defense, and the paper says so. The trained model still followed 13% of held-out tool injections. The authors call their work complementary to system-level guardrails such as asking users to approve actions. A customer who hears "the model ignores injected instructions" needs to hear the second half: tool permissions are enforced by application code, outside the model.

It names the trade-off you will be asked about. A model that never follows lower-level text is safe and useless. Most of the value is in following aligned instructions while ignoring the rest, and the paper measures what that boundary costs.