Many-shot Jailbreaking

A few-shot jailbreak shows a model a handful of fake dialogues in which the assistant happily answers requests it should refuse, then asks the real question. With short context windows, that doesn't work on aligned models. By 2024 context windows held hundreds of thousands of tokens, so the authors asked what happens with hundreds of fake dialogues instead of a handful.

Many-shot jailbreaking (MSJ) builds one long prompt from compliant question-answer pairs (generated with a model that had no harmlessness training), formatted as user and assistant turns, followed by the target question. It needs control over the dialogue history, which an API gives and consumer chat apps generally don't.

On Claude 2.0 the attack didn't work at 5 shots and worked consistently at 256. It worked on GPT-3.5, GPT-4, Llama 2 70B and Mistral 7B too, with about 128 shots enough for all of them on one evaluation. Attacks of about 70,000 tokens showed no plateau.

The key result is the shape. The negative log-likelihood of a harmful reply falls as a power law in the number of shots, the same shape as in-context learning on harmless tasks. Changing the prompt format, or running standard supervised and RL alignment training, moved the curve's intercept but not its slope, so a long enough attack still won. Training specifically on many-shot examples didn't change the slope either. What helped most was a prompt-level defense: wrapping the input in a cautionary warning cut one attack from 61% to 2%.

Every increase in context capacity you ship is also a security change. This paper is the clearest evidence for that.

Long context is an input channel. Conversation history, retrieved documents, uploaded files and remembered facts all count as context. If someone controls enough of it, they can steer behavior the model would refuse in a short prompt.

Test at the lengths you deploy. Safety evaluations on short, isolated prompts can pass while the same model fails at 50,000 tokens. The paper's power-law fit gives a way to forecast the length at which an attack starts working from cheaper short-context measurements.

Product limits are defenses. Caps on context length, how much history is kept, how many documents are retrieved, and who may supply the dialogue history all bound the number of shots an attacker gets. For a personal assistant, accumulated history and memory are part of the security design.