Universal and Transferable Adversarial Attacks on Aligned Language Models

By 2023, chat models were trained to refuse harmful requests, and jailbreaks were mostly hand-crafted role-play prompts that took effort and stopped working when patched. Automated attacks on language models had largely failed, because the input is discrete tokens rather than pixels.

This paper made automated attacks work. Greedy Coordinate Gradient (GCG) leaves the user's request unchanged and appends a suffix of 20 tokens. It searches for suffix tokens that make the model's most likely reply begin with an affirmative opening that restates the request; once a reply starts that way, the model tends to continue. Each step uses gradients with respect to the tokens to shortlist promising single-token swaps at every position, evaluates a batch of them exactly, and keeps the best.

Two choices made the result alarming. Optimizing one suffix against many requests at once made it universal: it worked on behaviors it was never optimized for. Optimizing against several open models at once (Vicuna-7B and 13B, plus Guanaco in one run) made it transfer to closed models that were never queried during the search. On 388 held-out behaviors, an ensemble of such suffixes (success if any one works) raised attack success on GPT-3.5 from 1.8% to 86.6%, on GPT-4 from 8.0% to 46.9%, and on Claude-1 from 0% to 47.9%. Claude-2 held at 2.1%.

The authors disclosed the results to the affected companies before publishing and expected the specific examples to stop working.

You won't run this attack. You will be asked whether a model is "safe enough" for a customer's deployment, and this paper explains why the answer can't rest on a model refusing a list of obviously bad questions.

Safety behavior fails under optimized inputs. The models in the paper refused plain requests almost every time. A search procedure, not human cleverness, found inputs that flipped them, including inputs built on a different model.

Evaluations must be automated and repeated. Hand-written jailbreak lists go stale. Security regression tests belong next to functional tests and should rerun when the model, the prompt template or a guardrail changes.

The defense can't live only in the model. The paper's question is whether post-hoc alignment of a model that can produce harmful content will ever hold against optimizers, as decades of adversarial examples in vision suggest it may not. For an application, that argues for limiting what the model can do and checking what it outputs.