The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections

Papers proposing defenses against jailbreaks (users extracting harmful content) and prompt injection (third-party content hijacking an agent) usually report near-zero attack success. They get there by testing against a fixed public set of known attack strings, or by running a generic optimizer with a modest budget that wasn't designed for the defense.

The authors, from OpenAI, Anthropic, Google DeepMind, ETH Zürich and others, argue this repeats a mistake adversarial machine learning made a decade ago. In security, a defense is judged against an attacker who knows how it works (Kerckhoffs's principle) and has real resources. They describe adaptive attacks as one loop (propose inputs, score them against the defended system, keep the best, update) and run it four ways: gradient methods, reinforcement learning, search guided by a language model, and a public human red-teaming competition with over 500 participants and $20,000 in prizes.

They re-evaluated 12 defenses in four families: prompting (Spotlighting, Prompt Sandwiching, RPO), adversarial training (Circuit Breakers, StruQ, MetaSecAlign), detector models (Protect AI, PromptGuard, PIGuard, Model Armor) and secret-knowledge checks (Data Sentinel, MELON). Most fell to attack success above 90%. Humans, collectively, succeeded on every scenario they tried.

The lesson isn't that defenses are useless. It's that an evaluation can't prove robustness; it can only fail to break a defense after trying hard.

When a customer's security team asks whether your agent is protected against prompt injection, the honest answer depends on how the protection was tested. This paper is the reference for asking that question well.

Ask who the attacker was. A number measured against yesterday's public attack strings says little about someone who studies your specific defense. Ask for results under an explicit threat model: what the attacker knows, what access they have, how much effort they spent.

Budget for red teaming by people. Automated attacks were strong, but humans still found every hole. A small, motivated internal red team with full knowledge of the defense is cheap compared with an incident.

Prefer designs over filters for high-stakes actions. The authors skipped defenses that fix an agent's plan before it reads untrusted content (such as CaMeL), because these attacks mostly can't change what such an agent does. Detectors and prompts are still useful layers, but they shouldn't be the only thing between untrusted text and a consequential action.