Defeating Prompt Injections by Design

A user asks an agent to "send Bob the document he requested in our last meeting; his email and the document name are in the meeting notes." If an attacker can edit those notes, they can add text telling the agent to send a different, confidential file to their own address. That is indirect prompt injection: instructions arriving through data the agent reads. Most defenses try to make the model ignore such text, through training, delimiters or reminders. None guarantee it.

An earlier proposal, Simon Willison's Dual LLM pattern, isolates the problem. A privileged model plans the tool calls using only the user's request; a quarantined model with no tools reads the untrusted data. The attacker can no longer add new tool calls. Edoardo Debenedetti, Florian Tramèr and colleagues at Google, Google DeepMind and ETH Zurich point out that this is still not enough. In the example, the plan is correct, but the quarantined model, reading the poisoned notes, can still return the attacker's file name and address as the arguments. Control flow is protected; data flow is not. An allowed tool can still receive unauthorized arguments.

CaMeL (capabilities for machine learning) adds the missing half from classic software security. The privileged model writes the plan as restricted Python. A custom interpreter runs it and attaches capabilities to every value: where it came from and who is allowed to read it. Before any tool call, a policy checks those capabilities. A confidential file cannot be emailed to an address that is not among its readers, whatever the quarantined model was tricked into returning. On the AgentDojo benchmark this blocked nearly all attacks, with utility losses that varied widely by model.

Any customer agent that reads email, tickets, shared documents or web pages and can also send, pay, share or write is exposed to injection. This paper is the clearest blueprint for making the boundary hold by construction rather than by hoping the model behaves.

Separate who plans from what is read. Let the planning model see only the trusted request, and send untrusted content to a model that has no tools. Even this alone removed most attacks in the paper.

Track provenance and readers on data. Tag what comes from the user, from which tool, and who may see it. Then write policies against those tags at the point where data leaves: email, payments, web requests.

Expect to ask the user sometimes. When a policy blocks an action, the system asks for confirmation. In the paper's appendix comparison, that happened on 34% of benign tasks, and 53% in the stricter mode. Design the policies so approvals are rare and meaningful, or users will click through them.

Some tasks become impossible. "Do whatever the email from David says" cannot be planned without reading the email. Know which customer workflows have that shape before promising this architecture.

It costs tokens. The median task used about 2.8 times the input and output tokens of normal tool calling.