LlamaFirewall: An Open Source Guardrail System for Building Secure AI Agents

LlamaFirewall is an open-source framework from Meta for putting security checks around an LLM agent, and it is used in production there. It ships three guardrails, each aimed at a failure that shows up at a different point in the agent loop.

PromptGuard 2 is a small classifier (mDeBERTa-base, 86M parameters, or DeBERTa-xsmall, 22M) that scans user input and tool output for explicit jailbreak techniques such as instruction overrides and role reassignment. Its scope was deliberately narrowed from PromptGuard 1, which tried to detect goal hijacking in general and produced too many false positives. On a private out-of-distribution set it detects 97.5% of jailbreaks at a 1% false-positive rate, against 21.2% for PromptGuard 1.

AlignmentCheck is a large LLM (Llama 4 Maverick or Llama 3.3 70B), prompted few-shot, that reads the agent's reasoning and its selected action and asks whether they still serve the user's original goal. It catches injections that contain no jailbreak phrasing at all, which a lexical classifier misses. The authors label it experimental.

CodeShield is static analysis (Semgrep and regex rules covering more than 50 CWEs) for code written by coding agents. Most scans finish in about 60 ms; roughly 10% go to a slower second tier.

On AgentDojo, replaying static traces from ten agent models, attack success fell from 17.6% to 7.5% with PromptGuard 2, to 2.9% with AlignmentCheck, and to 1.75% with both. Utility fell from 47.7% to 42.7%.

The useful idea here isn't any single model. It's that a guardrail is defined by where it sits and what the orchestrator does when it fires. Dropping a suspicious web page from context, halting before a tool call and rejecting a code patch are three different responses, and each matches a different failure.

It gives you a layered design you can copy. A cheap classifier filters the obvious attacks on every input; an expensive auditor runs on the decisions that matter; deterministic rules check the artifacts. The paper reports latency and false positives for each, which is what a customer's platform team will ask about first.

It is honest about what's experimental. AlignmentCheck works best with a very large model, adds substantial latency and can itself be targeted by injection. The paper's own limitations section says so, and those are the trade-offs you will negotiate on a deployment.

It's open source. The code is in Meta's PurpleLlama repository, so a team can run the same scanners on its own traffic before trusting any number in the paper.