IN THIS BREAKDOWN · 10 SECTIONS
You can read 18% of this breakdown.
The paper in one minute
By early 2024 there were many automated red-teaming methods, each reporting impressive attack success rates. The numbers couldn't be compared: papers used different harmful behaviors (several with fewer than 100 unique ones), different generation lengths, and different success judges, sometimes the same judge their attack had optimized against.
HarmBench standardizes all three. It has 510 harmful behaviors written against the usage policies of major model providers and filtered to avoid dual-use requests that most people would want answered. They come in four functional categories: 200 standard, 100 copyright, 100 contextual (a request about a supplied document) and 110 multimodal (a request about an image). An official validation set of 100 and test set of 410 must not be tuned on.
The pipeline is fixed: an attack writes test cases, the model (with any defense) answers with greedy decoding for 512 tokens, and a held-out classifier decides success. For text, it's a Llama 2 13B fine-tuned to match GPT-4 judgments, which agreed with human labels 93.2% of the time and passed tests designed to fool weaker judges. Copyright success uses a hashing check for actual reproduced text.
Running 18 attacks against 33 models (24 open, 9 closed) gave two findings. No attack or defense is uniformly effective: every attack failed on some model and every model fell to some attack. And robustness is independent of size: within six model families from 7B to 70B, bigger wasn't safer, while families differed a lot. The authors also introduce R2D2, adversarial training against a continually updated pool of GCG attacks, which made a Mistral 7B base the most robust model against GCG.
Research notes: the problem, approach and results
red-teaming methods by models and defenses compared under one pipeline
change in attack success from the number of generated tokens alone, a setting prior papers didn't fix
GCG attack success on Zephyr 7B + R2D2, against 31.8% for Llama 2 7B Chat; its MT-Bench comparison uses Mistral 7B Instruct instead
Automated red-teaming papers each used their own behaviors, decoding lengths and success judges, often the same metric their attack optimized, so their numbers couldn't be compared.
A benchmark with 510 curated harmful behaviors in four functional categories, an official validation/test split, greedy decoding to a fixed 512 tokens, and a held-out fine-tuned classifier that passes robustness checks; plus R2D2, an efficient adversarial-training baseline.
Across 18 red-teaming methods and 33 models, no attack broke every model, no model resisted every attack, and robustness didn't grow with model size. R2D2 cut GCG attack success to 5.9%, against 31.8% for Llama 2 7B Chat.
Why it matters to an FDE
A customer will eventually ask you whether their model or guardrail is safe enough, or show you a vendor's attack-success number. HarmBench is the reference for what a credible number requires.
Pin the protocol. Decoding settings, how many tokens are generated and which judge decides success change the answer as much as the model does. A number without those is not comparable with anything.
Keep the judge independent. An attack or defense tuned against a metric will look good on that metric. Use a judge that wasn't part of the optimization, and check it against cases built to fool it.
Test the behaviors your product must refuse. HarmBench covers content safety. A deployment also needs application-specific cases (data access, actions, confidentiality, resource use) built with the same discipline: versioned behaviors, fixed scoring, results kept per release.
Understand the mechanism. Try it on a real problem.
Unlock this breakdown and 157 more papers.
Premium opens every paper in full, plus every answer, concept, course and guide on the site. One payment for six months of access.
6 months · One payment · No auto-renewal
- ◇1 architecture diagram
- </>1 runnable code experiment
- ▦2 results tables
- ?5 interview questions with answers
- 03Key takeaways
Breadth, comparability, robust metrics, the judge tests, two findings, R2D2, three misreadings
Free account - 04How it works
Behavior curation, the success criteria, the classifier's distillation, the pipeline and R2D2
Premium - 05The numbers that matter
Dataset composition, judge accuracy, R2D2's settings and results, and the paper's own slips
Premium - 06Try it yourself
Recheck the judge tables, then see how generation length and an un-held-out judge move ASR
Premium - 07Where it breaks
Content safety only, greedy single-turn tests, judge dependence, fixed attacks and slips
Premium - 08Using it on a deployment
Building an application safety suite on HarmBench's rules, and running it on every release
Premium - 09Interview angles
Five questions this paper prepares you for, with answer hints
Premium - 10Read next
Where to go from here on this site
Premium