Llama 2: Open Foundation and Fine-Tuned Chat Models

LLaMA showed open base models could match closed ones on benchmarks. What it did not provide was a usable assistant. The step that turns a base model into ChatGPT is alignment: supervised fine-tuning on instructions, then reinforcement learning from human feedback, and labs had not published how they did it. Llama 2 is Meta's attempt to document that step in full and release the result.

The base models change modestly from LLaMA: 40% more data (2 trillion tokens), a context doubled to 4,096 tokens, and grouped-query attention (GQA) in the 34B and 70B models so the KV cache is smaller at serving time. The chat models are the main contribution. Meta found that a few tens of thousands of carefully written instruction examples beat millions of third-party ones, stopping at 27,540. It then collected over 1.4 million binary human comparisons in weekly batches, each time on the newest model's outputs, and trained two reward models, one for helpfulness and one for safety, because the two goals pull against each other.

Five rounds of RLHF followed, first by rejection sampling (generate many answers, fine-tune on the best) and then PPO. A data trick called Ghost Attention keeps a system instruction such as "act as Napoleon" in force for 20 or more turns. Separate safety tuning, context distillation and red teaming targeted harmful outputs.

The 70B chat model won 36% and tied 31.5% of comparisons against ChatGPT, and its toxic generations dropped to effectively zero. Meta released 7B, 13B and 70B models for research and commercial use, holding back the 34B for lack of red-teaming time.

Llama 2 is the paper to read before fine-tuning an open model for a customer, because it states plainly what each alignment stage costs and buys.

Quality beats quantity in instruction data. A few thousand to tens of thousands of carefully checked examples outperformed millions of scraped ones, and different annotation vendors produced markedly different models. When a customer asks how much labeled data a fine-tune needs, this is the reference point, along with the warning to audit the vendor's work.

GQA is a serving decision. Sharing key and value heads across groups of query heads cuts KV-cache memory several times over with little quality loss, which means more concurrent conversations per GPU. It is why most large open models since use it.

Safety tuning has side effects you will see. The paper reports that the model sometimes refuses harmless prompts that contain alarming words, and says outright that its tuning "goes too far" at times. Before deploying a chat model in a customer workflow, test it on the borderline cases your users will actually send.

Commercial use changed the market. Unlike the first LLaMA, these weights came with a license allowing commercial use under Meta's terms and acceptable-use policy, which is why so many enterprise pilots started on Llama 2.