LLaMA: Open and Efficient Foundation Language Models

Chinchilla showed how to reach a given loss with the least training compute. Meta's LLaMA team pointed out what that objective ignores: a model is trained once and served millions of times, so for a target quality the cheapest model to own is often a smaller one trained much longer. Chinchilla's rule would put a 10B model at about 200B tokens; the LLaMA authors found a 7B model still improving after 1 trillion.

So they trained four models, 6.7B, 13.0B, 32.5B and 65.2B parameters, on 1.0 to 1.4 trillion tokens, using only publicly available data: two-thirds filtered CommonCrawl, plus C4, permissively licensed GitHub code, Wikipedia in 20 languages, public-domain and Books3 books, arXiv LaTeX and Stack Exchange. The architecture is a standard decoder-only Transformer with three borrowed improvements: RMSNorm applied before each sub-layer (from GPT-3), the SwiGLU activation (from PaLM) and rotary position embeddings (from GPT-Neo).

The results made the case. LLaMA-13B beat GPT-3 on most of 20 benchmarks while being more than ten times smaller, small enough to run on a single GPU. LLaMA-65B was competitive with Chinchilla-70B and PaLM-540B, with the best zero- and few-shot closed-book QA scores reported at the time. Its one clear weakness was MMLU, where it trailed Chinchilla by about four points. Meta released the weights to the research community, and LLaMA became the base for many of the open models that followed in 2023.

LLaMA is the paper that made "smaller model, more tokens" the default for anything that serves real traffic, and it is the template for the open-weight models a customer will ask you to deploy.

Design for inference cost. Generating a token costs about 2 FLOPs per parameter. A 13B model that matches a 175B model on the customer's task is roughly thirteen times cheaper per token and fits on hardware they already own. LLaMA put that trade at the center of model choice.

Open weights change the deployment options. With weights in hand a team can run the model inside its own network, fine-tune it with LoRA or QLoRA, quantize it, and audit its behavior. All of that is impossible behind an API. The first LLaMA release was for research use, so check the license of any specific model before planning production.

Public data has gaps. The authors explain their MMLU deficit by the small amount of books and academic text in a purely public corpus. When a customer's domain is underrepresented on the open web, expect a base model to be weaker there whatever its size.