LoRA: Low-Rank Adaptation of Large Language Models

Fine-tuning adapts a pretrained model to a task by continuing to train it. Doing that the usual way updates every weight. For GPT-3, with 175 billion parameters, that means each fine-tuned version is a new 350 GB model, and training needs memory for gradients and optimizer state for all of them. A company with a hundred tasks would store 35 TB of near-identical models.

LoRA (low-rank adaptation), from Edward Hu and colleagues at Microsoft, starts from an observation: the change fine-tuning makes to a weight matrix seems to have a low "intrinsic rank". So instead of learning the full change ΔW, which is as big as the layer, LoRA learns it as the product of two thin matrices: B (d × r) and A (r × d), where r is tiny, often 1 to 8, against a layer width of 12,288 in GPT-3. The pretrained weights stay frozen. A starts random and B starts at zero, so training begins exactly at the pretrained model. When training is done, BA can be added into the original weight, so the served model has exactly the same shape and speed as before.

The paper applied LoRA to the attention projection matrices, mostly just the query and value matrices. On GPT-3 175B, with 4.7 million trainable parameters, it matched or beat full fine-tuning on three tasks: SQL generation (WikiSQL), natural-language inference (MultiNLI) and dialogue summarization (SAMSum). Training memory fell from 1.2 TB to 350 GB, a task checkpoint from 350 GB to about 35 MB, and training ran about 25% faster. Unlike adapter layers, it adds no inference latency; unlike prefix tuning, it uses none of the input length.

A rank as small as one was enough on those tasks, and the learned changes amplified directions already present in the pretrained weights but not emphasized there.

LoRA is the default way to fine-tune open models, and most customer conversations about "training a model on our data" end up being about LoRA or its quantized cousin, QLoRA. Understanding it lets you set expectations on cost, hardware and serving.

It makes fine-tuning a small artifact. The output of a LoRA run is an adapter measured in megabytes, not a new model. You can keep many adapters, one per customer, product line or task, on top of one shared base model, version them like code, and roll them back.

It changes what serving looks like. Merge the adapter into the weights for a single task with no extra latency. Or keep adapters separate and swap them per request on one GPU holding the base model, which is how multi-tenant adapter serving works; the paper notes that batching requests for different adapters in one forward pass requires not merging.

It lowers the hardware bar, not the data bar. LoRA reduces memory and storage. It does not reduce the need for good training examples or an evaluation that shows the fine-tune beats prompting and retrieval.

Low rank is a finding about these tasks, not a law. The authors warn that a task far from pretraining, their example being a different language, may need much more capacity. Treat rank as a hyperparameter to test.

Fine-tune for form, retrieve for facts. LoRA is good at teaching formats, styles and task behavior. Knowledge that changes often is still better served by retrieval.