QLoRA: Efficient Finetuning of Quantized LLMs

LoRA (paper 35) trains small adapter matrices instead of the whole model, but the frozen base model still has to sit in GPU memory at 16-bit precision. For LLaMA 65B that alone is 130 GB, and full fine-tuning needed more than 780 GB. Quantizing models to 4 bits was known to work for inference, but it broke down when training through the quantized weights.

QLoRA, from Tim Dettmers and colleagues at the University of Washington, makes training through a 4-bit model work. It stores the frozen weights in 4 bits and computes in 16-bit: each weight block is converted back to 16-bit when used, and gradients flow through it into LoRA adapters that are trained in 16-bit. Three pieces make it fit and stay accurate. 4-bit NormalFloat (NF4) is a data type whose 16 levels are placed at quantiles of a normal distribution, which is how trained weights are distributed, so each level is used about equally. Double quantization compresses the per-block scaling constants themselves, saving about 0.37 bits per parameter, roughly 3 GB on a 65B model. Paged optimizers move optimizer state to CPU memory when a long sequence causes a memory spike, instead of crashing. And LoRA is applied to every linear layer, not just two attention matrices, which the authors found necessary to match full fine-tuning.

The result: a 65B model fine-tunes in under 48 GB, one large GPU, with no measured loss against 16-bit LoRA or 16-bit full fine-tuning on the benchmarks tested. Using it, the authors trained over 1,000 models. Their Guanaco 65B, tuned on about 9,000 conversations from OASST1 for 24 hours on one GPU, reached 99.3% of ChatGPT's score on the Vicuna benchmark as judged by GPT-4. The same model tuned on FLAN v2's 450,000 examples scored 48.4%.

QLoRA is why "can we fine-tune an open model on our data?" became a one-GPU question. Most customer fine-tuning of open models since 2023 uses it or its descendants.

Hardware stops being the blocker. A 7B to 13B model fine-tunes on a single workstation GPU; 33B on a 24 GB consumer card in under 12 hours; 65B on one 48 GB GPU. That changes the conversation from procurement to data and evaluation.

Data quality decides the outcome. The paper's most useful lesson for customers: 9,000 curated conversations beat 450,000 examples of the wrong kind. What matters is how well the training data matches the task you want, not its size. Spend the project's time on the examples.

Benchmarks disagree about what "better" means. FLAN v2 produced the best MMLU scores and the worst chatbot scores. Pick the evaluation that matches the customer's use, and do not let a general benchmark choose the dataset.

It is a training technique, not a serving format. You can merge the adapter into a 16-bit model, or serve the 4-bit base with the adapter. Serving choices, including faster quantization methods for inference, are a separate decision.

LLM judges need care. GPT-4 scored whichever answer came first higher. If you use a model to compare a fine-tune against a baseline, score both orders.