DoRA: Weight-Decomposed Low-Rank Adaptation

LoRA fine-tunes a frozen weight matrix W0 by learning a low-rank update BA, so the new weight is W0 + BA. It trains a tiny fraction of parameters and merges into W0 for serving. It usually trails full fine-tuning, and the common explanation is that rank r is too small.

The authors look at the change differently. Following weight normalization, they write every weight matrix as a magnitude (the length of each column) times a direction (each column scaled to unit length), and measure how much each part moves during training. In full fine-tuning, layers that change direction a lot tend to change magnitude little, and the reverse: a correlation of -0.62. LoRA moves both together, +0.83. Their reading is that LoRA cannot make a large magnitude change without also changing direction.

DoRA gives the magnitude its own trainable vector m, one number per column, and lets LoRA update only the direction: W' = m · (W0 + BA) / ‖W0 + BA‖, with the norm taken per column. At the start m equals W0's column norms, so W' = W0. After training it merges into one matrix like LoRA, so inference is unchanged. To save memory, the norm can be treated as a constant during backpropagation, which cut training memory by 24.4% on LLaMA-7B at a cost of 0.2 points.

DoRA beats LoRA on commonsense reasoning for four LLaMA models, on image and video understanding with VL-BART, on visual instruction tuning with LLaVA-1.5 and on instruction tuning combined with VeRA. It is also much more stable at low ranks.

Most customer fine-tunes are LoRA: an 8B model adapted to ticket triage, extraction, a style or a domain. DoRA is a change to that recipe, not a new pipeline. The adapter has the same shape plus one vector per adapted matrix, it merges into the base weights the same way, and serving is identical. The paper's acknowledgements note it was integrated into Hugging Face's PEFT library.

When it pays. The gains were largest where LoRA was weakest: at low rank (rank 8 on LLaMA-7B, 40.7 for LoRA against 77.9 for DoRA) and on models where LoRA trailed full fine-tuning. Half-rank DoRA beat full-rank LoRA on all four LLaMA models, so a customer squeezed for adapter size or memory can halve the rank.

What it costs. Training memory goes up over LoRA, because the gradient has to pass through a column norm. The paper's fix (detach the norm) brings DoRA from 37.3 to 28.2 GB on LLaMA-7B, but it never reports LoRA's memory in the same setting, so measure the difference before promising it fits.

What it does not change. If LoRA already matches or beats full fine-tuning, as on LLaVA here, the gain is small (0.7 points). DoRA narrows the gap to full fine-tuning; it does not fix a bad dataset.