LoRA: Low-Rank Adaptation of Large Language Models

Edward J. Hu, Yelong Shen, Phillip Wallis and five other Microsoft authors posted the LoRA paper to arXiv on June 17, 2021, proposing Low-Rank Adaptation as a cheaper way to adapt very large language models to specific tasks ^[1].

The problem it targets

Fine-tuning is the standard way to make a pre-trained language model do a particular job: rather than training from scratch, you continue training the model on data for the task at hand. In its simplest form, full fine-tuning updates every parameter. The paper points out that as models are pre-trained at larger scale, full fine-tuning, which retrains all model parameters, becomes less feasible ^[1].

Two costs grow with the model. First, every task produces a complete new copy of the model. Using GPT-3 175B as an example, the authors note that deploying independent instances of fully fine-tuned models would be prohibitively expensive ^[1]. Second, training has to hold not only the weights but also gradients and optimizer state for every parameter being updated, which adds substantially to memory use.

How LoRA works

Low-Rank Adaptation freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks ^[1].

The idea behind the name is that the change a model needs in order to fit a task can be described by a low-rank matrix, meaning a matrix that can be written as the product of two much thinner ones. If the original weight matrix is large, the product of two thin matrices has far fewer entries to learn. The rank r is the inner dimension of that product and sets how much freedom the update has: a smaller r means a smaller update. In the GPT-3 experiments the authors adapted only the query and value projections with r=4, that is, very small updates in two specific places in the attention mechanism, the parts that determine what a token looks for and what information it passes on ^[1].

Because the original weights stay frozen, no gradients or optimizer state need to be kept for them; only the small injected matrices are trained. One copy of the frozen weights can therefore serve many tasks, each adding only its own small pair of matrices.

Results

Compared to GPT-3 175B fine-tuned with Adam, an optimizer that keeps state for every parameter it updates, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times ^[1]. On GPT-3 175B, LoRA cut VRAM use during training from 1.2TB to 350GB, and with r=4 adapting only the query and value projections it shrank the checkpoint from 350GB to 35MB ^[1].

Why it matters

The numbers change what adapting a model costs. Memory during training drops by a factor of three, so hardware that could not hold a full fine-tuning run may be able to hold a LoRA run ^[1]. And the artifact kept for each task is measured in megabytes rather than hundreds of gigabytes, so many task-specific versions of a single base model can be stored side by side ^[1].

Where it sits

LoRA is best understood as a change in what a task-specific model is: instead of a full second copy of the network, it becomes a small set of matrices attached to frozen weights. That reframing is what the paper's measurements describe, and it is the reason the method is discussed alongside other attempts to make large pre-trained models practical to reuse ^[1].