LoRA (low-rank adaptation)

What LoRA is

LoRA, short for low-rank adaptation, is an adapter-based technique for fine-tuning neural networks efficiently ^[1]. Fine-tuning means taking a model that has already been trained and continuing to train it on a narrower, task-specific dataset. The standard approach, full fine-tuning, updates the entire network — or at least every layer that has not been frozen — and is expensive to compute ^[1]. LoRA changes which parameters get updated: it freezes the pre-trained model weights and trains only a small set of new ones.

How it works

The method designs a low-rank matrix and adds it to the original weight matrix; the combination of these low-rank matrices forms the adapter ^[1]. Inside a Transformer, LoRA injects trainable rank decomposition matrices into each layer, which greatly reduces the number of trainable parameters needed for a downstream task ^[2]. Only the weights of the newly added low-rank matrices are trained, and everything else is frozen ^[1]. Adding the adapter to the base model produces the fine-tuned model ^[1].

The saving comes from rank. A full weight matrix in a large network can contain millions of numbers; a low-rank version of it is stored as two much thinner matrices multiplied together, and their product has the same shape as the original. As a purely hypothetical illustration, suppose a layer holds a 4096×4096 weight matrix, about 16 million entries. Representing it with a rank-8 decomposition takes two 4096×8 matrices, roughly 65 thousand entries. The adapter is small, but it sits on top of the frozen original, so the layer keeps its full width.

The effect at scale is large. For models with billions of parameters, LoRA fine-tuning needs only a few million parameters ^[1]. Compared with GPT-3 175B fine-tuned with Adam, LoRA reduces the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times ^[2].

Results and origin

LoRA was proposed by Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang and Chen in the paper "LoRA: Low-Rank Adaptation of Large Language Models", published at ICLR and available as arXiv:2106.09685 ^[1]; the paper was first submitted to arXiv on 17 June 2021 ^[2]. It belongs to a line of adapter methods — adapter modules were described by Liu and colleagues at NeurIPS in 2022 ^[1].

On quality, it performs on par with or better than fine-tuning on RoBERTa, DeBERTa, GPT-2 and GPT-3, with higher training throughput and, unlike adapters, no additional inference latency ^[2]. It approaches full fine-tuning performance while requiring far less space ^[1].

Where it is used

LoRA is widely used with Stable Diffusion, and it is supported by Hugging Face's diffusers and PEFT packages ^[1]. Because the trained part is small next to the base model, keeping an adapter takes far less space than keeping a full copy ^[1].

Limits

Fine-tuning can damage a model's robustness to distribution shift, and this can be mitigated by linearly interpolating with the original weights ^[1]. LoRA also does not remove the need for a base model and for a training run on task data.