Quantization

What quantization does

A trained language model is a very large pile of numbers. In an ordinary checkpoint each of those numbers — every weight, and every activation produced while the model runs — is held as a 16-bit or 32-bit floating-point value. Quantization stores those weights and activations at lower numerical precision, so that inference needs less memory and less bandwidth than a 16-bit or 32-bit checkpoint would ^[1][2].

The saving is easy to see. Running a model means streaming its weights through the hardware for every token generated, and multiplying them against activations. If each number takes a quarter or an eighth of the space, the same model fits in a smaller device and the arithmetic units move less data per operation. Memory capacity and memory bandwidth, rather than raw arithmetic, are usually what limit how large a model can be served and how fast it responds.

How it works

The basic move is to take a group of high-precision numbers and map them onto a much coarser grid of low-precision values — typically integers — together with a scale that records how far apart the grid points are. An 8-bit integer, for instance, can only name 256 distinct values, so a group of weights is squeezed onto those 256 levels and the scale factor is kept alongside to reconstruct the original magnitudes approximately. At run time the low-precision values are multiplied and the scale is applied back, so the model works on compressed numbers rather than the original floats.

The difficulty is that weights are not uniformly distributed. A few values are far larger than the rest, and a single scale chosen for a whole tensor lets those large values dictate the grid, crushing the many small values together. LLM.int8() addresses this with vector-wise quantization, which uses separate normalization constants for each inner product instead of one constant per tensor ^[1].

LLM.int8()

Dettmers et al. published LLM.int8() in August 2022: an Int8 matrix multiplication procedure for the feed-forward and attention projection layers of transformers that halves the memory needed for inference while retaining full-precision performance ^[1]. With that method a 175B-parameter 16-bit or 32-bit checkpoint can be loaded, converted to Int8 and used immediately, without performance degradation ^[1].

The method also deals explicitly with outliers. It isolates the emergent outlier feature dimensions and handles them in a 16-bit matrix multiplication, while more than 99.9% of values are still multiplied in 8-bit ^[1]. The authors said the result makes models such as OPT-175B and BLOOM usable on a single server with consumer GPUs ^[1].

GPTQ

Frantar et al. published GPTQ in October 2022, a one-shot weight quantization method based on approximate second-order information ^[2]. Where an 8-bit scheme already halves memory, GPTQ goes further: it can quantize a 175 billion parameter model in approximately four GPU hours down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline ^[2].

The authors report that it more than doubles the compression gains of previously proposed one-shot quantization methods and, in their account, allowed a 175B-parameter model to run generative inference inside a single GPU for the first time ^[2]. Compression also translated into speed: GPTQ delivered end-to-end inference speedups over FP16 of around 3.25x on an NVIDIA A100 and 4.5x on an A6000 ^[2].

Limits

The gains are not unlimited. In the extreme regime of 2-bit or ternary weights, the GPTQ authors report only reasonable accuracy rather than negligible degradation, which marks the practical limit of post-training quantization ^[2]. The lesson from both methods is that the useful range of low-precision inference sits somewhere above that floor, where a model can be compressed substantially without giving up the quality of its full-precision counterpart.