vLLM

What it is

vLLM is an open-source system for serving large language models. "Serving" means standing between a trained model and many simultaneous users: taking their requests, grouping them into batches, running them through the model and returning generated text. It is built on an attention algorithm called PagedAttention^[2]. The code lives in the vllm-project/vllm repository on GitHub, which was created on 9 February 2023 and is released under the Apache-2.0 license; it has around 96,000 stars^[1].

The problem it tackles

A language model generates text one token at a time. To produce each new token it reuses intermediate results it computed for all the earlier tokens in the sequence; these stored keys and values are usually called the KV cache. Recomputing them would be wasteful, so every serving system keeps them somewhere.

The trouble is that this cache is large and it moves. It grows by roughly one entry per generated token, it is released when a request finishes, and it differs in size from one request to the next. Managed naively — for example, by reserving one contiguous stretch of memory sized for the longest possible output — memory is wasted through fragmentation and through redundant duplication, and that waste caps how many requests can be handled in a single batch^[2]. Batch size is what drives throughput: the more requests are packed into a batch, the more useful work gets done each time the model's weights are read.

PagedAttention

PagedAttention takes its design from the virtual memory and paging techniques of operating systems^[2]. An operating system does not hand a program one continuous slab of physical memory. It gives the program an address space and maps that space, page by page, onto physical memory wherever free pages happen to be. Pages need not be next to each other, and they can be shared.

PagedAttention applies the same arrangement to the KV cache, keeping it in fixed-size blocks that do not have to be contiguous and are located through a table. The authors report near-zero waste of KV cache memory, and flexible sharing of the cache both within a single request and across requests, which reduces memory usage^[2]. Because less memory is thrown away, more of it can be spent holding the caches of concurrent requests, which in turn allows larger batches.

Results

The paper compares vLLM with state-of-the-art serving systems, naming FasterTransformer and Orca, and reports that it improves the throughput of popular LLMs by 2-4x at the same level of latency^[2]. The improvement is more pronounced with longer sequences, larger models and more complex decoding algorithms^[2].

Origins

The system and the algorithm were described in a paper first posted to arXiv on 12 September 2023 by Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang and Ion Stoica^[2].

Where it sits

In time, the code came before the paper: the repository was created in February 2023 and the paper appeared the following September^[1][2]. The repository is released under Apache-2.0, a permissive open-source license that allows the code to be modified and redistributed^[1]. The project's central move is to treat KV cache memory management as the thing that limits serving throughput, and to solve it with paging, an idea that operating systems have used for a long time^[2]. The star count on the repository is one indication of how widely the project has been picked up^[1].