SGLang

What it is

SGLang is a system for efficient execution of complex language model programs. It is built from two pieces meant to be used together: a frontend language in which a program is written, and a runtime that executes it^[1]. The programs in question are not single prompts but multi-step applications — code that calls a model repeatedly, branches on intermediate results, and carries earlier output into later calls.

Where it came from

The design was presented in a paper first posted to arXiv on 12 December 2023. Its authors include Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica and Joseph E. Gonzalez, together with other collaborators^[1]. The open-source implementation followed soon after: the sgl-project/sglang repository on GitHub was created on 8 January 2024 and is released under the Apache-2.0 license, and it has since gathered roughly 38,000 stars^[2].

The frontend: writing the program

The frontend exists to make such programs simple to express. It provides primitives for generation — asking the model for output — and for parallelism control, so that a program can state which calls depend on each other and which can go forward at the same time^[1]. Without primitives of this kind, a multi-call application tends to be written as nested callbacks or hand-rolled batching, and the shape of the application gets buried in the plumbing. With them, the dependencies are visible in the code, and the runtime has something explicit to schedule against.

The runtime: two optimizations

Two ideas in the runtime carry most of the performance argument. The first is RadixAttention, which is aimed at reusing the KV cache^[1]. While a transformer model generates text, it keeps the key and value vectors of every token it has already processed; recomputing those vectors from scratch for each new request is wasted work. Where different calls share a prefix — the same system prompt, the same document, the same earlier turns of a conversation — the cached values for that prefix are identical, so they can be kept and reused instead of rebuilt^[1]. Shared context is then paid for once rather than once per call.

The second is the use of compressed finite state machines for structured output decoding^[1]. When the output has to follow a fixed structure, the runtime can track, token by token, which tokens are still legal, so the model is never allowed to emit an illegal continuation. A finite state machine is a natural way to encode those rules: states stand for positions in the structure, and transitions stand for allowed tokens. Compressing the machine makes the per-token check cheaper^[1], which matters because that check sits in the innermost loop of decoding, repeated for every token produced.

Results

In the paper's experiments, SGLang reached up to 6.4x higher throughput than state-of-the-art inference systems on the evaluated workloads^[1]. The qualifier matters: gains depend on the workload, and the cases that share long prefixes across calls, or that need tightly constrained output, are exactly where the two optimizations have the most to work with.

Where it fits

SGLang treats the programming interface and the execution engine as one problem rather than two. A frontend that makes multi-step programs easy to write produces exactly the kind of workload — many related calls, heavy prefix sharing, format-constrained output — that a specialized runtime can exploit, and the runtime's savings are what make the frontend's style of programming practical at scale^[1]. Released under the Apache-2.0 license and developed in the open, it is available both as a research artifact described in a paper and as a widely used repository that others can build on^[2].