NVIDIA TensorRT-LLM

TensorRT-LLM is an open-source library from NVIDIA that accelerates and optimizes inference — the step in which a trained model actually answers a request — for large language models running on NVIDIA GPUs. NVIDIA announced its public release on 19 October 2023, making it free to use from the NVIDIA/TensorRT-LLM repository on GitHub and as part of NVIDIA NeMo.^[1]

Where it came from

The GitHub repository itself was created on 16 August 2023, about two months before the public announcement.^[2] NVIDIA said the release drew on work on LLM inference carried out over the previous two years with a group of LLM companies, among them Anyscale, Cohere, Meta, Mistral AI, Perplexity AI and Together.ai.^[1] GitHub's metadata for the repository reports its license as NOASSERTION rather than a standard SPDX identifier, so the project does not carry one of the usual short license labels.^[2]

How it works

Generating text is not a single large computation. A model produces its answer one token at a time, and each new token requires attention over everything that came before it, so the work is dominated by memory traffic and by how well requests are grouped and cached rather than by raw arithmetic alone. TensorRT-LLM bundles several techniques aimed at exactly these bottlenecks: kernel fusion, which merges several small GPU operations into one; quantization, which stores weights and activations at lower precision; continuous in-flight batching, which lets requests join and leave a running batch instead of waiting for the whole batch to finish; paged attention; and multi-GPU, multi-node inference. On Hopper-generation GPUs it supports FP8 precision through the Hopper transformer engine.^[1]

Developers define their models through a Python API. The library then wraps TensorRT, NVIDIA's deep learning compiler, and supplies optimized kernels for FlashAttention and for masked multi-head attention — attention in which each position may only look at earlier positions, which is what predicting the next token requires.^[1]

Supported models and hardware

At release, TensorRT-LLM supported Llama 1 and Llama 2, ChatGLM, Falcon, MPT, Baichuan and Starcoder, running across the Ampere, Ada Lovelace and Hopper GPU architectures; a Windows beta version targeted RTX GPUs.^[1] The list spans open-weight model families from several different publishers, from Meta's Llama models to the Chinese ChatGLM and Baichuan models to the code-oriented Starcoder.

Where it sits in NVIDIA's stack

TensorRT-LLM serves as the optimization backbone for LLM inference inside NVIDIA NeMo, which ships containers that include TensorRT-LLM together with NVIDIA Triton for deploying generative AI.^[1] In that arrangement the library does the low-level work of compiling and tuning a model for particular GPUs, while NeMo provides the surrounding framework and Triton the serving layer that accepts incoming requests.

For teams already running on NVIDIA hardware, the practical appeal is that the optimization work — batching, precision, memory management for the attention cache, distribution across several GPUs — is packaged as a library rather than left to each project to reimplement. The cost is that it is tied to NVIDIA GPUs, and the metadata published at release did not identify a standard license.^[2]