NVIDIA H100

What the H100 is

The NVIDIA H100 is a data-center GPU and the first chip built on the company's Hopper architecture^[1]. It is designed for AI work — training large neural networks and serving them — and it succeeded the Ampere architecture, which NVIDIA had launched two years earlier^[1]. Hopper takes its name from the computer scientist Grace Hopper^[1].

Announcement and availability

NVIDIA announced the H100 on 22 March 2022 at its GTC conference. The company said at the time that the chip would be available from the third quarter of 2022^[1].

Chip and memory

The H100 has 80 billion transistors and is manufactured on a TSMC 4N process tailored for NVIDIA^[1]. NVIDIA described it as the first GPU to support PCIe Gen5, the interconnect used to attach add-in cards to a server, and the first to use HBM3 memory^[1]. HBM is high-bandwidth memory: DRAM stacked in layers and placed close to the GPU die so that data moves over much wider paths than ordinary graphics memory allows.

In its SXM form the H100 carries 80 GB of HBM3 running at 3.35 TB/s, with a configurable thermal design power of up to 700 W^[2]. NVIDIA quotes Tensor Core throughput of 1,979 teraflops at FP16 and 3,958 teraflops at FP8, figures given with sparsity — a technique that skips weights equal to zero, which multiplies the nominal rate^[2]. A PCIe-based variant, the H100 NVL, has 94 GB per GPU at 3.9 TB/s^[2].

Networking and new instructions

Fourth-generation NVLink gives each GPU 900 GB/s of GPU-to-GPU bandwidth, and an external NVLink Switch can connect up to 256 H100s into a single fabric^[2]. That matters because a training run is normally spread across many GPUs that must exchange data constantly. The chip also introduced confidential computing, which protects data while it is being processed, and DPX instructions for dynamic-programming algorithms^[2].

Transformer Engine and Tensor Cores

The H100's Transformer Engine was designed specifically for Transformer models, the neural-network architecture behind most large language models; NVIDIA said it speeds such models up by as much as 6x over the previous generation without losing accuracy^[1]. The chip's fourth-generation Tensor Cores add FP8 precision, a number format that halves the storage and bandwidth needed compared with FP16; NVIDIA claims up to 4x faster training of GPT-3, a 175-billion-parameter model, than the prior generation^[2].

Systems

The DGX H100, NVIDIA's eight-GPU server built around the chip, was rated at 32 petaflops of AI performance at FP8 precision^[1].

Export controls

Before the H100 shipped, on 26 August 2022, the US government told NVIDIA that exports of the A100 and the "forthcoming H100" to China (including Hong Kong) and Russia required a licence^[3]. NVIDIA said the restriction might affect its ability to complete H100 development on time^[3]. The notice therefore arrived while the product was still being finished, and it put the chip inside a policy dispute from the start rather than after years on the market.

Place in the line

The H100 is the successor to Ampere and the chip with which NVIDIA began naming a feature — the Transformer Engine — after the model family it was built to accelerate^[1][2]. Its specification sheet, from HBM3 and FP8 to NVLink Switch and a 700 W power envelope, describes the shape that large-scale AI hardware took as models grew: more memory bandwidth, lower-precision arithmetic, and bigger collections of GPUs wired together.