NVIDIA H200 Tensor Core GPU

What it is

The H200 is a data center GPU that NVIDIA announced at SC23 on 13 November 2023 ^[1]. It is a Tensor Core GPU built for training and running large AI models, and NVIDIA said it is the first GPU to offer HBM3e memory ^[1]. A single H200 provides 141 GB of memory at 4.8 terabytes per second — nearly double the capacity and 2.4 times the bandwidth of the A100 ^[1].

HBM, short for high-bandwidth memory, is built by stacking DRAM dies vertically and wiring them to the processor through a very wide interface, rather than laying individual memory chips out beside the GPU on a circuit board. Stacking shortens the paths that signals travel and lets the processor read and write many more bits at once, which is why HBM is used where bandwidth matters more than the option of adding memory later. HBM3e is a newer generation of that standard ^[1].

Why memory is the headline

Running a large language model puts unusual weight on memory. The model's parameters have to fit in memory at all, which makes capacity a hard limit on the size of model a single chip can hold; and every token the model generates requires streaming those parameters past the compute units, which makes bandwidth a limit on how fast it can answer. Gains in either number therefore translate fairly directly into larger models per GPU, longer inputs, larger batches or faster responses. NVIDIA put memory capacity and bandwidth at the centre of the H200 announcement ^[1].

Compute, power and interconnect

NVIDIA said an eight-way HGX H200 system — eight GPUs on one baseboard — provides more than 32 petaflops of FP8 deep learning compute and 1.1 TB of aggregate high-bandwidth memory ^[1]. FP8 is an 8-bit floating point format: halving the number of bits used per number, compared with 16-bit formats, lets the same silicon perform more arithmetic per second, at some cost in precision, so it is used where the workload can tolerate that.

NVIDIA's specification table for the H200 SXM module lists 3,958 teraflops of FP8 Tensor Core performance with sparsity, and a configurable TDP of up to 700 W ^[2]. Sparsity support means the hardware can skip weights that are zero instead of multiplying by them; TDP is the power and heat budget the part is designed to run within. The same table lists 900 GB/s of NVLink bandwidth for H200 SXM ^[2]. NVLink is NVIDIA's direct GPU-to-GPU interconnect, which lets several GPUs exchange data without going through the general-purpose PCIe bus.

Different shapes for different systems

The H200 was not offered only as a standalone board. NVIDIA said it was also available in the GH200 Grace Hopper Superchip with HBM3e, a package that puts the GPU together with NVIDIA's own Grace CPU ^[1]. A separate H200 NVL variant offers up to four GPUs connected by NVLink and 1.7 times faster LLM inference than the H100 NVL ^[2].

Where it sits

NVIDIA said systems built on the H200 were expected to begin shipping in the second quarter of 2024 ^[1]. Alongside the memory figures, the company's central performance claim was that the H200 nearly doubles inference speed on Llama 2 70B, a 70-billion-parameter model, compared with the H100 ^[1]. That comparison points at the workload the chip was positioned for: serving trained models, where reading weights out of memory dominates, rather than training runs, where raw arithmetic throughput matters most. For buyers, the practical questions the announcement answered were how large a model one device can hold and how quickly it can be served — and the capacity and bandwidth numbers were the answer.