NVIDIA DGX H100

What it is

DGX H100 is a server-scale AI computer that NVIDIA announced at its GTC conference on 22 March 2022, as the fourth generation of the company's DGX line ^[1]. NVIDIA rates a single system at 32 petaflops of AI performance at FP8 precision, six times the figure for the prior generation ^[1].

Two pieces of jargon sit inside that sentence. A petaflop is 10^15 floating-point operations per second. FP8 means those operations are counted in a format where each number occupies eight bits of memory. Narrow formats like this are a normal part of deep learning: shrinking the numbers makes the arithmetic cheaper and cuts down the memory traffic that dominates large models, and training methods are built to tolerate the loss of range and precision. So the 32-petaflop figure describes throughput in a fast, compact format, not the 64-bit arithmetic used for physics simulation.

Inside the box

Each DGX H100 holds eight H100 GPUs joined by NVLink ^[1]. NVLink is NVIDIA's own high-bandwidth link between GPUs; it lets the chips exchange data directly instead of routing everything through the CPU and the PCIe bus. That matters because a model split across eight GPUs spends much of its time passing activations and gradients around, and the speed of that path sets how well the hardware is used. Together, NVIDIA says, those eight GPUs contain 640 billion transistors ^[1].

Each system also carries two BlueField-3 DPUs and eight ConnectX-7 400 Gb/s InfiniBand adapters ^[1]. A DPU is a processor built to take networking, storage and security work off the main CPU. InfiniBand is a high-speed network fabric common in supercomputers and GPU clusters, and 400 Gb/s is the per-port signalling rate of the ConnectX-7 cards. Fitting all of this into every node means a cluster can be assembled from identical, already-networked units rather than from separately sourced servers and switches.

Scaling to a SuperPOD

NVIDIA also described a configuration it calls a DGX SuperPOD: a cluster built from DGX H100 nodes that would deliver one exaflop of FP8 performance across 256 H100 GPUs ^[1]. An exaflop is 10^18 operations per second, a thousand petaflops, so this is the point where a collection of eight-GPU servers starts to be described in the same units as the largest supercomputers.

Where it sits

DGX is NVIDIA's packaged product line rather than a bare GPU: the company sells a complete machine, with the GPUs, the links between them, and the networking that connects one machine to the next. For buyers, the appeal is a unit that arrives ready to run rather than a parts list to be integrated. The fourth generation's headline numbers, 32 petaflops at FP8 per system and an exaflop per SuperPOD, state the scale at which AI infrastructure was being described: the rack and the cluster, not the individual chip.