NVIDIA GB200 NVL72

What it is

GB200 NVL72 is a rack-scale computer from NVIDIA. The company describes it as connecting 36 Grace CPUs and 72 Blackwell GPUs in a liquid-cooled design^[1]. The aim is not a rack full of separate servers but a rack that behaves like one large accelerator: the 72 GPUs form a single NVLink domain that NVIDIA says acts as a single massive GPU^[1]. That matters for how the machine is programmed, because a model too large for one GPU can be spread across the domain while the hardware handles communication between chips.

Where it came from

NVIDIA announced the Blackwell platform, together with a DGX SuperPOD built on GB200 Grace Blackwell Superchips, at GTC on 18 March 2024^[2]. The repeating unit is the GB200 Grace Blackwell Superchip, which connects two Blackwell GPUs and one Grace CPU through NVLink-C2C^[1]. The rack keeps that two-to-one GPU-to-CPU ratio at large scale: each DGX GB200 system in the announcement carries 36 Grace CPUs and 72 Blackwell GPUs, the same 36-to-72 ratio used in the GB200 NVL72 rack^[2].

How it is put together

Communication inside the rack runs through the NVLink Switch System, which NVIDIA specifies at 130 terabytes per second of low-latency GPU communications^[1]. The specification table lists 13.4 TB of HBM3E memory at 576 TB/s, and 2,592 Arm Neoverse V2 CPU cores for the full rack^[1]. The GPUs carry a second-generation Transformer Engine that NVIDIA says enables FP4 AI, meaning four-bit floating-point arithmetic in the matrix operations that dominate transformer workloads^[1]. Cooling is liquid rather than air, and NVIDIA's efficiency comparison is drawn specifically against H100 systems built with air-cooled infrastructure^[1].

The published numbers

NVIDIA's specification table gives 1,440 petaflops of NVFP4 Tensor Core performance with sparsity for the full rack, and 720 petaflops of FP8/FP6 Tensor Core performance^[1]. The two figures are not a contradiction: FP4 arithmetic represents each number with half as many bits as FP8, so it fits more work into the same silicon, and the NVFP4 number is quoted in the sparse mode in which many weights are skipped.

Against the previous generation, NVIDIA states that GB200 NVL72 delivers 30 times faster real-time inference on trillion-parameter language models and 4 times faster training than the H100 GPU^[1]. It also states that GB200 delivers 25 times more performance at the same power compared with H100 air-cooled infrastructure^[1]. These are vendor-published comparisons, and the precision formats, sparsity settings and system configurations behind them differ from one line to the next, so they are best read as the direction and rough size of the gain rather than as a like-for-like benchmark.

Where it sits

GB200 NVL72 is the unit in which NVIDIA's Blackwell generation is packaged for large-scale AI: a rack, not a single board, is the thing that is bought, cooled and networked^[1][2]. Its design puts weight on two things at once — the link fabric that lets dozens of GPUs share work, and low-precision arithmetic that makes each GPU cheaper to run. Both are aimed at the same target: training and, especially, serving language models with around a trillion parameters, where the model does not fit comfortably on a single chip and the cost of moving data between chips becomes as important as the cost of computing.