NVIDIA GH200 Grace Hopper Superchip

What it is

The GH200 Grace Hopper Superchip is an NVIDIA product that puts a CPU and a GPU into a single package instead of placing them as two separate chips wired together over PCIe. The CPU half is a Grace CPU, the GPU half is a Hopper GPU, and the two are joined inside the package by NVLink-C2C, a die-to-die interconnect.^[1] NVIDIA calls the combined part a superchip.

Inside the package

The Grace side carries 72 Arm Neoverse V2 cores, with 4×128-bit SVE2 — Arm's second-generation scalable vector extension, the instruction set Arm cores use for wide data-parallel work — and 117 MB of L3 cache.^[1] The GPU side carries 132 Hopper streaming multiprocessors (SMs) at compute capability 9.0.^[1]

Two levels of memory

A GH200 ships with either 96 GB of HBM3 or 144 GB of HBM3e as GPU memory.^[1] Alongside that it supports up to 480 GB of LPDDR5X CPU memory running at up to 500 GB/s.^[1] The two tiers do different jobs: HBM sits next to the GPU and is fast but comparatively small, while LPDDR5X is much larger and slower. Because both are attached within one package through the same link, a program can draw on both pools without shipping data back and forth over a PCIe bus between separate CPU and GPU boards.

The link between the halves

NVLink-C2C runs at 900 GB/s. NVIDIA says that is seven times the bandwidth of PCIe Gen5 at one fifth of the power.^[1][3] The link is what makes the CPU and GPU behave like one unit rather than two machines sharing a cable, and it is the number that most clearly separates the GH200 from a conventional CPU-plus-GPU server.

Two superchips: GH200 NVL2

NVIDIA also defines the GH200 NVL2, which links two superchips over NVLink. Such a pair offers 288 GB of high-bandwidth memory, 10 TB/s of memory bandwidth and 1.2 TB of fast memory.^[2]

Power

The thermal design power of a GH200 is quoted as 550 to 1,000 W including memory.^[1] The wide range covers the different memory configurations the part is offered in.

Benchmark results

The GH200 first appeared in MLPerf Inference in the v3.1 round. MLPerf Inference is an industry benchmark suite for measuring how fast trained models answer queries; NVIDIA said the GH200 ran every one of the tests it submitted for that round.^[3]

Where it sits in system design

The usual way to build an AI server is to buy a CPU board and a GPU board and connect them over PCIe. The GH200 takes the opposite approach: the two processors and the memory attached to each are designed as one part, with the interconnect fixed at build time rather than negotiated on a slot. The result is a package with an unusually large combined memory pool and a link whose bandwidth NVIDIA states explicitly against PCIe Gen5, traded against a power envelope in the hundreds of watts. Those two figures — memory capacity and link bandwidth — are the numbers the design is meant to be judged on.