NVIDIA announces the Hopper architecture and H100

Background

An AI accelerator is a processor built to carry out many arithmetic operations in parallel, which is what training and running large neural networks requires. NVIDIA designs GPUs for this role, and its data-centre chips are among the hardware on which large models are trained and served. A new generation of such a chip is therefore a notable event for anyone who follows AI: it sets the arithmetic, memory and networking that the next round of models will run on.

What was announced

At its GTC event on 22 March 2022, NVIDIA announced the Hopper architecture and its first Hopper GPU, the H100. The company described the H100 as the world's largest and most powerful accelerator, with 80 billion transistors^[1].

The Transformer Engine

The headline feature was the Transformer Engine, a part of the chip built specifically for Transformer models. A Transformer is the network architecture in which each element of a sequence — a word, a patch of an image — is combined with the others through attention, so that the model can weigh which parts of its input matter for each output. Transformer networks became the basis of most large language models, and they are heavy consumers of a particular kind of arithmetic. NVIDIA said Transformer models had become the standard model choice for natural language processing, and claimed the Transformer Engine would speed them up by as much as 6x over the previous generation without losing accuracy^[1].

Performance claims

NVIDIA made two further claims about what the H100 would allow. It said chatbots could run Megatron 530B — which it called the world's most powerful monolithic transformer language model — with up to 30x higher throughput than the previous generation. It also said a 395-billion-parameter mixture-of-experts model could be trained up to 9x faster^[1]. A mixture-of-experts model is one in which only part of the network is used for any given input, so that a very large model can be run at a fraction of the cost of using all of it. These figures are the company's own comparison of the new chip with the generation before it; they describe speed-ups on particular workloads rather than general-purpose gains.

Memory, interconnect and other features

NVIDIA also listed a set of features around the processor core. The H100 was announced with HBM3 memory at 3 TB/s of bandwidth, PCIe Gen5, a second-generation Multi-Instance GPU capability that lets one chip be split into separate smaller instances, confidential computing, and DPX instructions for dynamic programming^[1]. On the networking side, the fourth-generation NVLink gives 900 GB/s per GPU inside a DGX H100, and an external NVLink Switch can connect up to 256 GPUs^[1]. Connecting many GPUs matters because a model too large for one chip has to be spread across several, and the links between chips then decide how fast the whole thing runs.

Systems, clouds and availability

The fourth-generation DGX H100 system, which holds eight H100s, was rated at 32 petaflops of AI performance at the new FP8 precision^[1]. FP8 is an 8-bit floating-point format; lower-precision arithmetic is cheaper, so the same chip performs more operations per second, and the rated throughput is quoted at that precision. Alibaba Cloud, Amazon Web Services, Baidu AI Cloud, Google Cloud, Microsoft Azure, Oracle Cloud and Tencent Cloud all planned H100 instances, and NVIDIA said the chip would be available from the third quarter of 2022^[1]. Separately, NVIDIA introduced the Grace Hopper Superchip, which pairs a Hopper GPU with an NVIDIA Grace CPU over the NVLink-C2C interconnect^[1].

Why it matters

The H100 announcement is a marker of how far AI hardware had come to be designed around one model family. The feature NVIDIA chose to headline was aimed at Transformers by name, the precision the chip is rated at is a low one chosen for throughput, and the memory, interconnect and multi-instance features all point at the same problem: how to run a very large model quickly across many chips. The list of cloud providers planning instances says where the chip was expected to be used — not on desks, but in rented data-centre capacity behind chatbots and other language services. NVIDIA's claimed factors of 6x, 30x and 9x are attached to named models and specific workloads, and are the company's own comparisons against the previous generation.