NVIDIA launches the A100 GPU

Background

The A100 was the newest entry in a line of eight generations of NVIDIA data-center GPUs, and the company described it as the largest performance leap in that sequence^[1]. The main job for such chips is machine learning: training a model means repeating the same arithmetic over enormous amounts of data, and inference means doing a smaller amount of similar work for every request, so both reward a processor that can carry out many calculations at once.

The announcement

On 14 May 2020, at GTC 2020, NVIDIA announced the A100 — the first GPU built on its Ampere architecture — and said it was in full production and already shipping to customers worldwide^[1]. The company claimed up to 20 times the AI training and inference performance of its predecessors^[1]. Jensen Huang described it as "a 20x AI performance leap and an end-to-end machine learning accelerator"^[1].

The five breakthroughs

NVIDIA framed the launch around five items. The first was the Ampere architecture itself, with more than 54 billion transistors, which the company called the world's largest 7 nm processor^[1]. The second was a third generation of Tensor Cores, the specialised units that carry out the matrix arithmetic behind neural networks, now with support for TF32^[1]. The third was Multi-Instance GPU, which lets a single A100 be divided into as many as seven separate instances so that several jobs or users can share one card^[1]. The fourth was a third generation of NVLink, the high-speed connection that lets GPUs exchange data directly rather than through the CPU and system memory^[1]. The fifth was structural sparsity: many weights in a trained network are zero or close to zero, and hardware that can skip them does less work, which NVIDIA said doubles performance^[1].

Systems, cloud and supercomputers

Alongside the chip, NVIDIA announced the DGX A100, a system with eight A100s joined by NVLink that was available the same day^[1]. The eight-GPU HGX A100 board was rated at 10 petaflops of AI performance — a petaflop being a thousand trillion (10^15) floating-point operations per second^[1].

Cloud providers planning A100 services included Alibaba Cloud, AWS, Baidu Cloud, Google Cloud, Microsoft Azure, Oracle and Tencent Cloud^[1]. Microsoft said it had trained Turing NLG on the then-current generation of NVIDIA GPUs and would train "dramatically bigger AI models" on thousands of A100s in Azure^[1]. Among the early supercomputer users named at launch were Indiana University with Big Red 200, Germany's Jülich Supercomputing Centre, the Karlsruhe Institute of Technology, the Max Planck Computing and Data Facility, and NERSC's Perlmutter in the United States^[1].

Software

A new chip is only useful if software can target it, and NVIDIA released CUDA 11 together with new versions of more than 50 CUDA-X libraries to support the A100^[1]. CUDA is the programming environment developers use to write general-purpose code for NVIDIA GPUs; the CUDA-X libraries package ready-made routines for common jobs such as deep learning, so users do not have to build everything from scratch.

What it meant

The launch followed a pattern that has become familiar for data-center AI chips: a new architecture, a multiple-of-previous-generation performance claim, a flagship system built from it, commitments from cloud providers and a list of named supercomputer installations, all announced on the same day^[1]. For people building AI systems, the practical consequence was that a large amount of compute would be available both by the hour from major clouds and inside national supercomputing centres^[1]. The 20-fold figure was NVIDIA's own comparison with earlier generations; how much faster a real workload runs depends on the model and the numerical precision used.