NVIDIA DGX A100

What it is

The NVIDIA DGX A100 is a server-class computer built for artificial-intelligence work. NVIDIA describes it as the third generation of its DGX systems, a family of machines the company assembles itself and sells as complete units: the GPUs, the high-speed links between them, the memory and the system software arrive as one integrated box rather than as parts a customer wires together^[1]. The point of such a machine is scale. Training a large neural network means repeating enormous numbers of matrix multiplications, and that work has to be spread across many processors running in parallel.

How it came about

NVIDIA unveiled the DGX A100 at its GTC 2020 conference on 14 May 2020^[1]. The first order for the system went to the US Department of Energy's Argonne National Laboratory, which took it for COVID-19 research^[1].

What is inside

A single DGX A100 carries eight A100 GPUs with 320 GB of GPU memory in total^[1]. Getting eight accelerators to behave like one machine is mostly a communications problem: during training the GPUs have to swap gradients and intermediate results constantly. NVIDIA's technical blog says the system uses six second-generation NVSwitch fabrics to connect the eight GPUs, giving each GPU 600 GB/s of bandwidth and the machine 4.8 TB/s of aggregate bidirectional bandwidth^[2]. NVSwitch is a switch chip inside the server that does for GPUs roughly what a network switch does for computers: it lets any GPU talk to any other at high speed without routing traffic through the CPU or over an external network.

NVIDIA rates the system at five petaflops of AI performance^[1]. A petaflop is a quadrillion floating-point operations per second. Applied to AI work, the figure summarises how much arithmetic the machine can push through in a given stretch of time on the kinds of number formats such work uses.

Splitting the machine

The A100 supports multi-instance GPU, a feature that carves one physical GPU into isolated slices, each with its own memory and compute resources. NVIDIA says a DGX A100 can be partitioned into as many as 56 instances this way^[1]. That matters because not every job needs the whole machine. A group running many smaller experiments, or a cluster serving inference to many users at once, can give each job its own slice instead of queueing for the full system. The same hardware can therefore run as one large training computer or as dozens of smaller ones, and the choice is made in software rather than by buying different equipment.

Where it sits

The DGX A100 is the third generation of the DGX line^[1], and its place in the market is bound up with the A100 GPU inside it: the DGX is how NVIDIA packages that chip, together with its interconnect and system software, into a turnkey system rather than a component. The first order placed one of the machines in a national laboratory working on a pandemic, an early public use of the system for scientific computing^[1].