NVIDIA A100

What it is

The A100 is a data-center GPU from NVIDIA and the first GPU built on the company's Ampere architecture. It was announced on 14 May 2020 at GTC 2020, and NVIDIA said at the time that the chip was already in full production and shipping. The headline claim was up to 20 times the AI performance of its predecessor^[1].

The chip

The processor at its center is the GA100, which packs 54.2 billion transistors onto an 826 mm² die manufactured on TSMC's 7 nm N7 process. NVIDIA described it as the world's largest 7-nanometer processor^[2]. A large die like this is a manufacturing gamble as much as an engineering one: the bigger the chip, the fewer good ones come out of each wafer, which is why announcing volume production on day one mattered to the launch story.

Tensor Cores and number formats

The third-generation Tensor Cores — dedicated matrix-multiply units inside the GPU — added several number formats. TF32 speeds up work that was written for ordinary 32-bit floating point without requiring code changes, and the hardware also supports BF16 and FP64. The published peak dense figures are 312 TFLOPS for FP16/BF16 on the Tensor Cores, 156 TFLOPS for TF32, and 19.5 TFLOPS for FP64 Tensor Core work^[2]. The logic behind the mixed formats is general to this kind of hardware: representing numbers with fewer bits lets the chip move more of them through the same silicon per second, so a format that keeps enough range for neural-network weights buys throughput.

Sparsity

A second source of throughput is fine-grained structured sparsity. Many trained networks end up with weights that are zero, and the Tensor Cores can skip those zeros rather than doing arithmetic on them, which doubles throughput in the supported pattern. That is where the sparse peak numbers come from: 624 TFLOPS at FP16 and 1,248 TOPS at INT8^[2].

Splitting one GPU into seven

Multi-Instance GPU, or MIG, lets a single A100 be partitioned into as many as seven isolated GPU instances^[2]. The point is utilization: a large accelerator is wasteful for small jobs, and isolated instances let several smaller workloads share one card without interfering with each other.

Memory

The original A100 carried 40 GB of HBM2 memory running at 1,555 GB/s, alongside a 40 MB L2 cache. A later version, the A100 80GB in the SXM form factor, moved to HBM2e at 2,039 GB/s, with a maximum thermal design power of 400 W^[3].

Systems and cloud availability

NVIDIA does not sell only chips. Announced the same day was the DGX A100 system, which contains eight A100s^[1]. Cloud providers lined up as well: Alibaba Cloud, AWS, Baidu Cloud, Google Cloud, Microsoft Azure, Oracle and Tencent Cloud all planned A100 services^[1]. Microsoft said it would train "dramatically bigger AI models using thousands of" A100s in Azure^[1].

Where it sits in the story of large models

The A100 became one of the machines on which large language models were actually trained, and papers from that work record what it looked like in practice. Meta's LLaMA paper, from February 2023, reports that training the 65B-parameter model processed about 380 tokens per second per GPU across 2,048 A100 80GB GPUs, so that 1.4 trillion tokens took about 21 days; the team estimated that developing the models used 2,048 A100-80GB GPUs for about five months^[4]. The Llama 2 paper, from July 2023, says pre-training ran on Meta's Research SuperCluster and on internal production clusters, both using NVIDIA A100s, and took 3.3 million GPU hours on A100-80GB hardware, which has a thermal design power of 350–400 W^[5].

Those numbers describe more than a fast part. They show the unit in which frontier training was budgeted at the time — GPU count, tokens per second, GPU hours — and they explain why a chip's memory capacity and bandwidth matter as much as its arithmetic peak: the model has to be fed.