NVIDIA A30

What the A30 is

The NVIDIA A30 is a data-center GPU built on NVIDIA's Ampere architecture. NVIDIA describes the card's target as AI inference and mainstream enterprise workloads.^[1] Inference is the stage at which a trained model answers requests, as distinct from training, the stage in which the model is built; the A30 is presented as a part for running models in production and for everyday business computing.

What NVIDIA lists on the datasheet

NVIDIA lists 24 GB of HBM2 memory and 933 GB/s of memory bandwidth for the A30.^[1] HBM2 is a stacked form of high-bandwidth memory: capacity sets how large a model or a batch can fit on the card, and bandwidth sets how quickly data can be fed to the compute units.

For arithmetic, NVIDIA lists 10.3 TFLOPS of FP32, 165 TFLOPS of TF32 Tensor Core throughput with sparsity, 330 INT8 TOPS and 661 INT4 TOPS.^[1][2] These numbers count different things. FP32 is ordinary single-precision floating point, the format used across most general-purpose GPU work. TF32 is a Tensor Core format that keeps the range of FP32 with less precision, and the listed TF32 figure assumes sparsity, a technique that skips weights equal to zero and so raises the effective rate. INT8 and INT4 are low-precision integer formats used mainly for inference, where shorter numbers mean faster arithmetic and less data moved.

The card is listed with a 165 W maximum thermal design power and a PCIe Gen4 interface.^[1] Thermal design power is the ceiling a cooling system and a server must be designed for; PCIe Gen4 describes how the card attaches to the host system and moves data over its bus.

Splitting one card into several

NVIDIA says the A30 can be partitioned into as many as four MIG instances with 6 GB of memory each, into two instances with 12 GB each, or left as a single 24 GB device.^[1] MIG, for multi-instance GPU, divides one physical GPU into slices with their own memory and compute, so that several jobs or several users can share a card without interfering with one another. The three options NVIDIA lists differ only in how finely the same 24 GB is divided.

Performance claims

NVIDIA says the A30 delivers up to three times the AI training throughput of the V100 and six times that of the T4.^[1] These are vendor figures for a particular comparison; what a given job actually sees depends on the model, the batch size and the precision used.

Where it sits

The A30 belongs to NVIDIA's Ampere generation and connects over PCIe Gen4.^[1] Its combination of 24 GB of memory, a 165 W maximum power draw and up to four partitions describes the kind of work it is sold for: inference and mainstream enterprise computing, where a card is often shared among several smaller jobs or users rather than given over to one large job.^[1] The V100 and T4 comparisons NVIDIA offers are training-throughput comparisons, which is how the company places the card against earlier generations of its own data-center GPUs.^[1]