Tensor Cores

What it is

Tensor Cores are specialised units inside NVIDIA GPUs that carry out a small matrix multiply-and-add in a single step. That matters because matrix multiplication, usually referred to as GEMM, is the core arithmetic of neural-network training and inference: most of what a network does when it learns from data or answers a query reduces to multiplying matrices together and summing the results ^[4]. A unit that performs this operation faster, and with less circuitry than general-purpose math, speeds up the whole job.

How it works

In their first generation, each Tensor Core evaluates the expression D = A×B + C on 4×4 matrices, taking FP16 inputs and accumulating in either FP16 or FP32, at 64 fused multiply-adds per clock ^[1]. A fused multiply-add is a single instruction that multiplies two numbers and adds the product to a running total, which is exactly the inner loop of a matrix product.

The size of the tile is what makes the difference. Multiplying a 4×4 matrix by a 4×4 matrix produces 16 output entries, each of which is a sum of 4 products — 64 multiply-adds in total. A single Tensor Core completes that whole block in one clock, because it is built to do 64 fused multiply-adds per clock ^[1]. Ordinary general-purpose cores would work through the same block one multiply-add per lane at a time.

The arithmetic is deliberately mixed. Products are formed in lower precision (FP16), but the running sum is kept in higher precision (FP16 or FP32), so that adding up many small terms does not steadily lose accuracy ^[1]. This "mixed precision" pattern — cheap multiplies, careful accumulation — is the idea every later generation builds on.

Where it came from

Tensor Cores debuted in the Tesla V100, based on the Volta architecture, announced in May 2017. That chip contained 640 Tensor Cores, eight per streaming multiprocessor ^[1]. The payoff was measured against the previous generation: the V100 delivered 125 Tensor TFLOPS, up to 12 times the peak training throughput of standard FP32 math on the earlier P100 ^[1].

Later generations

The third-generation Tensor Cores in the A100, introduced in 2020, added TF32, a format that accelerates FP32 workloads without requiring code changes, along with BF16 and FP64, and support for fine-grained structured sparsity that doubles throughput by skipping zero weights ^[2].

The fourth generation, in the H100 of 2022, added FP8 and works with a "Transformer Engine"; NVIDIA claims up to 4x faster training of GPT-3 (175B) than the prior generation ^[1].

The fifth generation, in Blackwell in 2024, added 4-bit floating point for inference, which NVIDIA said doubles compute and the model size that can be supported ^[3].

Beyond NVIDIA

The idea is not exclusive to one vendor. Google's TPUs are built around matrix multiply units, and AMD's MI300X carries 1,216 "matrix cores" ^[5].

Limits

Lower precision has costs. The DeepSeek-V3 report found that FP8 matrix-multiply accumulation on the H800 kept only about 14 bits of precision, so it periodically moved partial results to FP32 on regular CUDA cores in order to train accurately in FP8 ^[1]. Sparsity features likewise only help when weights actually are zero, and mixed-precision speedups depend on the workload being dominated by the matrix operations these units accelerate.