NVIDIA Tesla P100

What it is

The Tesla P100 is a data-center GPU that NVIDIA introduced on 5 April 2016 at its GPU Technology Conference. It is based on the Pascal architecture, and NVIDIA presented it at the time as the most advanced hyperscale data-center accelerator it had ever built ^[1]. It is a compute card rather than a graphics card: it exists to do floating-point arithmetic inside servers, not to drive a display.

The chip

The processor behind it, the GP100, carries 15.3 billion transistors on a 610 mm² die manufactured with 16 nm FinFET technology; NVIDIA described it as the world's largest FinFET chip at the time ^[2]. The design contains 56 streaming multiprocessors holding 3,584 FP32 CUDA cores in total ^[2]. With GPU Boost the part reaches 5.3 teraflops in double precision, 10.6 teraflops in single precision and 21.2 teraflops in half precision (FP16) ^[2]. That last figure is the one that mattered for machine learning: the new half-precision instructions were designed specifically for deep learning, where neural-network arithmetic tolerates a narrower number format in exchange for much higher throughput per second.

Memory and interconnect

The P100 was the first GPU to use HBM2 stacked memory. 16 GB sits on the same package as the GPU, assembled with CoWoS (Chip-on-Wafer-on-Substrate), and provides 720 GB/s of bandwidth, three times that of the previous Maxwell generation. Thermal design power is 300 W ^[2]. Putting memory on the package instead of across a circuit board keeps the connections short, which is how that bandwidth is reached within the power budget.

It also introduced NVLink, a GPU-to-GPU interconnect delivering 160 GB/s of bidirectional bandwidth, five times the 31.5 GB/s of PCIe Gen3 x16. Up to eight P100s can be linked in a single server, so the cards can exchange data with each other directly instead of passing everything through the CPU and the PCIe bus ^[2].

Performance claims

NVIDIA claimed over 12 times the neural-network training performance of its Maxwell-based solution, and said that matching what eight P100s do on AlexNet training would take 250 dual-socket CPU server nodes ^[1]. The comparison is the point of the product: work that once needed a room of processors was being pulled onto a handful of accelerators.

The first system

The P100 first shipped in the NVIDIA DGX-1 system, which became generally available in the United States in June 2016. That machine packed eight P100s with 16 GB each, for up to 170 teraflops of FP16 performance ^[3].

Where it sits

The clearest evidence of what the card made possible comes from later work built on it. The 2017 Transformer paper, "Attention Is All You Need", reports training its models on a single machine with 8 NVIDIA P100 GPUs: the base model took about 0.4 seconds per step for 100,000 steps, roughly 12 hours, and the large model 1.0 second per step for 300,000 steps, about 3.5 days. When estimating the cost of that training, the authors used 9.5 TFLOPS as the P100's sustained single-precision figure ^[4].

Read together, the specifications and that usage describe a shift in what a single server could do. Before the P100, training a large neural network was spread over many CPU machines; with HBM2 feeding the chip, FP16 arithmetic for the matrix multiplications, and NVLink tying eight cards together, an eight-GPU box became a practical unit of computation. The P100 is the point where high-bandwidth stacked memory, a low-precision fast path for learning and a dedicated GPU interconnect arrived in one data-center product, and the deep-learning systems that followed were designed around that combination.