NVIDIA Tesla V100

What the V100 is

The NVIDIA Tesla V100 is a data-center GPU announced on 10 May 2017 by Jensen Huang during NVIDIA's GTC keynote^[1]. It was the first product based on Volta, which NVIDIA called its seventh-generation GPU architecture^[1]. Huang said that thousands of engineers had spent more than three years building Volta^[1]. A data-center GPU is a processor sold on a board for servers rather than for a desktop: it has no display output and is judged on how much large-scale numerical work — training neural networks, running simulations — it can do per watt and per rack.

The chip

The die, called GV100, holds 21.1 billion transistors on an area of 815 mm², manufactured by TSMC on a 12 nm FFN process customised for NVIDIA^[2]. The version used in the V100 exposes 80 streaming multiprocessors (SM) with 5,120 CUDA cores in total^[2]. CUDA cores are the small arithmetic units that execute the ordinary floating-point instructions of a GPU program.

Tensor Cores

The feature that separated the V100 from earlier NVIDIA data-center chips was a new kind of unit inside each SM. The V100 introduced Tensor Cores, 640 of them in total, eight per SM^[2]. Each one multiplies two 4×4 matrices of 16-bit floats (FP16) and adds the result into an FP16 or FP32 matrix on every clock^[2]. One such operation therefore covers 64 multiply-adds, and it is performed by fixed-function hardware in a single step rather than as a long sequence of ordinary instructions. Matrix multiplication dominates both the training and the running of neural networks, so setting aside silicon for small matrix multiply-accumulate steps is a way to buy throughput. Doing the multiply in FP16 halves the storage and bandwidth needed per number, while accumulating in FP32 keeps the running sums from losing precision.

Performance figures

NVIDIA's architecture whitepaper quotes 125 Tensor TFLOPS for deep learning, described as up to 12 times the peak training throughput of standard FP32 operations on the earlier P100^[2]. The same document gives 15.7 TFLOPS of FP32 and 7.8 TFLOPS of FP64^[2]. The launch press release rounded the deep-learning figure to 120 teraflops^[1][2].

Memory, links and power

The board carries 16 GB of HBM2 memory developed with Samsung, delivering 900 GB/s of peak bandwidth, about 1.5 times the delivered bandwidth of the P100^[2]. HBM2 stacks memory dies and places them on the same package as the GPU, which is how such a wide connection fits on one board. Second-generation NVLink links GPUs to each other with up to six links and 300 GB/s of total bandwidth, against four links and 160 GB/s on the P100^[2]. Thermal design power is 300 W^[2].

Who said they would use it

At launch, Oak Ridge National Laboratory said that Summit, its supercomputer to be completed in 2018, would be powered by Volta GPUs, and Amazon Web Services, Google Cloud, Microsoft, Baidu and Tencent announced support^[1].

Where it ended up

The longer effect showed in machine learning. The GPT-3 paper published by OpenAI in May 2020 states in Section 2.3 that all of its models, up to 175 billion parameters, "were trained on V100 GPU's on part of a high-bandwidth cluster provided by Microsoft"^[3]. A chip announced in 2017 was thus the hardware behind the training of a 175-billion-parameter language model in 2020.