NVIDIA Tesla P40

What the Tesla P40 is

The Tesla P40 is a GPU accelerator for servers. NVIDIA's datasheet for the product is dated September 2016, and it identifies the P40 as a card built on NVIDIA's Pascal architecture and positioned specifically for deep learning inference ^[1]. Inference is the stage of using a trained neural network: the model has already learned its parameters, and the accelerator's job is to apply them to new inputs and return outputs. That workload is different from training, where the model's parameters are updated. In a data center, inference can be run for many users and many requests, so throughput per watt and per server matters.

The P40 is a PCI Express 3.0 x16 card with a 250 W maximum board power ^[1]. It is therefore a component that goes into a server rather than a complete computer. The PCIe interface provides the connection between the card and the host system; the power figure sets the thermal and electrical budget a server must support. NVIDIA positioned the product around inference, so its published performance numbers are chosen to describe how quickly it can process neural-network work rather than only how fast it can draw graphics.

Specifications and what they mean

NVIDIA's product brief lists 3,840 CUDA cores, a 1,303 MHz base clock and a 1,531 MHz boost clock for the P40 ^[2]. CUDA cores are the parallel arithmetic units in an NVIDIA GPU; a large number of them lets the card work on many calculations at once. The base and boost clocks describe the range of operating frequencies NVIDIA specifies. These are hardware specifications, not measured results for a particular neural network.

The card has 24 GB of GPU memory and 346 GB/s of memory bandwidth ^[1]. Memory capacity determines how much model data and intermediate data can reside on the card at once, while bandwidth describes how quickly data can be moved to and from that memory. For inference, both can matter: the hardware must hold the model and feed its arithmetic units with data quickly enough to keep them busy.

NVIDIA rated the P40 at 47 TOPS of INT8 inference performance and 12 teraflops of single precision ^[1]. TOPS means trillions of operations per second, and INT8 refers to 8-bit integer arithmetic. Neural-network inference can often use lower-precision arithmetic than training, so a chip can gain throughput by handling more operations with smaller numbers. The separate 12-teraflop single-precision figure describes the card's FP32 throughput. Together, these ratings show that NVIDIA was selling the P40 on both low-precision inference throughput and conventional floating-point capability.

Server consolidation

NVIDIA made a concrete deployment claim for the P40: a single server with eight P40s could replace up to 140 CPU-only servers for deep learning inference ^[1]. This is a statement by NVIDIA about server consolidation, not a neutral benchmark of every workload. It illustrates why a GPU accelerator can be attractive for inference at scale. If one server can handle the work of many CPU-only servers, the operator may need fewer machines, less rack space and less supporting infrastructure. The claim depends on the workload, the software and the configuration, but it captures the economic argument NVIDIA was making for a dedicated inference card.

Place in the product line

The P40's design combines a server form factor, a fixed power ceiling and a memory configuration suited to keeping model data on the card. Its Pascal architecture and its explicit positioning for deep learning inference place it among accelerators built for deploying trained models rather than for graphics display or for training alone ^[1]. The specifications are those of a data-center component: a PCIe interface, a 250 W board power limit, 24 GB of memory and published throughput figures for inference ^[1]. For readers following the history of AI hardware, the P40 is an example of how GPU makers began to name inference as a distinct target and to describe performance in the units that inference buyers care about, such as INT8 TOPS.