NVIDIA Tesla T4

What it is

The NVIDIA Tesla T4 is a graphics processor built mainly for inference: the stage where a model that has already been trained is asked to answer a question, classify an image, or generate text. NVIDIA built the card on its Turing architecture and packaged it as a 70-watt, single-slot, low-profile PCIe card ^[1]. Those packaging choices describe who the card is for. 70 watts is a power budget an ordinary rack server can supply and cool without special provision; a single slot means several cards can sit side by side in one machine; the low-profile shape fits into shorter chassis. The T4 is therefore a card added to servers that already exist, rather than one that requires a machine designed around it.

What sits on the chip

NVIDIA's specification table lists 320 Turing Tensor Cores and 2,560 CUDA cores ^[1]. The two kinds of core do different work. CUDA cores are the general-purpose arithmetic units a GPU uses for graphics and for ordinary vector maths. Tensor Cores are specialised units that compute a small matrix multiplication in a single operation, which is the dominant piece of arithmetic inside a neural network layer. A card with many CUDA cores but no matrix units has to build those multiplications out of many smaller steps; Tensor Cores collapse them. The ratio on the T4 — far more CUDA cores than Tensor Cores — reflects that most of the silicon is still conventional, with matrix throughput supplied by a smaller specialised block.

The throughput numbers

NVIDIA lists 8.1 TFLOPS of FP32, 65 TFLOPS of FP16, 130 INT8 TOPS and 260 INT4 TOPS for the T4 ^[1]. Read together, these figures say the card is designed to trade precision for speed. FP32 is the standard 32-bit floating-point format used in graphics and scientific computing. FP16 halves the bits per number, and INT8 and INT4 go further, storing values as 8-bit and 4-bit integers. On the T4 each step down in precision raises the quoted throughput, roughly doubling it between FP16 and INT8 and again between INT8 and INT4. That matters because inference usually tolerates less precision than training: once a network's weights are fixed, rounding them to shorter representations often leaves the output close enough for the task. The headline 260 INT4 TOPS is only reachable by models prepared for that format.

Memory

NVIDIA lists 16 GB of GDDR6 memory and more than 320 GB/s of memory bandwidth for the T4 ^[1]. Capacity decides how large a model, or how large a batch of requests processed together, fits on the card without data being shuttled back to the host machine over the PCIe link. Bandwidth decides how quickly weights and intermediate results can be fed to the cores. Inference is frequently limited by memory traffic rather than raw arithmetic, because generating a single answer may require reading the whole set of weights once. 16 GB is enough for many production models of the card's era but not for the largest ones, and the bandwidth figure is the more useful guide to what the card can sustain.

Video work

NVIDIA says the T4's dedicated transcoding engines can decode up to 38 full-HD video streams ^[1]. Transcoding is the step behind nearly all video delivery: a compressed stream is decoded, then re-encoded at a different resolution, bitrate or format for a particular viewer or network. Doing that on general-purpose cores is expensive, so dedicated video engines handle it, and on the T4 they are separate from the CUDA and Tensor Cores. A single card can therefore decode video and run inference on the results at the same time — for example analysing frames for content moderation or search while serving the stream itself.

Where it sits

NVIDIA states that the T4 delivers up to 40 times higher performance than CPUs on inference workloads ^[1]. That comparison is the card's central argument. Inference at scale means a very large number of small jobs arriving continuously, so what a data centre wants is throughput per watt and per rack unit rather than peak speed on one large computation. A low-power card that fits many units per server, supports reduced-precision arithmetic and also handles video is shaped for exactly that pattern. A vendor's speed-up figure is a best case, measured on selected models, precisions and batch sizes, so the multiple should be read as an indication of the gap rather than a promise for any particular workload. The design direction, however, is clear: the T4 marks the point where a GPU could be treated as a general-purpose inference and media accelerator for ordinary servers, not as specialised equipment.