Google TPU (first generation)

What it is

The Tensor Processing Unit, or TPU, is a custom chip — an application-specific integrated circuit (ASIC) — that Google designed to accelerate the inference phase of neural networks, the stage at which an already-trained model answers requests instead of being trained. Google had the chips running in its data centers from 2015 and first described them publicly in May 2016^[1].

Where it came from

In 2013 Google ran a projection: if people used voice search for three minutes a day, the speech-recognition networks behind it would need enough computing power to double the company's data centers^[1]. That estimate turned into a high-priority project to build an inference ASIC. The team kept buying off-the-shelf GPUs for training and put custom silicon where the volume was — in serving^[1]. The target was concrete: 10 times better cost-performance than GPUs^[1].

How it was built

The schedule was short. The chip was designed, verified, built and deployed in data centers in 15 months, and Google said the team had first silicon running applications at speed within 22 days^[1].

Its heart is a matrix multiply unit holding 65,536 8-bit multiply-accumulate units, giving a peak of 92 trillion operations per second, backed by 28 MiB of software-managed on-chip memory^[1]. Doing arithmetic in 8 bits rather than the 32-bit floating point of a general-purpose processor makes each unit smaller and cheaper in silicon area and power, which is the trade an inference chip wants: many more multipliers per chip, at precision that inference tolerates.

Fabricated on a 28 nm process, the chip runs at 700 MHz with a 75 W thermal design power, and measured about 40 W when busy^[1]. Weights are read from 8 GiB of off-chip DDR3 memory at 34 GB/s^[1]. It attaches over the PCIe bus as a coprocessor, and a TPU board fits into a hard-disk slot in Google's racks, so existing machines could take one without being redesigned^[1].

Results

The public performance analysis came in a 2017 ISCA paper by Norman Jouppi and colleagues at Google. On Google's production workloads the TPU was on average about 15–30 times faster than a contemporary Nvidia K80 GPU or Intel Haswell CPU, with 30–80 times higher performance per watt^[1].

The comparison rested on workloads chosen to be representative rather than convenient: multilayer perceptrons, convolutional neural networks and LSTMs that together represented 95% of Google's data-center inference demand^[1]. They included RankBrain for search, part of Google's neural machine translation system, and DeepMind's AlphaGo^[1]. Google said TPUs powered RankBrain and Street View, and that AlphaGo ran on TPUs during its matches against Lee Sedol, letting it think faster and look further ahead between moves^[2].

Why it matters

The TPU is an early and well-documented example of a narrow chip beating general-purpose hardware at a job that general-purpose hardware was already doing. The claim was not made on synthetic benchmarks but on services in production, and the paper set out the machine's architecture, its memory hierarchy and the workloads behind the numbers^[1]. It also put a marker on where the money went: training stayed on off-the-shelf GPUs while serving moved to custom silicon^[1]. Just as telling is the deployment detail — a coprocessor on PCIe that slides into a hard-disk bay, and a software-managed on-chip scratchpad rather than caches — choices that let a new kind of chip be dropped into racks that were already full of ordinary servers^[1].