Google TPU v2

What it is

Google TPU v2 is the second-generation Tensor Processing Unit, a chip for machine-learning computation. Google announced it in May 2017. It is the successor to Google's first TPU. The company also said the chip would be available on Google Compute Engine for TensorFlow applications. That statement tied the hardware to Google's cloud service and to the TensorFlow software stack.^[1]

Why a second generation

Google described the first-generation design as limited by memory bandwidth. Memory bandwidth is the rate at which data moves between memory and compute units. If bandwidth is too low, fast arithmetic units can spend time waiting for data instead of calculating. For the second generation, Google moved to 16 GB of HBM, a high-bandwidth memory technology, and raised memory bandwidth to 600 GB/s. The same design came with a performance figure of 45 teraFLOPS.^[1]

Chip, module, pod

The v2 chip is manufactured on a 16 nm process and has a thermal design power of 280 W. Thermal design power, or TDP, is a measure of the heat a chip is expected to dissipate under sustained load; it is used in cooling and system design. Google built the v2 into larger units as well. Four v2 chips are assembled into a module rated at 180 teraFLOPS, and 64 of those modules form a 256-chip pod rated at 11.5 petaFLOPS. The hierarchy—chip, module, pod—shows how the design scales from one processor to a larger system through packaging and interconnection. A teraFLOPS is one trillion floating-point operations per second, and a petaFLOPS is one quadrillion, so the pod figure is stated at a much larger scale than the single-chip figure.^[1]

Floating point and bfloat16

The first-generation TPUs could only compute on integers. The second generation added floating-point support and introduced the bfloat16 format, which was developed by Google Brain. Integer arithmetic represents whole numbers exactly. Floating-point arithmetic represents numbers with a fractional part and a range, which is important for many machine-learning calculations. bfloat16 is a floating-point format intended for such calculations. The addition of floating-point support made the second-generation TPUs usable for both training and inference. Training is the process of adjusting a model from data; inference is running a trained model to produce outputs.^[1]

Place in the line

Taken together, the v2's changes mark the point where Google's TPU line moved from integer-only computation to floating-point computation and from individual chips to pod-scale systems. The chip's 45 teraFLOPS, the 180-teraFLOPS module, and the 11.5-petaFLOPS pod describe the same scaling path at three levels. Google's statement that the second-generation TPUs would be available on Google Compute Engine for TensorFlow applications placed the hardware in a cloud context where users could run TensorFlow work. That combination of floating-point support and pod-scale packaging is the v2's main place in the line.^[1]