Groq LPU (GroqChip)

What it is

GroqChip is the AI processor designed by Groq, and the chip at the centre of the products Groq sells under the LPU name. The property Groq puts first is determinism: the company describes the GroqChip as a fully deterministic processor with no run-to-run variation^[1]. Concretely, that means running the same program over the same data gives the same answer in the same amount of time every time, instead of drifting slightly with scheduling, cache state or whatever else the machine happens to be doing. Determinism is partly a hardware property and partly a matter of how work is scheduled, and it matters most when many chips have to cooperate, or when a system's timing has to be predictable rather than merely fast on average.

Specifications

Groq lists the GroqChip on a 14-nanometer process node and lists it as available in production^[1]. A process node describes the manufacturing generation a chip is built on; the number is a rough label for how small the features are and how much logic fits into a given area of silicon.

Throughput is quoted as up to 750 INT8 TOPS and 188 FP16 TFLOPs at 900 MHz^[1]. TOPS counts trillions of 8-bit integer operations per second, and TFLOPs counts trillions of floating-point operations per second; the two figures describe different kinds of arithmetic, because neural-network computation can often be carried out in low-precision integer or half-precision floating point. Both figures are given at a 900 MHz clock. A clock rate is the pace at which the chip's logic steps, and the throughput numbers follow from how much work is done per step.

Memory is where the design looks most distinctive on paper. Groq lists 230 MB of on-die SRAM and up to 80 TB/s of on-die memory bandwidth per chip^[1][2]. SRAM is memory built from the same kind of fast circuitry as the processor itself, so it can sit on the same piece of silicon and be read at very high speed; memory bandwidth is simply how many bytes can be moved per second. Keeping hundreds of megabytes on the die, with bandwidth measured in tens of terabytes per second, means that most of the data a computation needs can stay close to the arithmetic units instead of travelling out to separate memory chips.

Connectivity is handled on the chip as well: Groq lists 16 integrated RealScale chip-to-chip interconnects and an integrated PCIe Gen4 x16 controller^[1]. PCIe is the standard interface used to attach an accelerator to a host computer, and Gen4 x16 denotes a particular generation and width of that link. RealScale is Groq's name for the connections that join chips directly to one another, so that several of them can be wired into a larger machine without routing all traffic back through the host.

Power is quoted three ways: a 215 W thermal design power, 300 W maximum and 185 W average^[1]. Thermal design power is the figure a system builder uses when sizing cooling and power delivery; the maximum and average figures describe how consumption moves between peak and sustained workloads.

The GroqCard

For systems, Groq packages the silicon as the GroqCard, which puts a single GroqChip on a PCIe Gen4 x16 card with up to 11 RealScale chip-to-chip connectors^[2]. The chip itself carries 16 such interconnects^[1], while the card exposes up to 11 of them to the outside. The card is the unit that goes into a host machine; the connectors are what let several cards be linked together, so a machine built from these cards is a set of chips wired to each other rather than a single device.

Where it sits

Read together, the specs describe three design choices: a large, very high-bandwidth memory placed on the chip itself; execution that is deterministic rather than statistically fast; and direct chip-to-chip links built into the silicon rather than added around it. The trade-offs are visible in the same numbers. On-die memory is fixed at 230 MB per chip, so a working set larger than that has to be spread across several chips, which is what the RealScale links and the card connectors are there to make possible; and deterministic timing makes a machine assembled from many chips easier to reason about as a whole. These are the terms in which this particular answer to the question of how to build AI hardware is stated.