Intel Gaudi 2

What Gaudi 2 is

Intel Gaudi 2 is a data-centre AI accelerator: a chip that goes into a server to carry out the dense matrix arithmetic behind large neural networks, rather than to run an operating system on its own. Its design is dominated by memory. In Intel's own comparison table the chip carries 96 GB of HBM2e with a peak bandwidth of 2.46 TB/s^[1]. HBM2e is high-bandwidth memory built from stacked memory dies packaged alongside the processor; the very wide connection gives it far more bandwidth than ordinary server memory. For AI work that matters twice over: the model's parameters have to fit inside that memory, and the speed at which they can be read limits how fast the chip can work through them.

Compute

In the same table, Gaudi 2 reaches 432 TFLOPS of BF16 peak compute^[1]. BF16 is a 16-bit floating-point format that keeps the exponent range of 32-bit floats while cutting the mantissa, which is why it became a common format for training. Tom's Hardware reported that the chip contains 24 tensor processor cores and two matrix multiplication engines^[2] — the units that perform the multiply-and-accumulate operations that dominate deep learning. The same report puts FP8 matrix performance at 865 TFLOPS^[2]. FP8 stores each number in 8 bits instead of 16, roughly halving the memory traffic per value and letting the same silicon do more work per second, at some cost in precision.

Not all of the chip's arithmetic is matrix work. Tom's Hardware lists 11 TFLOPS of BF16 vector performance for Gaudi 2, well below the 28.7 TFLOPS Intel quotes for Gaudi 3^[2]. Vector units handle the element-wise and other non-matrix operations that surround the big multiplications.

Connectivity, packaging and power

Intel's table gives Gaudi 2 64 GB/s of bidirectional PCIe bandwidth^[1]. That is the link to the host server, used to feed data in and send results out; a slow host link leaves a fast accelerator waiting. The same table lists 4.8 Tb/s of embedded NIC bandwidth^[1] — networking built into the accelerator so that many of them can be wired together into a cluster. Training large models is usually spread across many chips, so the links between chips matter as much as the link back to the server. The chip's thermal design power is 600 W, and it is built in the OAM form factor^[1], a module standard for accelerators that lets server makers design boards around a common shape.

Where it sits

Intel publishes the Gaudi 2 and Gaudi 3 figures side by side in the same table, and the vector figure is the clearest gap between them: 11 TFLOPS against 28.7 TFLOPS^[2]. Reading two generations together is the natural use of published specifications — both chips are described in the same terms (memory capacity, bandwidth, BF16 and FP8 throughput, power), so buyers and engineers can compare them directly. What a specification sheet cannot say is how these numbers turn into throughput on a real workload. That depends on the model, the batch size, the software stack and the network, so the figures above describe the ceiling of the hardware rather than the speed it delivers in practice.