Intel Gaudi 3

What it is

Intel Gaudi 3 is an AI accelerator: a processor for data-center servers, designed to run both the training and the inference of generative AI models ^[1]. Intel formally introduced the chip in 2024 ^[2]. It is delivered as an OAM module, a mezzanine card that server vendors mount on a shared baseboard, and an eight-accelerator baseboard is the unit Intel quotes a price for ^[1][2].

Two dies, several kinds of compute unit

The compute is split across two dies built on a 5 nm process and joined through an interposer bridge ^[1]. Across the whole chip Intel counts 64 tensor processor cores and eight matrix multiplication engines ^[1]. The two do different jobs. The matrix multiplication engines are fixed-function blocks that perform dense matrix multiplies — the dominant arithmetic in neural-network training and inference — while the tensor processor cores are programmable units that handle the surrounding vector and elementwise work. The chip also carries 96 MB of on-die SRAM that delivers 12.8 TB/s ^[1].

Memory and arithmetic throughput

Memory is 128 GB of HBM2e spread over eight stacks, with 3.67 TB/s of bandwidth ^[1]. For matrix operations Intel rates the chip at 1,835 TFLOPS for BF16 and the same 1,835 TFLOPS for FP8 ^[1]. The supported matrix formats are FP8, BF16, FP16, TF32 and FP32 ^[2]. The narrower formats trade precision for throughput and capacity; FP8 uses half as many bits per number as BF16. Vector performance is quoted separately: 28.7 TFLOPS in BF16 and 14.3 TFLOPS in FP32 ^[1][2].

Networking built into the module

The OAM module uses PCIe Gen5 x16 with 128 GB/s of bidirectional bandwidth, and it integrates 24 200GbE RoCE ports carried over 48 112G PAM4 Serdes ^[1]. RoCE, RDMA over Converged Ethernet, lets one accelerator read from and write to another's memory across an Ethernet fabric without routing the traffic through the host processor. Here those Ethernet ports sit on the accelerator module itself rather than on a separate card.

Power and cooling

Thermal design power is 900 W with air cooling and 1,200 W with liquid cooling ^[1]. The gap between the two figures reflects how much more heat a liquid loop can pull off a module of this density.

Price and availability

Intel said an eight-accelerator Gaudi 3 baseboard would cost about US$125,000, which works out to roughly US$15,625 per chip ^[2]. Systems from Dell, HPE and Supermicro were due in the fourth quarter of the launch year ^[2].

Where it sits

Intel published the architecture at Hot Chips 2024 ^[1]. Taken together, the design puts large HBM2e capacity and bandwidth, equal BF16 and FP8 matrix throughput, and integrated 200GbE networking onto a single module, and Intel paired that with a per-chip price of about US$15,625 when bought as part of an eight-accelerator baseboard ^[1][2].