AMD Instinct MI300X

What it is

AMD Instinct MI300X is a data-center GPU (graphics processor) built for generative AI. AMD first showed it on 13 June 2023 at its Data Center and AI Technology Premiere and made it available on 6 December 2023; it is based on the CDNA 3 architecture ^[4]. Accelerators of this kind sit in servers rather than on desks, and take on the two large jobs of modern AI: training a model on quantities of data, and then running that model to answer requests. Much of the MI300X's design is shaped by the second job, where the whole model has to stay in memory while it works.

A chiplet design

The MI300X is not one monolithic piece of silicon. It is a chiplet design: 5 nm compute dies are stacked with 6 nm I/O dies inside a single package, 153 billion transistors in total ^[2]. The idea behind chiplets is that smaller dies are cheaper to manufacture and easier to build up into a large part than one very big die, and stacking lets closely coupled circuitry sit near the compute. Inside the package are 304 compute units, 19,456 stream processors and 1,216 matrix cores running at up to 2,100 MHz ^[2]. Matrix cores are the units that carry out the dense multiplication-and-accumulation work that neural networks are largely made of.

Memory as the headline

The feature AMD led with is memory. The MI300X carries 192 GB of HBM3 — high-bandwidth memory, DRAM stacked in layers and reached through a very wide interface — on an 8,192-bit bus, with 5.3 TB/s of peak bandwidth and a 256 MB Infinity Cache ^[3]. Capacity decides whether a model fits on a single chip; when AMD revealed the part in June 2023 it compared the 192 GB with the 80 GB on Nvidia's H100 ^[3]. Bandwidth matters as well, because generating text is mostly a matter of streaming weights past the compute units over and over.

Compute and power

AMD's data sheet gives peak theoretical figures of 1,307.4 TFLOPS for FP16 and BF16 and 2,614.9 TFLOPS for FP8, doubling to 2,614.9 and 5,229.8 TFLOPS respectively when structured sparsity is used ^[1]. FP8 is a low-precision number format that halves the storage and movement per value, and sparsity support lets the hardware skip weights that are zero; both buy throughput at some cost in numerical precision, which is why peak numbers are ceilings rather than what a real workload will see. Maximum board power is 750 W ^[1], so the density of the chip is matched by its demands on power delivery and cooling.

Against the previous generation

Compared with the earlier MI250X, AMD cites nearly 40% more compute units, 1.5 times the memory capacity and 1.7 times the peak memory bandwidth, plus new support for FP8 and sparsity ^[3]. The direction of the change — more memory per chip, arithmetic at lower precision — tracks what work on large models had come to need.

Systems and software

Eight MI300X accelerators connected by AMD Infinity Fabric, with seven links of 128 GB/s each per GPU, form the AMD Instinct Platform, which holds 1.5 TB of HBM3 in total ^[2]. AMD said a single MI300X can run inference on a 70B-parameter model such as Llama 2 by itself ^[2], a concrete illustration of why capacity was the selling point. Among the customers named at launch were Microsoft's Azure ND MI300x v5 virtual machines and planned Oracle Cloud Infrastructure bare-metal instances, and AMD released its ROCm 6 open software platform alongside the hardware ^[3]. Software matters as much as silicon here: an accelerator is only useful to most AI teams if the frameworks and libraries they already depend on run on it.

Where it sits

The MI300X marks how data-center AI hardware became a contested market: a chip pitched explicitly against Nvidia's H100 on the one number — memory per accelerator — that most directly limits what a single device can serve. Interconnection, software and power draw all remain part of the picture alongside raw capacity.