Microsoft Azure Maia 100

What it is

Azure Maia 100 is Microsoft's first custom chip designed for artificial intelligence. The company announced it at its Ignite developer conference in November 2023 ^[1]. Unlike a general-purpose processor, it is a chip Microsoft intends to use for its own cloud service: the company said it would not sell the chips and would use them only in its own Azure data centres ^[1].

That makes Maia 100 a piece of internal infrastructure rather than a product customers buy. It sits in the same racks as the rest of Microsoft's cloud hardware, and its job is to run the large models that the cloud hosts.

How it is built

According to reporting by China Times, Maia 100 is manufactured on TSMC's 5-nanometer process and carries 105 billion transistors ^[1]. Transistor count and manufacturing process are the two numbers the chip industry usually quotes first: a finer process packs more devices into the same area, and a higher transistor count generally means more logic and memory can sit on one die.

The same report says the chip supports data types below 8 bits, which it described as Microsoft's first support for the MX data type ^[1]. Precision is a long-standing trade-off in AI hardware. Model weights and activations are numbers, and storing each one in fewer bits means less data has to be moved and multiplied, which makes computation cheaper and faster; the cost is that very low precision can lose accuracy unless the format and the training method are designed around it. Formats narrower than 8 bits are one way chip designers try to keep the arithmetic units busy without moving so much data.

What it is for

China Times reported that the chip was designed for training and inference of large language models such as GPT-3.5 Turbo and GPT-4 in Microsoft's cloud ^[1]. Training is the expensive process of adjusting a model's parameters on large amounts of data; inference is running the finished model to answer requests. Both are heavy users of the same kind of arithmetic, which is why a single chip design can be aimed at both.

The same report said OpenAI had already tested Maia 100 on GPT-3.5 Turbo ^[1], and that the chip would come to market in 2024 ^[1].

Announced alongside Cobalt 100

Maia 100 was not announced alone. Microsoft also introduced the 128-core Cobalt 100, an Arm-based server CPU, at the same time ^[1]. The pairing is telling: an AI accelerator does not work by itself. It needs a general-purpose processor to orchestrate data movement, networking and the rest of the software stack around it, so a chip programme usually involves both kinds of silicon.

Where it sits

Maia 100 marks a change in how Microsoft sources computing power. For years the accelerators in a cloud data centre came from outside suppliers; designing a chip in-house means the hardware can be shaped around the specific models and workloads the operator expects to run, and around the software it already owns. The 105-billion-transistor, 5-nanometer specification reported for Maia 100 ^[1] puts it in the class of parts built for serious model training rather than small edge devices.

What the announcement does not settle is scale. A chip only matters once it is deployed in volume, with a mature compiler, libraries and operational experience behind it, and Microsoft's stated plan of keeping Maia 100 inside its own data centres ^[1] means the evidence of how well it works will show up as Azure capacity rather than as a retail product. The reported 2024 arrival ^[1] is the point at which that becomes observable.