AWS Inferentia

What it is

Inferentia is a chip built by AWS for inference — the stage of machine learning where a model that has already been trained is used to make predictions. Rather than training models, the chip is designed to run them, and customers reach it through Amazon EC2, where it appears inside Inf1 instances.^[1][2]

How it is built

Each first-generation Inferentia chip contains four first-generation NeuronCores, the compute units that the software schedules work onto.^[1] The chip carries 8 GB of DDR4 memory, plus a large amount of memory on the chip itself.^[1] It supports the FP16, BF16 and INT8 data types, the lower-precision number formats commonly used to run models quickly.^[1]

A single EC2 Inf1 instance can carry up to 16 Inferentia chips, so one instance can hold many NeuronCores at once.^[1][2] Inf1 instances also use second-generation Intel Xeon Scalable processors and offer up to 100 Gbps of networking.^[2]

Programming model

Developers reach the chip through the AWS Neuron SDK, which integrates with the PyTorch and TensorFlow frameworks.^[1] That means a model written in one of those familiar frameworks can be compiled and run on Inferentia without the user having to write chip-specific code from scratch.

Why it matters

AWS states that Inf1 instances deliver up to 2.3 times higher throughput and up to 70% lower cost per inference than comparable EC2 instances.^[1] Inference is the part of machine learning that runs continuously once a model is in production, so throughput and cost per prediction matter a great deal to anyone serving a model to many users. A chip built specifically for that job, rather than a general-purpose processor, is one way to push both numbers in the right direction.

Where it sits

Inferentia is part of a broader move by large cloud providers to design their own silicon for machine learning rather than relying only on off-the-shelf hardware. For users, the appeal is that the chip is reached through the same EC2 instances and the same frameworks they already use, with the Neuron SDK sitting in between.