AWS Inferentia2

What Inferentia2 is

AWS Inferentia2 is a chip built by Amazon Web Services for inference: running an already-trained neural network to produce answers, as opposed to training the network in the first place. It is the second generation of the Inferentia family, and each chip carries two second-generation NeuronCores, the compute units on which a model's operations execute^[1].

Inside the chip

Each Inferentia2 chip carries 32 GB of high-bandwidth memory (HBM). AWS describes that as four times the total memory and ten times the memory bandwidth of the first generation^[1]. Memory bandwidth matters in inference because a large model streams its weights out of memory on every pass; if memory cannot keep the compute units fed, the arithmetic units sit idle.

The chip is rated at up to 190 TFLOPS in FP16, a measure of how many half-precision operations it can perform per second^[1]. It accepts a range of numeric formats — FP32, TF32, FP16, BF16 and INT8, plus a configurable FP8 data type^[1]. Offering several precisions lets a user keep sensitive parts of a model in a wider format while running the bulk of the work in a narrow one, which cuts memory traffic and speeds execution. FP8 is the narrowest of the set at eight bits per number, and making it configurable means the split between exponent and mantissa can be chosen to suit the range and precision a particular model needs.

Systems: EC2 Inf2

The chip is reached through EC2 Inf2 instances. One instance can hold up to 12 Inferentia2 chips and up to 384 GB of shared accelerator memory^[2]. Set against the earlier Inf1 instances, AWS states that Inf2 delivers three times higher compute, four times more accelerator memory, up to four times higher throughput and up to ten times lower latency^[2].

Inf2 instances were the first inference-optimised EC2 instances to support scale-out distributed inference with ultra-high-speed chip-to-chip connectivity^[1]. Distributed inference means splitting a single model across several chips or instances, which is needed when a model does not fit on one device or when one device cannot meet a latency target. Fast links between chips keep the hand-offs between pieces of the model cheap enough that splitting the work does not swallow the gain.

Efficiency

AWS rates Inf2 at up to 50% better performance per watt than comparable EC2 instances^[1]. For inference this figure counts for as much as raw throughput, because the cost of serving a model comes from running many machines continuously rather than from a one-off training run.

Features aimed at real workloads

Inferentia2 adds hardware optimisation for dynamic input sizes, for custom operators written in C++, and for stochastic rounding^[1]. Dynamic input sizes are inputs whose length or shape changes from request to request, as happens with text of varying length; handling that in hardware avoids repeated reshaping work. Support for custom operators written in C++ lets teams run operations that a standard library does not cover without falling back to a general-purpose processor. Stochastic rounding rounds a low-precision result up or down with a probability tied to its distance from the exact value instead of always taking the nearest representable number, which over many operations avoids the systematic bias that plain rounding introduces.

Where it sits

Inferentia2 is a chip dedicated to inference rather than a general-purpose accelerator, so choosing it means trading some flexibility for hardware tuned to a narrower job. Its design choices — large memory with high bandwidth, a spread of numeric formats down to FP8, chip-to-chip links for splitting models, and attention to performance per watt — are all aimed at the economics of serving models at scale rather than at the training run that produces them.