Meta MTIA v1

What it is

MTIA v1 is the first-generation chip that Meta designed in-house for AI inference. Inference is the stage where a model that has already been trained is run to produce outputs — a ranking, a recommendation, a prediction — rather than the stage where its parameters are learned. Meta describes MTIA v1 as an inference accelerator designed for its deep learning recommendation models ^[1].

Where it came from

Securities Times reported that Meta announced its first-generation MTIA AI inference accelerator in May 2023 ^[2]. Meta's own account says the company unveiled MTIA v1, its first-generation in-house AI inference accelerator, the previous year ^[1]. An in-house chip means the company defines the hardware itself rather than buying a general-purpose accelerator off the shelf.

Process, clock and power

Meta's specification table lists TSMC's 7-nanometer process for MTIA v1 ^[1], and Securities Times also reported the 7 nm process ^[2]. The table gives an 800 MHz clock and a 25 W thermal design power ^[1]; the 25 W design power is also reported by Securities Times ^[2]. Thermal design power is the amount of heat a cooling system is expected to remove, and it works as a rough power budget for the chip. The 800 MHz clock is the rate at which the chip's logic is driven.

Compute

The same table lists 102.4 teraflops of INT8 and 51.2 teraflops of FP16/BF16 GEMM performance ^[1]. A teraflop is a trillion operations per second, and GEMM is general matrix multiplication, the matrix-times-matrix operation that dominates much of neural-network computation. INT8 stores numbers as 8-bit integers, while FP16 and BF16 use 16-bit floating-point formats. Fewer bits per number let the same hardware process more values per cycle, which is why the INT8 figure is twice the 16-bit figure here ^[1].

Memory

Meta's table lists 128 MB of on-chip memory and 64 GB of off-chip LPDDR5 ^[1]; Securities Times likewise reported 128 MB of on-chip memory ^[2]. The table also lists 800 GB/s of on-chip memory bandwidth and 176 GB/s of off-chip LPDDR5 bandwidth ^[1]. On-chip memory is small and sits close to the compute units, so it can be read at high bandwidth; off-chip LPDDR5 is much larger but is reached through a memory interface with lower bandwidth. LPDDR5 is a low-power memory standard also used in phones and other energy-sensitive devices.

Why a recommendation chip cares about memory

Recommendation models rely heavily on embeddings: vectors that represent users, items or features, looked up from tables and fed into the rest of the network. The lookups do relatively little arithmetic but touch very large tables, so capacity and bandwidth often matter as much as raw compute. The 64 GB of off-chip LPDDR5 provides room for those tables, and 176 GB/s is the rate at which data can be streamed in ^[1]. The 128 MB of on-chip memory with 800 GB/s of bandwidth lets compute units reuse data locally instead of returning to off-chip memory for every access ^[1].

Place in the line

MTIA v1 is the first generation of Meta's MTIA family ^[1]. Its published specifications — 7 nm, 800 MHz, 25 W, 102.4 INT8 teraflops, 128 MB on-chip and 64 GB off-chip memory — describe a part aimed at inference in production systems ^[1][2]. It is an inference chip rather than a training chip, and its memory hierarchy is sized for the table-heavy workload of recommendation models.