SambaNova SN40L

What it is

SambaNova SN40L is an AI chip that SambaNova describes as a Reconfigurable Dataflow Unit (RDU). The design pairs a dataflow style of parallel execution with a memory system organised in three tiers ^[1]. Both halves of that description matter. In a conventional processor, an instruction stream shuttles data between memory and arithmetic units; in a dataflow design, the computation is laid out as a graph and data items move through it, with each processing unit acting as its inputs arrive. The second half — the memory hierarchy — determines how much model state can sit close to that arithmetic.

How it is built

SambaNova says the chip is fabricated on TSMC's 5-nanometer process and packaged as a 2.5D chip-on-wafer-on-substrate design. Each package carries two dataflow dies together with high-bandwidth memory (HBM) ^[1]. Co-packaging the memory rather than placing it elsewhere on a board shortens the distance between compute and the memory that feeds it, which matters for a design whose purpose is to keep arithmetic units supplied with data.

Compute

Per socket, SambaNova lists 638 BF16 TFLOPS of peak compute, delivered through 1,040 Pattern Compute Units ^[1]. BF16 is a 16-bit floating-point format that large models commonly use, so the figure describes peak throughput in a format that workloads actually run in. The number is a peak rating, not a measured result on a particular workload.

Three tiers of memory

The memory hierarchy is where much of the design's character comes from. On chip there are 520 MiB of SRAM; on the package there are 64 GiB of co-packaged HBM; and off the package there is up to 1.5 TiB of DDR DRAM ^[1]. Bandwidth drops as capacity grows: 1.8 TB/s for the HBM and 200 GB/s for the DDR ^[1].

The logic of such a hierarchy is general to computer architecture. The top tier is small and very fast, so it holds values that are reused immediately and repeatedly; the middle tier holds the working set that must be reached quickly but does not fit on chip; the bottom tier supplies capacity, accepting lower bandwidth in exchange. A large model's weights and the intermediate state produced while it runs can vastly exceed any single level, so where each piece of data lives at a given moment — and how often it has to be moved — becomes a first-order determinant of performance. SambaNova's three tiers give the compiler and runtime three distinct places to put data instead of one.

Composition of experts

SambaNova's Samba-CoE system running on the SN40L runs 150 expert models of seven billion parameters each, for a trillion parameters in total ^[1]. In a composition-of-experts arrangement, a request is handled by some subset of the available models rather than by one monolithic network, so the memory system has to hold many separate sets of weights and move between them efficiently. That is the workload the three tiers are built for: enough total capacity to keep a trillion parameters of experts addressable, and enough bandwidth in the upper tiers to keep the units busy while a request moves among them.

Scaling out

A single chip is not the largest unit SambaNova describes. Sixteen SN40L chips connected peer-to-peer provide 10.2 BF16 PFLOPS, and SambaNova says such a configuration can serve Llama 3.1 405B ^[2]. Serving a model of that size means the weights no longer fit on one socket, so the interconnect between chips becomes part of the memory system in practice.

Where it sits

The SN40L belongs to a line of thinking that treats memory, not arithmetic, as the binding constraint on large-model computing. SambaNova published the design in a paper whose title frames the problem as the AI memory wall and presents dataflow execution together with composition of experts as the response ^[1]. The chip is also presented by SambaNova specifically in the context of inference, with a separate account of why the company considers it well suited to that job ^[2]. Read together, the two sources describe a chip whose distinctive choices — a reconfigurable dataflow datapath, a deep memory hierarchy, and support for many small expert models rather than a single large one — are all aimed at the same problem: keeping a very large model fed with data without stalling on memory traffic.