The Llama 3 Herd of Models

A herd, not a single model

The paper "The Llama 3 Herd of Models" is credited to the Llama Team, AI @ Meta. Its first page carries the date 23 July 2024, and it was first submitted to arXiv on 31 July 2024 as 2407.21783^[1]. The word "herd" in the title is deliberate: the paper presents Llama 3 not as one network but as a group of foundation models, all built to natively support multilinguality, coding, reasoning and tool usage^[1]. "Natively" is the key claim — these are abilities the models are meant to acquire during training, rather than features bolted onto a text-only model afterwards.

Three sizes, one recipe

The herd contains three language models, with 8 billion, 70 billion and 405 billion parameters respectively^[1]. The largest of them is a dense Transformer with 405 billion parameters and a context window of up to 128K tokens^[1]. "Dense" here contrasts with sparse, mixture-of-experts designs: every parameter takes part in processing each token. The context window is the amount of text a model can hold in view at once; at 128K tokens it is long enough for a whole long document rather than a few pages.

Scale shows up most clearly in the numbers behind pre-training. The corpus held about 15 trillion multilingual tokens, against 1.8 trillion tokens for Llama 2^[1]. Compute grew in step: the flagship model was pre-trained with 3.8 × 10^25 FLOPs (floating-point operations, the usual unit for measuring training compute), almost 50 times more than the largest Llama 2 model^[1].

Post-training

Pre-training on raw text gives a model that can continue text but is not yet a reliable assistant. The paper's post-training stage combines three techniques: supervised fine-tuning, rejection sampling and direct preference optimization (DPO)^[1]. Supervised fine-tuning trains on curated example conversations. Rejection sampling has the model generate candidate answers that are then filtered, with the survivors used as further training data. DPO adjusts the model toward the answers people prefer, using pairs of better and worse responses instead of a separately trained reward model. Stacking the three means the model is first shown what good answers look like, then trained on its own filtered output, then tuned on human preference.

Results and release

The paper reports that Llama 3 delivers comparable quality to leading language models such as GPT-4 across a wide range of tasks^[1]. That is a comparison claim made in the paper itself, and it is the sentence most often cited from it.

What was shipped matters as much as what was measured. Meta released the pre-trained and post-trained versions of the 405B model, together with Llama Guard 3, a component for input and output safety^[1]. Releasing both versions means others can start either from the raw base model or from the tuned assistant, and the safety tool ships alongside rather than as an afterthought.

Beyond text

The paper also reports experiments that add image, video and speech capabilities to Llama 3 through what it calls a compositional approach^[1]. It states that those models are not yet broadly released because they remain under development^[1]. The herd, in other words, is documented more widely than it is distributed: the text models are the part the paper both measures and hands over, while the multimodal work is described as still in progress.