Llama 3.1 (including 405B)

What it is

Llama 3.1 is an update to the Llama 3 family. It came out on 23 July 2024 in three sizes: 8B, 70B and 405B parameters. Meta called the 405B model the world's largest and most capable openly available foundation model.^[1][2] A foundation model is a general-purpose model that other applications and fine-tuned models can be built on.

All sizes have a 128K-token context and support eight languages: English, German, French, Italian, Portuguese, Hindi, Spanish and Thai.^[2]

Origins and training

The models were pretrained on about 15 trillion tokens, with a December 2023 knowledge cutoff.^[2] The 405B model was trained on over 16,000 H100 GPUs, the first Llama model trained at that scale.^[1][2]

Meta chose a standard decoder-only Transformer for the 405B model rather than a mixture-of-experts design, a choice it made for training stability.^[1][3] A mixture-of-experts design routes each input to a few specialist parts of the network, which can be more efficient but harder to train reliably.

The compute totals were 30.84 million H100 GPU hours for the 405B model, 7.0 million for the 70B model and 1.46 million for the 8B model, or 39.3 million in total. The location-based emissions were estimated at 11,390 tCO2eq.^[2]

The paper that describes the family, *The Llama 3 Herd of Models*, had 559 authors. It was first submitted on 31 July 2024.^[3]

Main features

Post-training. After pretraining, the models went through several rounds of supervised fine-tuning, rejection sampling and direct preference optimization. Rejection sampling keeps only the best of several candidate answers, and direct preference optimization trains the model on pairs of answers that people or models have ranked. The 405B model was also quantised from BF16 to FP8, so it could run on a single server node.^[1]

Benchmark results. The instruction-tuned 405B model scored 87.3 on MMLU, 73.3 on MMLU-Pro, 89.0 on HumanEval, 96.8 on GSM-8K, 73.8 on MATH, 50.7 on GPQA and 96.9 on ARC-C.^[2] Meta described the 405B model as competitive with GPT-4, GPT-4o and Claude 3.5 Sonnet across more than 150 benchmark datasets.^[1]

Licence. The Llama 3.1 Community License was changed to let developers use model outputs, including those from the 405B model, to improve other models.^[1][2]

Partners. More than 25 partners offered the models on the first day. They included AWS, NVIDIA, Databricks, Groq, Dell, Azure, Google Cloud and Snowflake. Meta said Llama models had passed 300 million total downloads.^[1]

Safety and tools. The release came with the Llama Guard 3 and Prompt Guard safety models, and a proposal for Llama Stack interfaces. The 405B model could be tried in the United States on WhatsApp and meta.ai.^[1]

Its place in development

Meta's chief executive, Mark Zuckerberg, published the letter "Open Source AI Is the Path Forward" the same day. In it he compared open AI to Linux.^[4]

Later releases followed the same line. Llama 3.2, released in September 2024, added vision and small on-device models. Llama 3.3 70B, released in December 2024, matched the 405B model on several benchmarks.^[5][6]

For a learner, the lesson of Llama 3.1 is in the numbers. Its 405B model shows what the open approach could reach at the largest scale. The later 70B result shows that a smaller model could match it on several tests.^[2][6]