Meta releases Llama 3.1 405B, an open frontier-scale model

Background

Llama 3.1 was Meta's release of a model at this scale with openly available weights, following the Llama 3 release in April.^[1]

What happened

On 23 July 2024 Meta released Llama 3.1 in 8B, 70B and 405B sizes, with a 128K-token context and support for eight languages.^[1][2]

Meta called the 405B model the world's largest and most capable openly available foundation model. It was trained on over 15 trillion tokens using more than 16,000 H100 GPUs.^[1]

Key details

The 405B instruct model scored 87.3 on MMLU, 89.0 on HumanEval and 73.8 on MATH.^[2]

Meta said the model was competitive with GPT-4, GPT-4o and Claude 3.5 Sonnet.^[1]

The licence was changed to allow Llama outputs to be used to improve other models. More than 25 partners, including AWS, NVIDIA, Databricks, Groq, Dell, Azure and Google Cloud, offered it on the first day.^[1]

Mark Zuckerberg published “Open Source AI Is the Path Forward” the same day, arguing that open AI would follow Linux's path.^[3]

The technical paper “The Llama 3 Herd of Models” followed on 31 July 2024, with 559 authors.^[4]

A foundation model is a large general-purpose model that can be adapted to many tasks. Open weights let outside researchers and companies run the model on their own infrastructure instead of sending their data to a provider. The 405B model was the largest of the three sizes, and the 8B and 70B models were released alongside it.^[1]

Meta's blog post and the model card give the detailed specifications and benchmark tables.^[1][2]

The 405B model was the largest of the three sizes, and it is the version Meta compared with GPT-4, GPT-4o and Claude 3.5 Sonnet.^[1]

Impact

Llama 3.1 405B was the largest openly available model Meta had released at the time, and the licence change allowed others to use its outputs to train their own models. The paper and the essay accompanied the release.^[4][3][1]