Meta releases Llama 3

Background

Meta's Llama models were openly available language models, and Llama 3 was the next generation after Llama 2.

What happened

On 18 April 2024 Meta released Llama 3 in 8B and 70B sizes, in pretrained and instruction-tuned versions. Meta said the models set new state-of-the-art results at their scales, and that models over 400 billion parameters were still training.^[1]

Key details

The models were pretrained on over 15 trillion tokens, which Meta put at seven times Llama 2's data. They use a 128K-token vocabulary and an 8K context length.^[1][2]

On instruction-tuned benchmarks the 8B model scored 68.4 on MMLU and 62.2 on HumanEval. The 70B model scored 82.0 on MMLU and 81.7 on HumanEval.^[2]

The models were available on AWS, Databricks, Google Cloud, Hugging Face, Kaggle, IBM watsonx, Microsoft Azure, NVIDIA NIM and Snowflake, under the Meta Llama 3 Community License.^[1][2]

The same day, Meta AI was rebuilt on Llama 3, received the meta.ai website, and rolled out in English to 13 more countries, including Australia, Canada, Nigeria and Singapore.^[3]

A model's vocabulary is the set of tokens it can read and write, and its context length is how much text it can consider at once. Instruction-tuned versions are further trained to follow requests, while pretrained versions are trained only to predict text. MMLU and HumanEval are standard benchmarks that measure general knowledge and coding ability.^[2]

Both sizes were released in pretrained and instruction-tuned versions, so developers could choose the one that suited their work.^[1]

Llama 3 was offered through many cloud platforms as well as Hugging Face, so developers could use it in the environment they already worked in.^[1]

Impact

Llama 3 gave developers a widely available model in two sizes that could be run through many cloud platforms. The release also carried Llama 3 into Meta AI, which put the model in front of users in Meta's apps.^[1][3]