Llama 3

What it is

Llama 3 is a family of openly available language models from Meta. Meta released it on 18 April 2024 at two sizes, 8 billion and 70 billion parameters, and each size came in a pretrained version and an instruction-tuned version.^[1][2] The pretrained version is the base that learned from text. The instruction-tuned version was further trained to follow user requests in a chat style.

Both sizes use a decoder-only Transformer with grouped-query attention. Grouped-query attention shares key and value data across groups of attention heads, which cuts the memory needed while the model generates text. The family also uses a new tokenizer with a 128K-token vocabulary, and the context window is 8K tokens.^[1][2]

Origins and training

Meta pretrained Llama 3 on over 15 trillion tokens from publicly available sources. That is seven times the data used for Llama 2, and it includes four times more code. More than 5% of the data is non-English, covering more than 30 languages.^[1]

The knowledge cutoff is March 2023 for the 8B model and December 2023 for the 70B model.^[2]

Training ran on two custom-built clusters of 24,000 GPUs each. On 16,000 GPUs the model reached over 400 TFLOPS per GPU. Meta said training efficiency was about three times that of Llama 2.^[1]

The compute cost is recorded in the model card. Pretraining used 1.3 million GPU hours for the 8B model and 6.4 million for the 70B model, 7.7 million in total, with estimated emissions of 2,290 tCO2eq.^[2]

Main features

Benchmark results. The instruction-tuned models were tested on several benchmarks. For the 8B and 70B models, the scores were 68.4 and 82.0 on MMLU, 34.2 and 39.5 on GPQA, 62.2 and 81.7 on HumanEval, 79.6 and 93.0 on GSM-8K, and 30.0 and 50.4 on MATH.^[2] A higher score means better performance on that test, and comparing the two columns shows how much the larger model gained.

Licence. The models were released under the custom Meta Llama 3 Community License.^[2]

Platforms. At launch the models were available on AWS, Databricks, Google Cloud, Hugging Face, Kaggle, IBM watsonx, Microsoft Azure, NVIDIA NIM and Snowflake.^[1]

Meta AI. On the same day, Meta relaunched its Meta AI assistant on Llama 3. It was offered across Facebook, Instagram, WhatsApp, Messenger and the new meta.ai website.^[3]

Its place in development

Meta said that models with over 400 billion parameters were still training at launch. It planned to add multimodality, which means handling more than text, along with better multilingual ability and longer context.^[1]

The next release, Llama 3.1, came on 23 July 2024. It extended the context to 128K tokens and added a 405-billion-parameter model.^[4]

The benchmark figures above are the most direct record of what this release delivered, and they can be checked against the model card.^[2]