Llama 2

What it is

Llama 2 is a family of open-weight language models that Meta released on 18 July 2023, with Microsoft as its preferred partner. Meta's announcement stated that "Llama 2 is free for research and commercial use" ^[2]. The family comes in three sizes — 7B, 13B and 70B parameters — and each size is published twice: once as a pretrained base model and once as Llama 2-Chat, a version fine-tuned for dialogue ^[3]. All of them are auto-regressive language models built on an optimised Transformer, with a 4k-token context window at every size. The 70B model uses grouped-query attention (GQA), a scheme in which several attention heads share one set of key and value projections, which keeps memory use during generation lower than giving every head its own ^[1].

The context window is the amount of text the model can hold in view at once, so 4k tokens means a few thousand words of prompt and answer together.

It was not Meta's first release of this kind. Llama 1 had been offered to researchers on request, and Meta said that model drew more than 100,000 requests for access ^[2]. Llama 2 differed in being distributed as weights under a commercial licence rather than as a gated research arrangement.

How it was trained

The pretrained models were trained on 2.0 trillion tokens of data with a September 2022 cutoff, and training ran between January and July 2023 ^[1]. Meta reports 3.3 million GPU hours on A100-80GB hardware and estimated emissions of 539 tCO2eq for pretraining, which it says were fully offset ^[1].

The Chat versions were not the base models with a different prompt. They were tuned in two stages: supervised fine-tuning on human-written answers, and reinforcement learning from human feedback (RLHF), in which people rank candidate responses and a reward model learned from those rankings is used to steer the model toward the preferred ones. Meta used over one million new human-annotated examples for this step, and red-teamed the fine-tuned models both internally and externally before release ^[1].

Results

The accompanying paper is credited to Hugo Touvron, Louis Martin, Kevin Stone and others, and was first submitted to arXiv on 18 July 2023, with a revision on 19 July 2023 ^[1]. It reports that Llama 2-Chat beat other open-source chat models on most of the benchmarks tested, and that human evaluations of helpfulness and safety suggest it may substitute for closed-source models ^[1]. Both comparisons are the authors' own, made against the open chat models they had available to test.

Licence and distribution

Llama 2 is distributed under a custom commercial licence, the Llama 2 Community License, rather than a standard open-source licence ^[3]. The weights were offered through the Azure AI model catalog, Amazon Web Services and Hugging Face, and the intended use is in English ^[3]. That combination — downloadable weights, a commercial licence and distribution through several cloud providers — is what made the release notable: researchers could inspect and adapt the model, and companies could build on it without negotiating a separate agreement.

Where it sits

Llama 2 was followed by Llama 3 on 18 April 2024, released at 8B and 70B parameters and pretrained on over 15 trillion tokens ^[4].