LLaMA: Open and Efficient Foundation Language Models

The question On February 27, 2023, Hugo Touvron, Thibaut Lavril, Gautier Izacard and 11 other Meta AI authors submitted the LLaMA paper to arXiv. The paper's central claim is that it is possible to train state-of-the-art foundation language models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets ^[1]. This matters because training data and model access shape who can reproduce, study and build on a model. The authors present LLaMA as a collection of foundation language models ranging from 7B to 65B parameters and trained on trillions of tokens ^[1].

What was built LLaMA is not a single model but a family. Its sizes run from 7B to 65B parameters; B denotes billion, so the range is from about 7 billion to about 65 billion learned numerical settings. The models were trained on trillions of tokens, where a token is a piece of text such as a word or part of a word. The paper states that the authors trained these models using publicly available datasets exclusively ^[1]. The work therefore combines capability with openness: the models are meant to be competitive, and their data sources are not hidden behind proprietary restrictions.

Training at scale The paper gives a concrete picture of the compute needed for the largest model. Training the 65B model processed around 380 tokens per second per GPU on 2048 A100 GPUs with 80GB of memory. At that rate, a run over 1.4 trillion tokens took approximately 21 days ^[1]. These numbers describe a single training run and show the scale of the work: thousands of accelerators working together for weeks, processing text in very large quantities. Throughput, GPU count and total token count together show how training time follows from processing speed and data volume; they are not a claim about every model in the family.

Results LLaMA-13B outperforms GPT-3, a 175B-parameter model, on most benchmarks. LLaMA-65B is competitive with the best models named in the paper, Chinchilla-70B and PaLM-540B ^[1]. The comparison is notable because of the parameter counts: the 13B model is much smaller than GPT-3, and the 65B model is smaller than PaLM-540B. The paper reports that the 13B model wins on most benchmarks and that the 65B model is competitive with the named large models. That distinction is important for reading the result accurately.

Why it mattered The paper shows that state-of-the-art models can be trained using publicly available datasets exclusively ^[1]. It also reports that all the models were released to the research community ^[1]. Together, these two points are the paper's open-model claim: the training data were not proprietary and inaccessible, and the resulting models were made available for research. The results also give a clear comparison point between model size and benchmark performance, since a 13B model was compared with a 175B model and a 65B model with 70B and 540B models ^[1]. For readers, the paper is therefore both a technical report on training a family of language models and an argument about how such models can be built and shared.

What to take away LLaMA's contribution is a family of foundation language models from 7B to 65B parameters, trained on trillions of tokens, using only public datasets, with reported benchmark results and a release to the research community ^[1]. Its scale numbers make the training process tangible: 2048 A100 GPUs, 80GB each, about 380 tokens per second per GPU, and about 21 days for 1.4 trillion tokens ^[1]. The paper's importance lies in combining those results with an explicit claim about public data and an explicit release of the models for research.