Meta releases LLaMA to researchers

Background

Large language models (LLM) learn by reading very large amounts of text and practising the prediction of the next token, a small unit of text such as a word or part of one. Training such models is expensive. At the time, many of the most capable models were available only through paid interfaces or were not released publicly at all, which limited what outside researchers could study. Meta's release of LLaMA responded to this need for models that researchers could study and run themselves. Model sizes are usually described by parameter count. A figure such as 7B means the model has about seven billion learned numerical values, and more parameters generally require more computation to train and run.

What happened

On 24 February 2023 Meta announced LLaMA, a family of foundation large language models. Foundation models are general-purpose models meant to be adapted to many tasks. LLaMA came in four sizes: 7B, 13B, 33B and 65B parameters.^[1] The paper, “LLaMA: Open and Efficient Foundation Language Models,” was posted to arXiv on 27 February 2023. It has 14 authors, and its first author is Hugo Touvron.^[2]

Meta released the models under a noncommercial license focused on research use. Access was granted case by case to academic researchers, people at government, civil society and academic organisations, and industry research labs.^[1]

Key details

The 65B and 33B models were trained on 1.4 trillion tokens, and the 7B model on one trillion tokens.^[2] Meta said the models were trained exclusively on publicly available datasets. For language coverage, it chose text from the 20 languages with the most speakers, focusing on those written in Latin and Cyrillic alphabets.^[2]

The paper reported that LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and that LLaMA-65B is competitive with Chinchilla-70B and PaLM-540B.^[2] A benchmark is a standardised test used to compare models. A score on one benchmark does not capture everything a model can do, so such comparisons need to be read with that in mind.

Meta said the paper included evaluations of model bias and toxicity.^[2]

Impact

The paper's comparisons suggested that training a moderately sized model on more data could compete with much larger models on some tests. That points to training data and training length mattering alongside parameter count, not only the number of parameters.^[2]

The noncommercial license and case-by-case access kept LLaMA a research release. A noncommercial license permits research use but not use in products sold to customers, so companies building paid services could not rely on it. Its terms were narrower than those of the open releases that would follow, which is one reason the release is usually read as a research step rather than a product.

Offering several sizes from one family also allowed like-for-like comparisons. Researchers could see how results changed from the 7B model to the 65B model under the same general approach, which helps separate the effect of scale from other factors.