LLaMA

What it is

LLaMA is a family of foundation language models announced by Meta AI on 24 February 2023. The name stands for Large Language Model Meta AI ^[2]. It was released in four sizes — 7B, 13B, 33B and 65B parameters, where the number refers to the model's weights counted in billions ^[2]. A foundation model here means a general-purpose text model that others can adapt to particular tasks, rather than one built for a single application.

How it was trained

The authors trained LLaMA only on publicly available data ^[1]. The 33B and 65B models saw 1.4 trillion tokens during training, while the 7B and 13B models saw 1.0 trillion ^[1]. A token is a short chunk of text — roughly a word or a piece of a word — so a trillion tokens is a very large amount of reading material. The text came from the 20 languages with the most speakers, with a focus on the Latin and Cyrillic alphabets ^[1].

The architecture is a Transformer with three notable choices: pre-normalisation using RMSNorm, the SwiGLU activation function, and rotary positional embeddings (RoPE) ^[1]. Pre-normalisation means the input to each block is normalised before the block processes it, which tends to make deep networks easier to train. Positional embeddings carry information about where a token sits in the sequence; RoPE does this by rotating a token's vector by an angle that depends on its position.

What training cost

Training the 65B model processed about 380 tokens per second per GPU on 2,048 A100 80GB GPUs, so a single pass over 1.4 trillion tokens took about 21 days ^[1]. The paper reports these figures so that outside readers can judge for themselves what such a model costs to train.

Results

The paper's key claim, from Hugo Touvron and colleagues, is that LLaMA-13B outperforms GPT-3, a 175B-parameter model, on most benchmarks despite being more than 10 times smaller, and that LLaMA-65B is competitive with Chinchilla-70B and PaLM-540B ^[1]. In other words, the smaller members of the family were compared against models many times their size and reported to hold up on standard tests.

Release and access

LLaMA was released under a noncommercial license for research use, and access was granted case by case to academic researchers, government, civil society and industry research labs ^[2]. The paper states that the authors release all their models to the research community ^[1]. It was first submitted to arXiv on 27 February 2023 ^[1].

Where it sits in the field

Meta argued that smaller models trained on more tokens are easier to retrain and fine-tune, and that they let researchers without large infrastructure study large language models ^[2]. Both points are about who can take part in the work: fine-tuning a smaller model is cheaper than fine-tuning a very large one, and a model that runs on modest hardware opens the subject to groups that cannot train a model from scratch. The successor, Llama 2, arrived in July 2023 and allowed commercial use ^[2], moving the family from a research-only licence to one that businesses could adopt.