Code Llama

What it is

Code Llama is a family of language models specialised for code. Meta released it on 24 August 2023. It was created by further training Llama 2 on code-specific datasets, so it keeps the general language base of Llama 2 and adds a strong focus on programming.^[1]

Specialised here means the training mix leans heavily on source code and code-related text. A model trained this way is better at tasks such as writing functions and completing a half-written file, but it is still a language model that predicts text one token at a time.^[1][2]

Origins

The paper, *Code Llama: Open Foundation Models for Code*, had 26 authors. The first author was Baptiste Rozière, and it was submitted to arXiv on 24 August 2023.^[2]

The first sizes were 7B, 13B and 34B parameters. Together they were trained on 500 billion tokens of code and code-related data. A 70B version followed on 29 January 2024, trained on 1 trillion tokens.^[1]

Meta offered three variants. The base Code Llama is the general code model. Code Llama - Python was further trained on 100 billion tokens of Python. Code Llama - Instruct was tuned to follow natural-language instructions.^[1]

Main features

Long sequences. All models were trained on sequences of 16,000 tokens and show stable generations with up to 100,000 tokens of context. A token is a small unit of text, roughly a word or part of a word, and a longer context lets the model see more of a project at once.^[1][2]

Filling in the middle. The 7B and 13B base and instruct models were trained with fill-in-the-middle. This means they can insert code into the middle of existing code, using what comes before and after the gap, rather than only continuing from the end.^[1]

Launch scores. Benchmarks such as HumanEval and MBPP ask a model to write code that passes a set of tests. At launch, Code Llama 34B scored 53.7% on HumanEval and 56.2% on MBPP. Meta said this was the best among the open models it compared and on par with ChatGPT.^[1]

Paper results. The paper reports scores of up to 67% on HumanEval and 65% on MBPP across the family. These figures cover the family as a whole rather than the 34B model alone.^[2]

Hardware. Meta said the 7B model can run on a single GPU. The smaller models suit low-latency tasks, such as real-time code completion, where the suggestion has to appear while the developer is still typing.^[1]

Licence. Code Llama was released under the same community license as Llama 2, which is free for research and commercial use.^[1]

Its place in development

Code Llama shows how a general language model can be made more useful for one field. It did not start from nothing. It took a base that already existed and trained further on code.^[1]

Its launch scores were measured against open models and ChatGPT. The later paper figures show that the family as a whole reached higher scores than the 34B launch numbers.

Meta's 70B version, released in January 2024, extended the family to its largest size, with a training run of 1 trillion tokens.^[1]