Meta releases Code Llama

Background

Large language models can write and explain code, but general models are not always strong at programming. Meta set out to build a model family specialised for code, starting from its Llama 2 model.^[1]

What happened

On 24 August 2023 Meta released Code Llama. It was built by further training Llama 2 on code. The family came in 7B, 13B and 34B sizes, in base, Python and Instruct variants, and was trained on 500 billion tokens of code.^[1]

Key details

Meta reported that Code Llama 34B scored 53.7% on HumanEval and 56.2% on MBPP. HumanEval and MBPP are benchmarks that test whether a model can write working programs for given problems. Meta said these results were on par with ChatGPT.^[1]

The models were trained on 16K-token sequences and can handle inputs of up to 100K tokens. A long input window lets the model read large amounts of code at once.^[2]

Code Llama used the same community licence as Llama 2, which is free for research and commercial use.^[1]

On 29 January 2024 Meta added Code Llama 70B, trained on 1 trillion tokens.^[1]

HumanEval and MBPP both give a model a description of a programming task and check whether the code it writes passes tests. Scores are reported as the share of problems solved. Instruct variants are tuned to follow natural-language instructions, while base models simply continue the code they are given.^[1]

Because Code Llama was built on Llama 2, its community licence carried over from that base model.^[1]

The 70B version, added in January 2024, extended the family to a larger size than the first release.^[1]

Impact

Code Llama gave developers a set of openly licensed code models that could be downloaded and used in their own tools. The release extended Llama 2 with further training on code and kept the same community licence.^[1][2]