01.AI open-sources the Yi-Coder code models

Background

Code generation is one of the areas in which AI models are tested most often. A code model is trained mainly on programming material, so that it can write, complete and explain code. For a company that had already released general-purpose open models, a code model was a natural next step ^[1].

To understand the release, it helps to know how code models are judged. Benchmarks are standard tests run on every model, and the most common ones ask a model to write a function that passes a set of unit tests. Newer benchmarks use recent contest problems or ask a model to predict the output of code, because these are harder to answer by memory.

What was released

01.AI released Yi-Coder as open source on 5 September 2024, China time ^[2]. The family has two sizes, 1.5B and 9B, and each size has a base version and a chat version ^[1]. The base versions continue code, while the chat versions follow instructions.

The 9B model continued training from Yi-9B on 2.4 trillion additional tokens, covering 52 programming languages ^[1]. A token is a small unit of text, often a word or part of one, and 2.4 trillion is a very large amount of training material. Covering 52 languages means the model was trained on code written in many different languages, not only one.

The models support contexts of up to 128K tokens ^[1]. That means a single input can be very long, which is useful when a developer wants the model to read a whole file or several files at once.

Key details

The 9B chat model scored 23.4% on LiveCodeBench. The company said this made it the only model under 10 billion parameters to score above 20% ^[1]. LiveCodeBench uses recent programming problems, so its scores are less likely to reflect answers the model has memorised.

The same model scored 85.4% on HumanEval and 73.8% on MBPP ^[1]. These two benchmarks test whether a model can write short functions that pass a set of tests. They are among the most widely quoted tests for code models, so the scores give readers a common point of comparison.

Impact

The release showed that a relatively small open model could be presented as competitive on code tests. For developers, a smaller model lowers the cost of trying it and of running it inside their own tools, compared with very large models.

The company's claims about the results should be checked independently. The benchmark numbers come from the company's own reports, and each benchmark tests a particular kind of ability. Readers who want to choose a code model should compare results on the tasks they care about.

For general readers, the release is a clear example of how specialised models fit into a company's wider line-up. Yi-Coder was not a replacement for the general models but an addition aimed at one kind of work, and it showed the company's interest in tools that programmers use directly.