01.AI's Yi-Lightning ranks sixth on Chatbot Arena

Background

By the autumn of 2024 the large language model field had become a crowded contest. Companies in the United States, China and elsewhere were releasing flagship models and asking the public to judge them in shared arenas. 01.AI launched Yi-Lightning as its new flagship in this context ^[3].

Public rankings matter in this contest because they give a common reference point. The LMSYS Chatbot Arena collects votes from people who compare answers from two anonymous models without knowing which is which. The votes are turned into a ranking, so a model's position reflects what users prefer rather than what its developers claim.

What happened

In October 2024 01.AI launched Yi-Lightning as its new flagship model ^[3]. The model is built on an enhanced Mixture-of-Experts design, in which the network is divided into many specialised experts and only some of them work on each input ^[2].

Kai-Fu Lee said that Yi-Lightning ranked sixth globally and first in China on LMSYS's blind-test ranking, ahead of GPT-4o-2024-05-13 and Claude 3.5 Sonnet ^[1]. Those two models were the competing systems at the time of the ranking.

The model also placed second to fourth in the Chinese, Math, Coding and Hard Prompts categories of Chatbot Arena ^[2]. Each category tests a different kind of question, so a model can rank differently in each one.

Key details

The API for Yi-Lightning was priced at 0.99 yuan per million tokens ^[1]. A token is a small unit of text, and per-million-token pricing lets developers estimate what a given workload will cost.

In the LMSYS ranking of model makers, 01.AI tied for third place with xAI. OpenAI and Google were ahead of them ^[3]. The ranking of model makers takes all of a company's models into account, rather than only one.

Impact

The launch showed that a company from China could reach the top of a public arena that is used to compare models from many countries. For readers, the event is an example of how a benchmark's ranking can change over time, since rankings reflect the models available on a given day.

The launch also shows how a company can present one result in several ways. Lee's remarks gave the sixth-place ranking and the first-place ranking among Chinese models, while the company-level ranking put 01.AI tied for third among model makers. All three statements came from the same arena data but focus on different groups, so the most accurate reading uses all of them together.

For readers who are new to AI, the event is a useful introduction to how models are compared. A ranking based on blind votes, a price per million tokens and a technical report describing the training are three different kinds of evidence, and each one tells a reader something different about the same model.