Chatbot Arena / LMArena

What it is

Chatbot Arena is a benchmark platform for large language models (LLMs) built around human preference. A visitor is shown two answers to the same question and picks the one they think is better. The two models are presented anonymously and paired at random, so the person voting does not know which systems produced the answers^[1]. The votes are then aggregated into a public leaderboard that ranks models with the Elo rating system, the scheme borrowed from chess in which beating a strong opponent earns more points than beating a weak one, and losing to a weak opponent costs more^[1].

How it works

The platform's method is pairwise comparison: rather than scoring models against a fixed test set with known answers, it collects preferences from a diverse user base through crowdsourcing, and each vote is a single comparison between two anonymous systems^[2]. Within several months of operation it had amassed over 240K votes^[2]. The authors reported in a 2024 paper that questions submitted by the crowd are sufficiently diverse and discriminating, and that votes from ordinary users agree well with the judgments of expert raters^[2]. Because visitors type whatever they are curious about, the evaluation covers prompts that a hand-written benchmark is unlikely to contain — but the scores reflect what the crowd asks and prefers, not correctness on a fixed task.

Origins

LMSYS Org released Chatbot Arena on May 3, 2023^[1]. The first leaderboard was computed from votes collected between April 24 and May 1, 2023, and placed LMSYS's own vicuna-13b in first position with an Elo rating of 1169^[1]. The project grew out of academic work rather than a company: it was spun out of an academic project run by researchers and students at UC Berkeley^[4][5].

Reception

By the time of the 2024 paper, Chatbot Arena had emerged as one of the most referenced LLM leaderboards^[2]. Its appeal lay in the fact that the ranking came from real comparisons between real systems rather than from a static test set, and that new models could be added and evaluated in weeks.

Criticism

In 2025 a paper titled "The Leaderboard Illusion", written by researchers including Cohere-affiliated authors, examined how the arena's data and rules interact with who is able to use them. It found that undisclosed private testing practices benefit a handful of providers, who can test multiple variants before a public release and retract scores if they want to, and it identified 27 private LLM variants tested by Meta in the lead-up to the Llama-4 release^[3]. The same paper estimated that Google and OpenAI received 19.2% and 20.4% of all data on the arena respectively, while 83 open-weight models combined received only about 29.7%, and it estimated on conservative assumptions that access to Arena data could yield relative performance gains of up to 112% on the arena distribution^[3].

Relaunch as LMArena

On May 21, 2025 the team behind Chatbot Arena announced it had raised $100M in seed funding and was relaunching as LMArena with a rebuilt UI^[4]. The seed round was led by a16z and UC Investments, with participation from Lightspeed, Laude Ventures, Felicis, Kleiner Perkins and The House Fund, and Bloomberg reported the company valued at $600 million^[4][5]. Incorporated as Arena Intelligence Inc., the company was spun out of the academic project by researchers and students at UC Berkeley; co-founders include CEO Anastasios Angelopoulos, Ion Stoica and Wei-Lin Chiang^[4][5]. According to the company, more than 400 model evaluations had been conducted on LMArena and more than three million votes cast^[5].

Where it stands

Chatbot Arena turned preference voting into a routine part of how new models are presented: a model's arena standing is now something developers cite alongside conventional benchmarks, and the platform's scale made it a source of training and evaluation data as well as a scoreboard. Its history also shows the tension built into that role. A leaderboard that people trust because it reflects real usage is the same leaderboard that rewards whoever can run the most private trials and collect the most votes, which is what the 2025 critique was about.