Beyond the Imitation Game (BIG-bench)

Why build another benchmark

A benchmark is a fixed collection of problems together with a way of scoring answers. Its usefulness depends on being hard: once models solve most of a benchmark's tasks, the score stops telling us much about how one system differs from another, and the tests have to be replaced with harder ones. BIG-bench was organized around that problem. It focuses on tasks that are believed to be beyond the capabilities of current language models, so that a high score means something rather than reflecting skills that are already routine.^[1]

A benchmark written by many hands

The paper, titled Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models, was submitted to arXiv on June 9, 2022 by Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao and a large list of co-authors.^[1] The size of that list reflects how the benchmark was made. BIG-bench consists of 204 tasks contributed by 450 authors across 132 institutions, making it a collective effort rather than the product of a single lab.^[1]

The breadth of the task set follows from that. Problems are drawn from linguistics, childhood development, mathematics, common-sense reasoning, biology, physics, social bias, software development, and beyond.^[1] The point of mixing such different subjects is that a model can be strong on one kind of text and weak on another; a benchmark that only asks about, say, word puzzles will miss a great deal. Some of the tasks ask for knowledge, some for reasoning over several steps, and some for judgements about social bias — different demands that are hard to satisfy with a single skill.

How the models were compared

The authors evaluated the behaviour of OpenAI's GPT models, of Google-internal dense transformer architectures, and of Switch-style sparse transformers, and they did so across model sizes.^[1] The families differ in how compute is spent inside the network: a dense transformer uses all of its parameters for every token it processes, while a sparse, Switch-style design routes each token through only part of the network, so that total parameter count and per-token cost come apart. Comparing all three families at several sizes lets the benchmark be used not just to rank systems but to look at how capability changes as models grow.

BIG-bench Lite

The paper also describes BIG-bench Lite, a curated subset of 24 tasks intended for lightweight evaluation.^[2] Running all 204 tasks is expensive, and a smaller fixed set makes it practical to check a model quickly and compare results between groups. Because the subset is curated rather than randomly sampled, it is meant to stand in for the full benchmark without covering every kind of task in it.

Why it matters

BIG-bench made two contributions that are easy to separate. The first is the task collection itself: a broad, openly described set of problems covering domains that narrower benchmarks leave out, built by hundreds of contributors from many institutions.^[1] The second is the framing. By aiming at tasks that current models were not expected to solve, the project treats a benchmark as a moving target rather than a fixed scoreboard, and it invites the field to write down what models cannot yet do.^[1] The evaluation across GPT models, Google's dense transformers and Switch-style sparse transformers shows the intended use: measuring many model families and sizes on the same ground, and reading the differences as a picture of where language-model capability stood and how it might extend.^[1]