Google announces the 540-billion-parameter PaLM

Background

Large language models are trained on a simple objective: predict the next piece of text given what came before. Out of that objective come abilities that were not explicitly programmed, such as answering questions, translating, or writing code. In the years before this announcement, progress in the field was often reported in terms of parameter count, because adding parameters — the numbers the model learns during training — tended to improve performance across many different tasks at once. The open question was how far that trend could be pushed, and what new behaviour would appear at the far end of it.

The announcement

On 4 April 2022, Google Research announced PaLM, the Pathways Language Model, with 540 billion parameters.^[1] The name joined two things the team wanted to highlight: the size of the model, and the training system used to build it.

How it was trained

PaLM was trained on 6,144 TPU v4 chips spread across two pods, using the Pathways system.^[1] TPUs, or tensor processing units, are accelerators designed for machine learning; a pod is a group of them connected by fast links so that they can be treated as one large machine. Google Research described the 6,144-chip setup as the largest TPU-based system configuration used for training to date.^[1] Pathways is the software layer that coordinates the training of a single model across many chips, so that a model of this size can be trained without carving it into pieces that barely talk to each other.

Results

The headline result concerned reasoning rather than raw text generation. Google highlighted breakthrough performance on multi-step reasoning when the model was used with chain-of-thought prompting.^[1] Chain-of-thought prompting means that instead of asking only for a final answer, the prompt shows worked examples in which the intermediate steps of a solution are written out; the model then produces its own intermediate steps before concluding. Splitting a problem into steps gives the model room to compute, in text, what it would otherwise have to do in a single leap.

The same effect showed up on BIG-bench, a large and deliberately varied collection of benchmark tasks. In a 5-shot setting — five worked examples placed in the prompt — PaLM outperformed the average human on the benchmark.^[1] The human baseline matters here: it means the comparison was not only against other models but against people solving the same tasks.

A smaller but telling demonstration was that the model could explain jokes it had not seen on the web.^[1] Explaining why a joke is funny is not a matter of retrieval; it requires identifying the expectation the setup creates, noticing how the punchline breaks it, and saying so in plain language.

The paper

The technical paper, "PaLM: Scaling Language Modeling with Pathways," was first submitted to arXiv on 5 April 2022, the day after the announcement.^[2] It documented the training setup and the evaluations behind the claims in the announcement.

Why it matters

PaLM was notable for combining scale with a prompting technique, and for measuring the result on benchmarks that came with human baselines. Reporting a model against the average human on a broad task collection, and demonstrating reasoning written out step by step, made the case that large models were doing something more than surface pattern matching. The 6,144-chip configuration also stood as a concrete data point on how much hardware could be coordinated for a single training run, which is the practical constraint behind every model of this size.