PaLM

What PaLM is

PaLM, short for Pathways Language Model, is a language model with 540 billion parameters, announced by Google Research on 4 April 2022^[1]. Architecturally it is a dense, decoder-only Transformer^[1]. "Decoder-only" means the network is trained to continue a piece of text by predicting the next token, rather than to encode a sentence for a separate classifier; "dense" means the full parameter set is engaged for every token processed, unlike mixture-of-experts designs that send each token through a subset of the network. The first word of the name comes from the Pathways system with which it was trained^[1].

How it was trained

The training run spanned 6,144 TPU v4 chips in two Cloud TPU v4 Pods, used the Pathways system, and reached 57.8% hardware FLOPs utilisation^[1]. Google Research described this as the largest TPU-based system configuration used for training to date^[1]. Keeping one model running across more than one pod is the coordination problem the Pathways system is meant to address, and the utilisation figure says how much of the hardware's theoretical arithmetic throughput was actually spent on useful work.

The corpus mixed English and multilingual web documents, books, Wikipedia, conversations and GitHub code, and 22% of it was non-English^[1]. Including code and conversation alongside ordinary prose means the model saw formal, structured sequences as well as written text.

What it could do

Google reported that PaLM 540B surpassed the few-shot performance of prior large models on 28 of 29 widely used English NLP tasks^[1]. On grade-school mathematics it was tested with chain-of-thought prompting, a style of prompt in which the model is shown worked examples that spell out intermediate reasoning steps before the final answer. With 8 such examples, PaLM solved 58% of GSM8K problems; the previous best of 55% came from GPT-3 175B fine-tuned on 7,500 problems and paired with an external calculator and verifier^[1].

On BIG-bench, with 5-shot prompting, PaLM 540B performed better than the average human on the same tasks^[1]. Given two-shot prompts, it could also explain original jokes that were not found on the web^[1] — a task that draws on background knowledge about the world and on reasoning about why a setup and a punchline fit together, rather than on retrieving an explanation that already exists somewhere.

The paper

The technical report, "PaLM: Scaling Language Modeling with Pathways", was first submitted to arXiv on 5 April 2022, the day after the announcement^[1][2]. It describes PaLM as a 540-billion-parameter, densely activated Transformer language model trained with the Pathways system^[2].

Where it sits

PaLM belongs to a line of work that asks how far a single general-purpose language model can be pushed by making it larger and feeding it more varied text. Two of its reported results are the ones most often pointed to: the comparison against a fine-tuned model with external tools on GSM8K, and the comparison against average human performance on BIG-bench^[1]. Both were reported by Google alongside a training run that also served as a demonstration of the Pathways infrastructure, coordinating thousands of chips at 57.8% utilisation^[1].