T5 (Text-to-Text Transfer Transformer)

What T5 is

T5, short for Text-to-Text Transfer Transformer, is a family of Transformer language models from Google. It was introduced in the paper "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer", written by Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li and Peter J. Liu, and first posted to arXiv on 23 October 2019.^[1]

One format for every task

The organising idea is in the name. Every language task — translation, question answering, classification, summarisation — is cast as text in, text out, so that one model, one loss and one training procedure serve all of them.^[1] Instead of giving each task its own output layer or scoring rule, the framework keeps the interface identical everywhere: the model is handed text and returns text.

The usual pattern in this line of work is to pre-train a model on a large body of unlabeled text and then fine-tune it on a smaller labelled set for the task at hand. A single format makes that pipeline uniform. The same training loop and the same objective can be pointed at a different dataset without rebuilding the machinery around the model, and one model can be trained on several tasks at once.^[1]

A systematic comparison

The paper is not a report of a single trick but a systematic comparison of pre-training objectives, architectures, unlabeled datasets and transfer methods, carried out across dozens of language-understanding tasks.^[1] It holds the text-to-text framework fixed and varies the ingredients around it, so the results say something about which choices in a transfer-learning pipeline actually matter. That is why the paper reads as much as an empirical map of the field as an announcement of a model.

The C4 corpus

Alongside the model, the work introduced the Colossal Clean Crawled Corpus (C4), a cleaned web-crawl dataset that Google describes as two orders of magnitude larger than Wikipedia. C4 is distributed through TensorFlow Datasets.^[2]

Model sizes

Five checkpoints were released: T5-Small with 60 million parameters, Base with 220 million, Large with 770 million, and models of 3 billion and 11 billion parameters.^[3]

Results

The 11-billion-parameter model reached state of the art on GLUE, SuperGLUE, SQuAD and CNN/Daily Mail, with a near-human score on SuperGLUE.^[2] The team also tested closed-book question answering, where the model answers from what it absorbed during training rather than from a passage supplied with the question. Measured by exact match, it scored 50.1% on TriviaQA, 37.4% on WebQuestions and 34.5% on Natural Questions.^[2]

Open release and later position

The code, the pre-trained models and the dataset were released openly, and Google announced the release in its research blog post of 24 February 2020.^[2][3] The repository is under the Apache-2.0 license, and since July 2022 its maintainers recommend the newer T5X implementation.^[3] The same blog post notes that multitask learning could be close to competitive with the pre-train-then-fine-tune approach.^[2]