XLNet: Generalized Autoregressive Pretraining for Language Understanding

On June 19, 2019, Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le submitted the XLNet paper to arXiv under the title XLNet: Generalized Autoregressive Pretraining for Language Understanding. The paper presents XLNet as a pretraining method for language understanding and positions it against masked-language-model pretraining, especially BERT.^[1]

The problem

Masked pretraining replaces some tokens with a mask and trains the model to recover them from the remaining text. This lets the model use context on both sides of a masked position, but the paper identifies two problems with BERT's mask-corrupted input: the masked positions are not modeled in relation to one another, and the masked input used during pretraining differs from the unmasked text used during fine-tuning, creating a pretrain-finetune discrepancy.^[1] The first issue means the model does not learn dependencies among the masked positions; the second means the training and downstream use do not match exactly. XLNet is designed to keep the benefit of bidirectional context while avoiding this corrupted-input formulation.^[1]

The method

XLNet is a generalized autoregressive pretraining method. In an autoregressive model, the likelihood of a sequence is factorized into a product of conditional probabilities, and each token is predicted from the tokens that precede it in a chosen order. XLNet extends this idea by maximizing the expected likelihood over all permutations of the factorization order. For a given sequence, the model does not commit to a single left-to-right order; instead, different permutations define different orders in which the same tokens are predicted. By training across these orders, a token can be conditioned on context that appears to its left and to its right in the original sequence, so the model learns bidirectional contexts without replacing tokens with a mask. The objective is still autoregressive, because each prediction is made conditionally on the tokens that come earlier in the selected order. The paper describes this as a generalized autoregressive pretraining method that learns bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order.^[1]

XLNet also integrates ideas from Transformer-XL into pretraining. The paper describes Transformer-XL as a state-of-the-art autoregressive model, and XLNet brings its ideas into the pretraining setup rather than treating autoregressive modeling and bidirectional context as mutually exclusive.^[1]

Experiments and results

The pretraining data included BooksCorpus and English Wikipedia, which together amounted to 13GB of plain text, plus Giga5 at 16GB of text.^[1] Under comparable experiment settings, XLNet outperformed BERT on 20 tasks, often by a large margin. The tasks included question answering, natural language inference, sentiment analysis, and document ranking.^[1] The paper presents these results as evidence that the generalized autoregressive objective is effective across several task types, not only as a pretraining objective in isolation.^[1]

Why it mattered

The paper's importance lies in the design choice it makes. Masked pretraining had shown that a model can learn from context in both directions, but the mask introduces a mismatch between pretraining and fine-tuning and does not model dependencies among masked positions. XLNet's permutation-based autoregressive objective offers a different route: it keeps the autoregressive factorization, yet by maximizing expected likelihood over many factorization orders it exposes each token to varied context, including context from both sides. The paper also shows that ideas from an autoregressive model such as Transformer-XL can be integrated into pretraining. The reported comparison with BERT across 20 tasks gives the method an empirical basis in the paper.^[1]