GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

What GLUE is

GLUE is a benchmark and analysis platform built to evaluate and analyze how well models perform across a diverse range of existing natural language understanding (NLU) tasks^[1]. The tasks it gathers include question answering, sentiment analysis and textual entailment, and the benchmark comes with an online platform where models can be evaluated, compared and analyzed^[2]. The paper describing it was written by Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy and Samuel R. Bowman, and was submitted to arXiv on April 20, 2018^[1].

How it is put together

Two design choices shape the benchmark. First, it is model-agnostic: it does not prescribe an architecture or a training procedure, so any system that can be run on the included tasks can be scored in the same way^[1]. Second, it deliberately makes sharing knowledge across tasks worth doing, because some of the tasks come with very limited training data^[1]. A model that can carry over what it learned on a data-rich task has an advantage on a data-poor one, so the benchmark rewards a single system that handles many tasks rather than a collection of unrelated specialists.

Evaluation is not reduced to one aggregate score. The benchmark is accompanied by a hand-crafted diagnostic test suite that supports detailed linguistic analysis of NLU models^[1]. A score on the main tasks says how often a model is right; the diagnostic suite is there to say something about which linguistic distinctions the model handles and which it does not.

What the baselines showed

The authors did not only propose the benchmark; they also ran baselines on it, based on multi-task learning and transfer learning. These approaches did not immediately give substantial improvements over the straightforward alternative of training a separate model for each task^[1]. That is a useful negative result to report alongside a new benchmark: pooling several tasks into one training setup is not by itself a recipe for better scores, and the expected benefit of shared representations has to be demonstrated rather than assumed.

Why it matters

When models are measured on different tasks with different data, formats and scoring, results are hard to line up against each other. A shared collection of tasks plus an online platform for evaluation, comparison and analysis gives researchers a common ground on which results can be placed side by side^[2]. Because the benchmark is model-agnostic, the comparison is not tied to any one family of methods^[1].

Just as important is the split between scoring and analysis. The main tasks measure performance; the hand-crafted diagnostic suite measures behaviour, allowing a detailed linguistic reading of what a model has and has not learned^[1]. And by including tasks with very limited training data, the benchmark makes the question of knowledge sharing across tasks part of the evaluation rather than an afterthought^[1].

Read as a paper, GLUE's contribution is infrastructure: a fixed set of tasks, a public platform for comparing systems on them, and a separate instrument for looking inside the resulting scores.