HellaSwag: Can a Machine Really Finish Your Sentence?

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi and Yejin Choi posted *HellaSwag: Can a Machine Really Finish Your Sentence?* to arXiv on 19 May 2019 ^[1].

A task that had stopped being hard

The task is easy to describe. A machine is given a short event description, such as "A woman sits at a piano", and must choose the most likely continuation from a list of candidates. This follows the commonsense natural language inference setup the same authors had used in 2018 ^[1]. The appeal of the format is that no specialist knowledge is needed to answer it: a person reads the options and picks the one that makes sense. The difficulty is that "makes sense" requires a model of how ordinary life usually goes — who does what, in what order, and which outcome would be odd.

By 2019 that difficulty had been partly measured and partly solved. With the introduction of BERT, near human-level performance had been reached on the earlier version of the task ^[1]. Once a benchmark approaches saturation — when systems score about as well as people — it stops separating good models from better ones. HellaSwag was designed in response, to show that commonsense inference still proves difficult for even state-of-the-art models ^[1].

Wrong answers written by machines, chosen by machines

The interesting engineering is in how the wrong answers were made. Rather than asking people to write plausible-sounding distractors, the authors generated them automatically and then filtered them with Adversarial Filtering (AF), a data collection paradigm in which a series of discriminators iteratively select an adversarial set of machine-generated wrong answers ^[1]. The loop can be pictured like this: a model is trained to tell the true ending from generated ones; the generated endings it classifies correctly are discarded, the ones it gets wrong are kept, and the next round generates new candidates aimed at that harder set.

The key insight is to scale up the length and complexity of the examples towards a "Goldilocks" zone in which the generated text is ridiculous to humans, yet often misclassified by state-of-the-art models ^[1]. Extremes are useless in either direction. An obviously absurd ending is rejected by any model and measures nothing; a sensible-sounding ending that a model also gets right adds no difficulty. Only the middle band, where the text has drifted far enough from reality to be funny to a reader but still looks locally fluent, exposes what the model does not actually know. The authors report that Adversarial Filtering proves to be surprisingly robust ^[1].

The results

What the paper reports is essentially a distance between two scores on the same set of questions. Humans answer more than 95% of them correctly, while state-of-the-art models at publication were below 48% ^[1]. The questions themselves are therefore trivial for people — the dataset is not testing obscure knowledge — and still hard for the systems, which means the difficulty lies in ordinary expectations about how events unfold rather than in vocabulary or syntax.

Why it matters

The paper makes two contributions that outlive its own numbers. The first is a way of building hard benchmarks on purpose: difficulty is manufactured by fitting the wrong answers to the weakness of current models, instead of hoping that human-written distractors happen to be hard. The second is evidence about what the models of the moment were missing. High scores on the earlier version of the task had suggested that machines handled commonsense inference; HellaSwag showed that the earlier result depended heavily on the task being easy, and that raising the bar to a zone of near-miss wrong answers reopened a wide gap ^[1].

The question in the title

At publication the question in the title has no answer: the machine is not finishing the sentence, it is choosing among candidates, and it chose wrongly more than half of the time. The distance between the human score above 95% and the model score below 48% is where the following work sits ^[1].