SQuAD: 100,000+ Questions for Machine Comprehension of Text

The problem

Reading comprehension, as a task for machines, means showing a system a passage of text and a question about it, and asking the system to produce the answer. To train and compare such systems, research needs data: many passages, many questions, and known correct answers to measure against. The SQuAD paper, by Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev and Percy Liang, was submitted to arXiv on June 16, 2016, and its central contribution is a dataset built for exactly this purpose^[1].

How the dataset was built

SQuAD consists of more than 100,000 questions posed by crowdworkers on a set of Wikipedia articles^[1]. Crowdworkers are people recruited online to perform small pieces of work; here they wrote questions whose answers they could point to in the article they were reading. The design choice that shapes everything else is this: the answer to each question is a segment of text taken from the corresponding reading passage^[1]. A system is therefore not asked to write an answer in its own words, but to find and return the piece of the passage that answers the question. That makes the task easy to score automatically, because a system's output can be compared directly with the reference segment, and it also keeps the questions tied to what the passage actually says.

What kind of reasoning the questions require

A dataset is only useful if its questions are understood. The authors analyzed their data to characterise the types of reasoning needed to answer the questions, and in doing so they leaned heavily on dependency and constituency parse trees^[1]. Parse trees describe the grammatical structure of a sentence: constituency trees group words into nested phrases, while dependency trees link each word to the word it depends on. Using them, the authors could describe questions in terms of the syntactic relations involved rather than as an undifferentiated pile of sentences.

Results

To show that the dataset is learnable but not trivial, the authors built a strong logistic regression model, a standard statistical classifier that weights features of the input and combines them into a prediction. That model reaches an F1 score of 51.0%, a significant improvement over a simple baseline of 20%^[1]. F1 combines precision and recall into a single number, so it rewards a system both for returning correct answer spans and for not missing them.

The more telling number is the human one. Human performance on the dataset is 86.8% F1, far above the best model at the time^[1]. The gap between the two is what makes the dataset interesting: a model that scores around half of the available credit is clearly doing something, yet there is a large stretch of behaviour between it and a person reading the same passage.

Why it matters

The authors read the size of that gap as evidence that SQuAD presents a good challenge problem for future research^[1]. A benchmark is most useful when it is hard enough that progress on it is meaningful, and when it is measurable enough that different systems can be compared on the same footing. The span-based answer format supplies the second property; the distance to human performance supplies the first.

The dataset is also freely available^[1], so groups can work on the same questions rather than on private collections, and results can be checked against a shared reference. The paper's framing — a large set of crowdwritten questions over encyclopedic text, answers fixed as spans, human accuracy reported alongside a machine baseline — is what made SQuAD a reference point for work on machine reading.