Machine reading comprehension surpasses human baseline on SQuAD

Background

Machine reading comprehension is the task of handing a system a document together with questions about it, and asking the system to produce the answers itself. That is a different job from keyword search: a search engine returns pages that may contain the answer, while a reading comprehension system is expected to return the answer. SQuAD is a benchmark and public leaderboard built around this task. Systems on the leaderboard are scored on exact match, where the submitted answer must match the reference answer exactly, and on F1, which gives partial credit when an answer overlaps with the reference ^[2]. The leaderboard also lists a human score on the same questions: 82.304 exact match ^[1].

What happened

On January 3, 2018, Microsoft Research Asia's R-NET model submitted a score of 82.650 on the exact match portion of the SQuAD leaderboard, above the human performance of 82.304 on the same questions ^[1]. On January 5, 2018, researchers at Alibaba submitted 82.440 exact match, also about the same as a human, leaving the two companies tied for first place on the leaderboard ^[1].

Microsoft announced the human-parity milestone in a blog post describing technology that can read a document and answer questions about it about as well as a person ^[1].

The caveat

Ming Zhou, assistant managing director of Microsoft Research Asia, noted that overall people are still much better than machines at comprehending the complexity and nuance of language ^[1]. The qualification is worth keeping in view. The human figure on the leaderboard is one score on one set of questions, not a general measure of language ability. Clearing it says something specific: that on this set of questions a system can produce answers matching the reference answers about as often as a person can. It says much less about reasoning, background knowledge, or knowing when a question has no good answer.

Already in a product

The result was not purely a research exercise. Microsoft was already applying earlier versions of the models submitted for the SQuAD dataset leaderboard in its Bing search engine at the time of the announcement ^[1].

What came after

Microsoft Research later documented the sequence of results. R-net was the first system to exceed human parity on SQuAD in exact match. In September 2018, Microsoft Research Asia's nLnet became the first to exceed both exact-match and F1 human parity. Google's BERT then became the leader ^[2].

The ordering is a reminder of how quickly a public leaderboard moves once a target has been shown to be reachable: the milestone held for a matter of months before a stronger system took the top position.

Why it matters

Microsoft framed the result as a milestone in the push for more natural interaction with search engines and intelligent assistants ^[1]. The underlying capability — reading a passage and answering a question about it — is what separates a system that returns links from one a person can ask a question directly. In that sense a benchmark score is a proxy for something users experience: whether a question can be put in ordinary language and answered rather than deflected.

The episode also illustrates what benchmarks are good for, and where they stop. A single number gives teams a shared, comparable target and makes progress visible across organisations working in parallel; it does not, on its own, establish that a machine understands language the way a person does — a point Microsoft's own researcher made on the day of the announcement ^[1].