Google describes the Meena chatbot

A chatbot paper posted in January 2020

On 27 January 2020, a group of Google researchers posted a paper titled "Towards a Human-like Open-Domain Chatbot", describing a chatbot called Meena.^[1] A chatbot is a program that holds a conversation in text; "open-domain" means the conversation is not confined to a single task such as booking a flight, but can go wherever the user takes it. Meena is a 2.6-billion-parameter neural network trained on public-domain social media conversations.^[1] Parameters are the adjustable numbers a neural network learns during training; the count is a rough measure of how much the model can absorb from its training text.

A new way to score a conversation

Judging a chat is harder than judging a translation. A reply has to make sense in context, and it also has to say something — a safe, generic line like "I don't know" is sensible but useless. The paper introduced a metric built around those two demands, the Sensibleness and Specificity Average, or SSA.^[1] The full Meena scored 79% on it.^[1] The authors said that was 23% higher in absolute SSA than the existing chatbots they tested.^[1] Because the comparison was in absolute points rather than as a ratio, the gap is between Meena's score and the scores of the other systems on the same scale.

Perplexity as a predictor

The authors reported a strong correlation between perplexity and SSA.^[1] Perplexity is a standard measure of how well a language model predicts text: the lower it is, the less surprised the model is by the next word it encounters. A model with low perplexity on ordinary conversation is one that finds human turns of phrase predictable rather than strange. A strong correlation between the two means that a cheap, automatic number — perplexity, computed without asking any human to read a single reply — moved together with the expensive human judgement of whether replies were sensible and specific. That makes perplexity useful as a rough guide when comparing candidate models, while SSA remains the measure of what users actually get.

Revision and authorship

A revised version of the paper was posted on 27 February 2020.^[1] The authors include Quoc V. Le, Romal Thoppilan and Daniel Adiwardana.^[1]

Why it mattered

The paper's contribution is twofold. It put a large trained conversation model in front of readers with a stated size and training source, and it offered a way to put a number on chat quality that separates "does this reply make sense" from "does this reply say anything".^[1] Chatbots are easy to demo and hard to measure; a shared metric lets one system be compared with another on the same footing, and the reported 23-point absolute gap is exactly that kind of comparison.^[1] The link between perplexity and SSA is the other practical piece: it suggests that part of the work of evaluating a conversational model can be done automatically, and the rest still needs people to read the output and judge it.