Meena

What Meena is

Meena is an open-domain chatbot from Google, described in the paper "Towards a Human-like Open-Domain Chatbot", posted to arXiv on 27 January 2020 by Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel and six co-authors, with Quoc V. Le among the eleven authors.^[1] It is a 2.6-billion-parameter neural conversational model trained end-to-end, and the paper describes it as a multi-turn open-domain chatbot.^[1]

The three adjectives carry most of the meaning. "Open-domain" means the conversation is not confined to a fixed task such as booking a flight; the user can raise almost any topic. "Multi-turn" means quality is judged over a back-and-forth exchange rather than a single reply. "End-to-end" means one network maps the conversation so far directly to the next reply, instead of a pipeline of hand-built components stitched together.

Where it came from

Meena was developed by the Google Brain team, and it was later renamed LaMDA.^[2]

How it works

The training signal is deliberately plain: the model is trained simply to minimise the perplexity of the next token.^[1] Perplexity is a standard measure of how surprised a language model is by the next piece of text — roughly, how many plausible continuations it is spread across. A low perplexity means the model puts high probability on what actually comes next. Training on that objective turns conversation into next-token prediction over dialogue text, with no separate rewards or hand-written dialogue rules described in the paper.

The training data was mined and filtered from public-domain social media conversations.^[1] Filtering mattered enough that it appears as a separate ingredient in the reported results: the strongest configuration combined filtered training data with tuned decoding at generation time, on top of the same underlying model.

Measuring the conversation: SSA

A chatbot can be fluent and still dull or incoherent, so the authors proposed a human-evaluation metric, the Sensibleness and Specificity Average (SSA), intended to capture key elements of human-like multi-turn conversation.^[1] The name points at the two things it scores: whether a reply is sensible in context, and whether it is specific rather than a generic response that could follow almost anything.

Results

On that metric the best purely end-to-end Meena scored 72% SSA. The full version, with filtering and tuned decoding, scored 79%, which the authors said was 23% higher in absolute SSA than the existing chatbots they evaluated.^[1]

The more consequential finding was the link between the training objective and human judgement. The authors reported a strong correlation between perplexity and SSA, and suggested that a human-level SSA of 86% was potentially within reach;^[1] they stated that such a level is potentially within reach if perplexity can be better optimised.^[1] If that holds, evaluating a chat model becomes partly a matter of measuring how well it predicts text, which is far cheaper than running human raters on every iteration.

Why it mattered

The paper makes a narrow, testable argument: a single number, next-token perplexity, tracks how human-like a model's conversation is judged to be, so pushing perplexity down is a plausible route to better conversation. It also handed the field a human-evaluation protocol, SSA, for a task where automatic scores are hard to trust, and reported scores against it for competing systems. Meena itself did not keep its name — the line of work continued as LaMDA.^[2]