LaMDA

What it is

LaMDA stands for Language Model for Dialogue Applications: a language model built and tuned for conversation. Google announced it during the Google I/O keynote on 18 May 2021^[1]. Like other large language models, its basic job is to predict what text should come next; what makes it a dialogue model is that the text it is given is a back-and-forth conversation, and the text it produces is the next reply.

Where it came from

LaMDA grew out of Meena, a chatbot developed by the Google Brain team that was later renamed LaMDA. Daniel De Freitas and Noam Shazeer were its lead researchers^[1]. The name change marks a shift in emphasis: from a single chatbot toward a general language model that dialogue applications could be built on.

Architecture and training

The technical account came in the paper "LaMDA: Language Models for Dialog Applications", posted to arXiv on 20 January 2022. It described Transformer-based models of up to 137 billion parameters, pre-trained on 1.56 trillion words of public dialogue data and web text^[2].

Three terms in that sentence carry most of the weight. A Transformer is a neural network architecture that reads text as a sequence and uses attention to weigh how strongly each part of the input relates to every other part. A parameter is one of the numbers the network adjusts while learning; a model with 137 billion of them is large even by the standards of its time. Pre-training means the model first learns from a very broad body of text by repeatedly predicting what comes next, before any attempt to specialize it for a task. Because the training mix included public dialogue as well as web text, the model saw both ordinary prose and the shape of conversational exchanges.

Quality, safety and groundedness

The paper did not stop at fluency. It measured three things: quality, safety, and factual groundedness^[2]. Safety was scored with a metric based on a set of human values, so that the model's answers could be judged not only on whether they were sensible but on whether they were acceptable. Factual groundedness addressed a different problem: a language model can produce a fluent sentence that is simply wrong. The paper quantifies factuality with a groundedness metric, which enables the model to generate responses grounded in known sources^[2].

One way it did so was by reaching outside itself. The model could consult an information-retrieval system, a language translator and a calculator^[2]. In other words, instead of relying only on what it had absorbed in training, it could call tools and build an answer from what they returned.

The paper also drew a cautionary conclusion from its own experiments: scaling the model alone improves quality, but shows less improvement on safety and factual grounding^[2]. Making a model bigger makes it a better talker faster than it makes it a safer or more accurate one. That finding is why the work treats training-time scale and post-hoc safeguards as separate problems.

Filtering replies for safety

One safeguard the paper described works at the level of individual answers. The model generates candidate responses, and a LaMDA classifier fine-tuned on a small amount of crowdworker-annotated data filters them, which was presented as a way to improve safety^[2]. The classifier is a second model trained on human judgements about which replies are acceptable; the small annotated set is enough because the classifier only has to rank and reject, not to write. Responses that score poorly are dropped before the user ever sees them.

LaMDA 2 and public attention

LaMDA 2 was unveiled on 11 May 2022, together with the AI Test Kitchen app^[1], a way to let outside people try the model and see its behaviour for themselves.

Public attention soon arrived from an unexpected direction. In June 2022 a Google engineer, Blake Lemoine, publicly claimed LaMDA was sentient, and Google fired him on 22 July 2022^[1]. The episode became a widely cited example of how convincing conversational output can be mistaken for an inner life; the technical claims in the paper concern quality, safety and groundedness, not consciousness.

Where it sits

A lightweight version of LaMDA, which Google said required significantly less computing power, powered Bard when it was announced on 6 February 2023^[3]. That is the model family's clearest link to the products that followed: the dialogue research described in the paper became the starting point for a publicly released assistant.