Hallucination

What it is

Hallucination is false or misleading information that a model presents as fact^[2]. What makes it distinctive is not that the model is sometimes wrong, but that the error arrives in the same fluent, confident register as everything else it produces. A fabricated reference, an invented quotation, a misremembered number — none of these look any different from ordinary output. A reader with no independent way to check cannot tell them apart from the text alone, which is why the problem is treated as a property of the whole system rather than a defect in one answer.

Why it happens

One common cause is the training objective itself. Pretraining optimises next-token prediction, which means the model is always producing the most plausible continuation; when it has no information to base an answer on, it still makes a guess instead of declining^[2]. Fluency is exactly what that objective rewards, so an unsupported answer can also be a perfectly well-formed one.

Other cited causes include incomplete or unrepresentative training data, a tendency to agree with the user, and training and evaluation procedures that reward guessing over admitting uncertainty^[2]. Anthropic has reported that hallucinations can also arise when internal circuits that should inhibit answering are incorrectly suppressed^[2] — the machinery that ought to stop the model from speaking is itself failing.

A research agenda

In February 2022, Ji and co-authors published a survey of hallucination in natural language generation, noting that generation based on deep learning is prone to producing unintended text that degrades system performance and fails user expectations^[1]. The survey set out the vocabulary and the agenda: metrics for measuring hallucination, methods for mitigating it, and directions for future work^[1]. It then reviewed the problem task by task, covering abstractive summarisation, dialogue generation, generative question answering, data-to-text generation, machine translation and visual-language generation^[1]. A later revision of the survey added a section specifically on hallucinations in large language models^[1] — a sign of how quickly the centre of gravity moved.

What it looks like in practice

The clearest public evidence comes from references. In one widely cited measurement, 69 of 178 references generated by GPT-3 had incorrect or non-existent DOIs, and another 28 could not be retrieved at all^[2]. A separate analysis of 115 citations found that 47% were entirely fabricated, 46% were real but described incorrectly, and only 7% were accurate^[2].

Two public cases are often mentioned. Meta's Galactica demo cited non-existent papers and was withdrawn two days after launch^[2]. In the Mata v. Avianca case, a lawyer submitted six fabricated judgments and was fined US$5,000^[2]. In both, the output was plausible, correctly formatted and false.

How to read the numbers

The figures above are not a general error rate for any system. Each comes from a particular task, a particular model and a particular way of counting, and they measure bibliographic references rather than prose — one of the easiest things to check, and therefore one of the places where a hallucination surfaces most visibly. The practical definition stays the one the field uses: information presented as fact that is false or misleading^[2]. That is also why hallucination is hard to design away with a single fix: the same properties that make generated text useful — fluency, coverage, willingness to answer — are the ones that produce it.