Retrieval-augmented generation

What it is

Retrieval-augmented generation (RAG) is a method that lets a large language model retrieve information from external sources and integrate it before answering^[1]. Rather than relying only on what was baked into its parameters during training, the model is handed relevant material at the moment a question is asked, and it writes its answer with that material in front of it.

How it works

The method combines a parametric language model with a non-parametric external memory that is accessed by retrieval at inference time^[1]. "Parametric" means the knowledge lives inside the model's weights; "non-parametric" means it sits outside, in a store the model can look things up in.

In practice, data is converted into embeddings and stored in a vector database^[1]. An embedding is a list of numbers that represents a piece of text, arranged so that texts with similar meaning end up close together; the database can then be searched by similarity rather than by exact keywords. When a user asks a question, a document retriever selects the most relevant documents, which are inserted into the prompt alongside the query^[1]. The model's job is then to read the question together with those passages and produce an answer.

A simple illustration: suppose a company's support team has loaded its product manuals into such a store. A customer asks how to reset a device. The retriever compares the question against the stored passages and pulls out the few that discuss resetting, and those passages travel with the question into the model's context. The model answers from what it was given rather than from memory.

Where it came from

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni and others proposed the approach in 2020 in "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" at NeurIPS 2020^[1]. The paper, first submitted in May 2020, combines pre-trained parametric memory with non-parametric memory for language generation^[2]. In RAG models the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia accessed with a pre-trained neural retriever^[2].

The authors motivated the design by noting that pre-trained models' ability to access and precisely manipulate knowledge is limited, and that providing provenance for decisions and updating world knowledge remain open problems^[2]. A model trained once has no easy way to show where an answer came from, and no way to absorb new facts without further training; putting the knowledge in an index that is read at answer time addresses both.

What it is used for

It can supplement domain knowledge or information beyond the training data and reduce hallucination^[1]. Unlike fine-tuning, new information can be added to the external knowledge base without retraining the model, saving compute and cost^[1]: updating a document in the store is enough to change what the system knows. Answers can cite their sources so that they can be checked^[1], which matters wherever a reader needs to verify a claim rather than take it on trust.

Limits

RAG does not eliminate hallucination, since the model can still misread the retrieved context^[1]. Retrieval quality depends on the embedding model, chunking and hybrid search, and systems are exposed to RAG poisoning through correct but misleading sources or mixed old and new information^[1]. Models also struggle to judge when their own knowledge is insufficient^[1] — a system that always retrieves something will always have something to say, even when the retrieved passages do not actually answer the question.