Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (RAG)

The problem it takes on

Large pre-trained language models store what they learn in their parameters, and that is the only place the knowledge lives. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" — RAG for short — was submitted to arXiv on May 22, 2020 by Patrick Lewis, Ethan Perez, Aleksandra Piktus and nine other authors, and it starts from the limits of this parametric memory^[1]. The authors name three. A model has only limited ability to access and precisely manipulate the knowledge inside it. On knowledge-intensive tasks, where answering well depends on recalling specific facts, such models lag behind architectures built for the task. And two questions stay open: where a stated fact came from, and how to update a model's knowledge of the world after training^[1].

Two kinds of memory

The proposal is to pair parametric memory with non-parametric memory. The parametric half is a sequence-to-sequence model — an encoder that reads the input and a decoder that writes the output — whose weights carry what pre-training put there. The non-parametric half sits outside the weights, as a dense vector index of Wikipedia, and a pre-trained neural retriever fetches from it^[1].

A dense retriever works on vectors rather than on keywords. The question is mapped to a vector, and the index returns the passages whose vectors lie closest to it; those passages are passed to the generator together with the question, and the generator writes its answer conditioned on both. Because the index is separate from the weights, the knowledge the model can reach is not fixed at training time but sits in an artifact of its own.

In the paper the retriever is a Dense Passage Retriever and the generator is BART, and the two are combined end-to-end in a single probabilistic model^[1]. The retrieved documents are treated as latent: the model does not commit to one document in advance but sums over the possibilities, approximated by keeping only the top K candidates^[1].

One retrieval or many

Retrieval can be done once for an answer or repeated while the answer is written. The authors compare two formulations: one conditions on the same retrieved passages across the whole generated sequence, the other can use different passages for each token^[1]. The first gives the whole answer a single fixed evidence set; the second lets the evidence change from word to word, so a sentence can draw on more than one passage.

Results

The paper reports state of the art on three open-domain question answering tasks. On generation tasks, RAG produced language that was more specific, more diverse and more factual than a state-of-the-art seq2seq baseline that relied on parametric memory alone^[1].

Why it matters

Two things in the design are worth noticing. First, moving part of the knowledge out of the weights and into an index changes what "knowing" means for the system: the model's factual reach is set by what the retriever can find and feed in, not only by what pre-training baked into the parameters. Second, because retrieval happens while the answer is produced, the output can be connected back to passages that were actually fetched — which is exactly the provenance question the paper raises about parametric-only models^[1]. The comparison between the two formulations also shows that this is a design choice rather than a detail: how long a piece of evidence stays in play during generation is something the model can be built to decide.