ReAct: Synergizing Reasoning and Acting in Language Models

The problem

A language model can be prompted to write out intermediate reasoning steps before giving an answer. This kind of step-by-step reasoning happens entirely inside the model: it cannot look anything up, and a mistake made early in the chain tends to be carried forward, so the model can end up confident and wrong. A model can also be prompted to take actions, but acting on its own leaves no room for planning. The ReAct paper, submitted to arXiv on October 6, 2022 by Shunyu Yao, Jeffrey Zhao, Dian Yu and four other authors, asks whether the two abilities can be combined in one model. ^[1]

How it works

ReAct, short for reasoning plus acting, has the model generate reasoning traces and task-specific actions in an interleaved manner. ^[1] A reasoning trace is text in which the model states what it is trying to find out and how it intends to proceed; it helps the model form an action plan, keep track of that plan and update it, and handle exceptions when something unexpected comes back. An action is a call to something outside the model, such as a knowledge base or an environment, which returns a result the model can then reason over. ^[1]

The loop is therefore thought, action, observation, thought. A concrete shape: the model writes down that it needs a particular fact, issues a search action, reads the passage that comes back, notices that the passage settles only part of the question, and issues a second, narrower action. Reasoning decides what to do next; the action brings in information the model did not have; the next reasoning step uses it.

Because the reasoning is written out and each action and its result is visible, the whole trajectory can be read afterwards. That is a different situation from a model that only produces a final answer, or only a chain of internal reasoning.

Results

The experiments covered two kinds of task. On question answering with HotpotQA and fact verification with Fever, ReAct interacted with a simple Wikipedia API. Interleaving reasoning with lookups overcame the hallucination and error propagation that are common in chain-of-thought reasoning, and produced task-solving trajectories that were more interpretable and more human-like. ^[1]

On two interactive decision-making benchmarks, ALFWorld and WebShop, ReAct outperformed imitation learning and reinforcement learning methods by absolute success rates of 34% and 10% respectively, while being prompted with only one or two in-context examples. ^[1]

The experiments used a frozen PaLM-540B prompted with few-shot in-context examples, with GPT-3 results reported in an appendix. ^[1] The model's weights were not changed for the task; the behaviour comes from the prompt and the loop.

Why it matters

The paper's case rests on two points. The first is that reasoning and acting do different jobs. A reasoning trace helps the model induce, track and update a plan and deal with exceptions, while an action lets it interface with external sources such as knowledge bases or environments. ^[1] Reasoning alone stays inside the model; action alone has no plan to follow.

The second is legibility. The paper argues that the approach shows improved human interpretability and trustworthiness over methods that have no reasoning or acting components. ^[1] A reader can see what the model set out to do, what it actually did, and what came back, and can locate the step where things went wrong rather than judging only the final answer.

It is also worth being precise about what kind of method this is. It is a pattern for prompting an existing model in an interleaved way, not a new architecture and not a training procedure, and the reported results come from prompting a frozen model with a handful of examples in the prompt. ^[1] The reach of such a method depends on which actions are made available to the model and on how far the loop is allowed to run, and the quality of a trajectory is limited by the quality of what each action returns.