Reflexion: Self-Reflective Prompting for Language Agents

Who this is for and what you will get

This is for readers who have used ReAct or tool calling with a model, but find that their agent keeps repeating the same mistakes. After reading it, you can explain how Reflexion’s three roles work together, and you can bring the loop of acting, evaluating, reflecting and acting again into your own task.

What Reflexion is

Reflexion is a framework proposed by Shinn et al. in 2023 that strengthens language-based agents through linguistic feedback.^[1][2] Its core method is to convert feedback from the environment, whether free-form language or a scalar score, into a piece of linguistic feedback called self-reflection. That reflection is added to the context of the next trial, so the agent can learn from earlier mistakes.^[1]

The authors describe this as a new paradigm of verbal reinforcement.^[1][2] The advantage is that the agent can adjust quickly from concrete failures without retraining the model each time.^[1]

How the three roles work together

Reflexion is made of three models.

• The Actor generates text and actions from state observations, acts in the environment and receives observations, which forms a trajectory. Chain-of-thought (CoT) and ReAct can both serve as the Actor, and a memory component is added to give the agent extra context.^[1]
• The Evaluator scores the Actor’s output. It takes the generated trajectory, also called short-term memory, as input and outputs a reward score. Decision-making tasks use both LLM-based evaluation and rule-based heuristic rewards.^[1]
• The Self-Reflection model is also played by an LLM. It uses the reward signal, the current trajectory and its persistent memory to generate specific, relevant feedback, which is stored in the memory component. The agent then uses these experiences, kept in long-term memory, to improve its decisions quickly.^[1]

The full process comes down to five steps: define the task, generate a trajectory, evaluate, perform self-reflection, and generate the next trajectory.^[1] Reflexion extends the ReAct framework by adding self-evaluation, self-reflection and memory components.^[1]

What the results show

The experiments cover three kinds of task: sequential decision-making on AlfWorld, reasoning on HotpotQA, and programming on benchmarks such as HumanEval.^[1]

On AlfWorld, ReAct + Reflexion used heuristic and GPT-based self-evaluation for binary classification. It completed 130 of 134 tasks and clearly outperformed ReAct.^[1] HotpotQA is a question-answering dataset that requires reasoning over several documents. There, Reflexion significantly outperformed all baselines within only a few learning steps. When episodic memory made of recent trajectories was added, Reflexion + CoT outperformed both CoT alone and CoT with episodic memory.^[1]

For programming, Reflexion generally outperformed previous state-of-the-art methods when writing Python and Rust code on MBPP, HumanEval and Leetcode Hard.^[1]

Why it avoids fine-tuning

Traditional reinforcement learning (RL) usually needs large amounts of training data and costly model fine-tuning. Reflexion offers a lightweight alternative that does not fine-tune the underlying language model, so it is more economical in data and compute.^[1]

Its feedback is linguistic, which is more nuanced and specific than the scalar rewards used in traditional RL. The agent can therefore see more clearly where it went wrong, and its next attempt can be better targeted.^[1] Reflections are kept in the memory component, which also makes the learning process easier to analyze and understand than in traditional RL.^[1]

What it suits and where it falls short

Reflexion suits tasks that need learning from repeated trial and error, such as decision-making, reasoning and programming.^[1]

It also has clear limits:

• It depends on self-evaluation. The agent must judge its own performance accurately and produce useful reflections, which can be hard on complex tasks.^[1]
• Its long-term memory is limited. It uses a sliding window of maximum capacity; for more complex tasks, structures such as vector embeddings or SQL databases may work better.^[1]
• Test-based code generation has limits. Test-driven development struggles to specify an exact input-output mapping, for example with non-deterministic generator functions or with function outputs that depend on hardware.^[1]

Next steps

For the method details and the full experiments, read the paper itself.^[2]

This article is adapted from the Reflexion page in DAIR.AI’s *Prompt Engineering Guide*^[1], released under the MIT License.