Metacognition in AI agents

Who this is for and what you will get

This is for developers who want an agent to change its approach after a mistake. After reading, you should be able to write a small recommendation agent that reviews its own choices and switches strategy based on feedback, and you should know which parts of the example code are not ready for production.^[1]

What metacognition means

Metacognition is thinking about one's own thinking. For an AI agent, it means evaluating and adjusting its actions based on self-awareness and past experience. This helps an agent with transparency (explaining its reasoning and decisions), reasoning (combining information into sound judgments), adaptation (adjusting to new environments and changing conditions), and perception (reading data from its environment more accurately).^[1]

In practice, the agent gains four things: self-reflection (assessing its own performance and finding what to improve), adaptability (changing strategy based on past experience and changing conditions), error correction (detecting and fixing mistakes on its own), and resource management (using planning and evaluation to save time and compute).^[1]

What an agent is made of

An AI agent typically has three parts. The persona is its personality and how it interacts with users. Tools are the functions it can perform. Skills are the knowledge and expertise it has. Together they form an "expertise unit" that can carry out specific tasks.^[1]

A travel agent that reflects

Take a travel agent that helps a user plan a trip to Paris. It works through four steps: gather the user's preferences (dates, budget, interests, and special needs), retrieve flight, hotel, and attraction options, generate a personalized itinerary, and adjust it based on feedback.^[1]

Reflection happens after feedback arrives. The agent reviews the feedback to see which recommendations were well received and which were not, and it adjusts future suggestions. If a user dislikes crowded places, the agent stops recommending popular spots at peak hours. If it once suggested a hotel that was already fully booked, it checks availability more carefully from then on.^[1]

Build a hotel agent that reflects

The example below uses only the standard library and does not call a model. It has two strategies: cheapest and highest_quality.

1. Define the agent. It keeps two records, previous choices and corrected choices, and it offers the two strategies. The recommend method picks a hotel according to the strategy and records the result in previous_choices.^[1]

class HotelRecommendationAgent:
    def __init__(self):
        self.previous_choices = []  # Stores the hotels chosen previously
        self.corrected_choices = []  # Stores the corrected choices
        self.recommendation_strategies = ['cheapest', 'highest_quality']  # Available strategies

    def recommend_hotel(self, hotels, strategy):
        """
        Recommend a hotel based on the chosen strategy.
        The strategy can either be 'cheapest' or 'highest_quality'.
        """
        if strategy == 'cheapest':
            recommended = min(hotels, key=lambda x: x['price'])
        elif strategy == 'highest_quality':
            recommended = max(hotels, key=lambda x: x['quality'])
        else:
            recommended = None
        self.previous_choices.append((strategy, recommended))
        return recommended

1. Add the reflection method. It reads the feedback on the last choice. If the feedback is bad, it switches between the two strategies and records the corrected choice. Otherwise it reports that the choice was good.^[1]

    def reflect_on_choice(self):
        """
        Reflect on the last choice made and decide if the agent should adjust its strategy.
        The agent considers if the previous choice led to a poor outcome.
        """
        if not self.previous_choices:
            return "No choices made yet."

        last_choice_strategy, last_choice = self.previous_choices[-1]
        # Let's assume we have some user feedback that tells us whether the last choice was good or not
        user_feedback = self.get_user_feedback(last_choice)

        if user_feedback == "bad":
            # Adjust strategy if the previous choice was unsatisfactory
            new_strategy = 'highest_quality' if last_choice_strategy == 'cheapest' else 'cheapest'
            self.corrected_choices.append((new_strategy, last_choice))
            return f"Reflecting on choice. Adjusting strategy to {new_strategy}."
        else:
            return "The choice was good. No need to adjust."

1. Replace real users with simulated feedback. The rule is simple: a hotel priced under 100 or rated below 7 gets bad.^[1]

    def get_user_feedback(self, hotel):
        """
        Simulate user feedback based on hotel attributes.
        For simplicity, assume if the hotel is too cheap, the feedback is "bad".
        If the hotel has quality less than 7, feedback is "bad".
        """
        if hotel['price'] < 100 or hotel['quality'] < 7:
            return "bad"
        return "good"

1. Prepare the data and run the sequence: recommend with cheapest, reflect, then recommend again with the adjusted strategy.^[1]

# Simulate a list of hotels (price and quality)
hotels = [
    {'name': 'Budget Inn', 'price': 80, 'quality': 6},
    {'name': 'Comfort Suites', 'price': 120, 'quality': 8},
    {'name': 'Luxury Stay', 'price': 200, 'quality': 9}
]

# Create an agent
agent = HotelRecommendationAgent()

# Step 1: The agent recommends a hotel using the "cheapest" strategy
recommended_hotel = agent.recommend_hotel(hotels, 'cheapest')
print(f"Recommended hotel (cheapest): {recommended_hotel['name']}")

# Step 2: The agent reflects on the choice and adjusts strategy if necessary
reflection_result = agent.reflect_on_choice()
print(reflection_result)

# Step 3: The agent recommends again, this time using the adjusted strategy
adjusted_recommendation = agent.recommend_hotel(hotels, 'highest_quality')
print(f"Adjusted hotel recommendation (highest_quality): {adjusted_recommendation['name']}")

Working through the code: in step one the cheapest option is Budget Inn (80, quality 6). The simulated feedback marks it bad, so the strategy switches to highest_quality. In step three the agent recommends Luxury Stay (200, quality 9), the highest quality on the list. This is the simplest form of a metacognitive loop: evaluate the last choice, then change strategy based on that evaluation.^[1]

How this relates to retrieval-augmented generation

Retrieval-augmented generation (RAG) can be a prompting technique or a tool. As a prompting technique, the developer writes specific queries for each retrieval. This gives more control but takes effort to tune. As a tool, RAG is built into the agent's architecture and automates retrieval and generation. That is easier to integrate, and more efficient at scale.^[1]

Corrective RAG has three parts: a prompting technique that guides retrieval, a tool that evaluates how relevant the retrieved information is and produces accurate answers, and an evaluation step that keeps checking performance and adjusting it.^[1]

Pre-emptive context load is a different approach. Before the agent handles a query, relevant background information is loaded into the model, so the model has it from the start and does not need to retrieve it mid-process.^[1]

Relevance and search intent

For evaluating relevance, four key concepts matter: context awareness, accuracy, user intent, and a feedback loop.^[1]

A simple way to start is to score each candidate. This function adds one point each time the category matches the user's interests, the price is within budget, and the location matches the destination:^[1]

def relevance_score(item, query):
    score = 0
    if item['category'] in query['interests']:
        score += 1
    if item['price'] <= query['budget']:
        score += 1
    if item['location'] == query['destination']:
        score += 1
    return score

After scoring, sort from highest to lowest and keep the top ten:^[1]

def filter_and_rank(items, query):
    ranked_items = sorted(items, key=lambda item: relevance_score(item, query), reverse=True)
    return ranked_items[:10]  # Return top 10 relevant items

User intent falls into three types: informational ("What are the best museums in Paris?"), navigational ("Louvre Museum official website"), and transactional ("Book a flight to Paris"). The example guesses intent from keywords:^[1]

def identify_intent(query):
    if "book" in query or "purchase" in query:
        return "transactional"
    elif "website" in query or "official" in query:
        return "navigational"
    else:
        return "informational"

Keyword matching is a starting point. Real systems usually combine context, user history, and natural language processing to understand intent.^[1]

Risks in code generation and SQL

A code generating agent has a model write code and then runs it. The example runs the generated code with exec and shows no isolation. That is fine for a local demo. Before connecting anything real, run such code in a restricted environment, such as a container, and limit what it can access.^[1]

def execute_code(code):
    # Execute the generated code using exec
    exec(code)
    result = locals()
    return result

The SQL helper in the example inserts preference values into the query with f-strings and does not use query parameters. If a preference contains hostile text, it can change what the SQL does. Avoid this pattern in production code.^[1]

def generate_sql_query(table, preferences):
    query = f"SELECT * FROM {table} WHERE "
    conditions = []
    for key, value in preferences.items():
        conditions.append(f"{key}='{value}'")
    query += " AND ".join(conditions)
    return query

Limits and common questions

• The feedback in the example is simulated with fixed rules. A real system needs real user feedback, and that feedback can be noisy or biased.^[1]
• The example switches only between two fixed strategies and does not learn new ones. Deeper self-improvement needs more strategies and better evaluation criteria.^[1]
• Reflection happens after each recommendation. To catch problems earlier, add checkpoints partway through the process.

Where to go next

The examples are simplified teaching code, with no error handling and no real hotel or flight APIs. For what comes after launch, such as evaluation and monitoring, AI Agents in Production covers it.^[1]

Adapted from lesson 9 of Microsoft's open-source course *AI Agents for Beginners*^[1], released under the MIT License.