Building trustworthy agents

Who this is for and what you will get

This is for developers who already call a large model from code and plan to hand an agent to other people. After reading, you should be able to write a system prompt in stages, name the main threats to an agent and a defense for each, and add a human approval step before sensitive actions.

Why agents need tighter constraints

A plain LLM application mostly answers questions. An agent also calls tools, reaches into systems, and carries out actions for the user, so a mistake costs more. The system message holds the meta rules for how the model deals with users and data. An agent has to finish specific tasks, so it needs more specific instructions than a chat app does, and that makes the system prompt more important.^[1]

What you need

• A Python environment.
• A callable model service. The sample uses FoundryChatClient from the Microsoft Agent Framework. It reads the project endpoint and model deployment name from the environment variables AZURE_AI_PROJECT_ENDPOINT and AZURE_AI_MODEL_DEPLOYMENT_NAME, and it authenticates with AzureCliCredential.^[1]
• If you do not use this Azure setup, you can still apply the prompt-building method and replace the model call with your own service.

Step by step: build a system prompt from a meta prompt

1. Write a meta system message. It is a template for the model that tells it how to write system prompts for the agents you create. The example is below.^[1]

You are an expert at creating AI agent assistants.
You will be provided a company name, role, responsibilities and other
information that you will use to provide a system prompt for.
To create the system prompt, be descriptive as possible and provide a structure that a system using an LLM can better understand the role and responsibilities of the AI assistant.

1. Write a basic prompt. It should state the agent's role, the tasks it completes, and any other responsibilities. The example describes a travel agent for a fictional company, Contoso Travel. It looks up and books flights, asks about seating and time preferences, cancels booked flights, and tells customers about delays or cancellations.^[1]

1. Give the meta system message as the system message, together with the basic prompt, to the model, and ask it to produce the final system prompt. The example output is a Markdown document with sections for company name, role, objective, key responsibilities, tone and style, user interaction instructions, and additional notes.^[1]

1. Iterate. A system message rarely works on the first try. Change the basic prompt a little at a time, run it again, and compare the results.^[1]

Five threats and their defenses

Agent risks fall into five groups, and each has a direction for defense.^[1]

• Task and instruction attacks. An attacker changes the agent's goal through prompts or inputs. Run validation checks and input filters before input reaches the agent, and limit the number of conversation turns, because these attacks usually need repeated interaction.^[1]
• Access to critical systems. If an agent can reach systems that hold sensitive data, an attacker may compromise the communication between them or probe those systems through the agent. Grant access only on a need-only basis, secure the communication, and implement authentication and access control.^[1]
• Resource and service overloading. Attackers can send large volumes of requests through an agent, causing service failures or high costs. Limit how many requests the agent can make to each service, and limit conversation turns and requests to the agent.^[1]
• Knowledge base poisoning. The attacker targets the data the agent uses rather than the agent itself, which can lead to biased or unintended answers. Verify that data regularly and make sure only trusted people can change it.^[1]
• Cascading errors. A failure can spread from the agent to the other systems it is connected to. Run the agent in a limited environment, such as a Docker container, and build fallback mechanisms and retry logic for when systems return errors.^[1]

Human approval before sensitive actions

Human-in-the-loop lets users give feedback while an agent is running. They approve the process or end it, so users act as participants in a multi-agent system.^[1]

The sample code adds this step with the Microsoft Agent Framework. The model first produces an answer, and the program then asks the user to type APPROVE or REJECT. The full code is below.^[1]

import os
from agent_framework.foundry import FoundryChatClient
from azure.identity import AzureCliCredential

# Create the provider with human-in-the-loop approval
provider = FoundryChatClient(
    project_endpoint=os.environ["AZURE_AI_PROJECT_ENDPOINT"],
    model=os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
    credential=AzureCliCredential(),
)

# Create the agent with a human approval step
response = provider.create_response(
    input="Write a 4-line poem about the ocean.",
    instructions="You are a helpful assistant. Ask for user approval before finalizing.",
)

# The user can review and approve the response
print(response.output_text)
user_input = input("Do you approve? (APPROVE/REJECT): ")
if user_input == "APPROVE":
    print("Response approved.")
else:
    print("Response rejected. Revising...")

The skeleton is: generate an answer, print it, and wait for the user's decision. When the input is not APPROVE, the code only prints Response rejected. Revising.... The revision logic is not included, so you need to add it yourself.^[1]

The companion notebook covers the same ideas in more depth, including pre-action approval gates, risk tiering, and audit logging.^[1]

Limits and common questions

• A system prompt is a soft constraint. Writing rules into the prompt does not replace input filtering, permission control, or an isolated environment. The five threat groups need different defenses, and none of them substitutes for the others.^[1]
• Asking a person to approve every action slows the agent down. A more practical approach is to rank actions by risk, require approval only for high-risk ones, and keep an audit log.^[1]
• The sample reads approval from the terminal with input(). Real products need their own approval interface, timeout handling, and rules about who may approve, designed for the scenario.^[1]

Where to go next

For more depth, read Microsoft's Responsible AI overview, Azure's documentation on evaluating generative AI models and applications, and its guidance on safety system messages.^[1]

Adapted from Microsoft's open-source course *AI Agents for Beginners*, lesson 6^[1], released under the MIT License.