AI agents in production

Who this is for and what you will get

This is for developers who have a working agent prototype and are about to put it in front of real users. After reading, you should be able to add traces and metrics to an agent, set up offline and online evaluation, and control costs with smaller models, routing, and caching.^[1]

Why you need to see inside the agent before launch

Once an agent moves from prototype to real application, you need to understand its behavior, monitor its performance, and evaluate its output systematically. Without observability, the agent is a black box: its internal state and reasoning are hidden, so problems are hard to locate and performance is hard to tune. With observability, it becomes a glass box. That transparency is vital for building trust and confirming the agent behaves as intended.^[1]

In production, observability is a critical capability for four purposes: debugging and root-cause analysis, managing latency and cost, supporting trust, safety, and compliance, and feeding a continuous improvement loop.^[1]

Step 1: record each run as a trace with spans

Observability tools such as Langfuse or Microsoft Foundry usually represent an agent run as a trace made of spans. A trace is one complete agent task from start to finish, such as handling a user query. A span is one step inside the trace, such as a model call or a data retrieval.^[1]

Microsoft Agent Framework integrates with OpenTelemetry natively. OpenTelemetry is described as an industry standard for LLM observability, with APIs, SDKs, and tools for generating, collecting, and exporting telemetry data. The instrumentation example is below:^[1]

from agent_framework.observability import get_tracer, get_meter

tracer = get_tracer()
meter = get_meter()

with tracer.start_as_current_span("agent_run"):
    # Agent execution is traced automatically
    pass

Automatic instrumentation is not always enough. You can add attributes, also called tags or metadata, to spans, for business context such as user_id, session_id, and model_version. Manually created spans let you wrap custom application logic. This is how a span is created by hand with the Langfuse Python SDK:^[1]

from langfuse import get_client

langfuse = get_client()

span = langfuse.start_span(name="my-span")

span.end()

Step 2: choose the metrics to watch

The right metrics depend on the agent's job, but a few are almost always needed:^[1]

• Latency. Measure how long each task and each step takes. If all model calls add up to 20 seconds, you can switch to a faster model or run the calls in parallel.^[1]
• Cost. Measure what each run costs. If an agent calls an LLM five times for a small quality gain, decide whether that is worth it, or reduce the calls or switch to a cheaper model. Real-time monitoring can also catch spikes, such as a bug that sends the agent into an API loop.^[1]
• Request errors. Count failed API calls and failed tool calls. Add fallbacks or retries, such as switching to a backup provider when the main one is down.^[1]
• User feedback. Explicit feedback includes thumbs-up and thumbs-down, one-to-five star ratings, and written comments. Implicit feedback includes rephrased questions, repeated queries, and clicks on a retry button. Steady negative feedback, or many users asking the same thing again, means the agent is not working as expected.^[1]
• Accuracy. Track how often the output is correct or desirable. Define what success means first, then measure it with automated checks, evaluation scores, or task labels such as "succeeded" or "failed."^[1]
• Automated evaluation. An LLM can score outputs for helpfulness and accuracy. Open source libraries cover specific angles: RAGAS for RAG agents, and LLM Guard for detecting harmful language or prompt injection.^[1]

Step 3: set up offline and online evaluation

Evaluation means analyzing observability data, and running tests when needed, to judge how well the agent performs and how to improve it. It has to be regular, because agents are often non-deterministic and can change through updates or drifting model behavior. Without evaluation, you cannot tell whether a "smart" agent is still doing its job.^[1]

Offline evaluation runs the agent in a controlled setting with curated test datasets whose correct outputs are known, not live user queries. It is repeatable, and because the answers are known, it gives clear accuracy numbers. For example, a math word-problem agent could be tested on a dataset of 100 problems with known answers. Offline evaluation usually happens during development and can run in a CI/CD pipeline to catch regressions.^[1]

The hard part of offline evaluation is keeping the dataset complete and current. An agent can score well on a fixed test set and still meet very different queries in production, so add new edge cases over time. Small smoke-test cases give quick checks, and larger sets give broader performance metrics. A mix of both works best.^[1]

Online evaluation runs in live use and monitors the agent's performance on real user interactions. It can catch things the lab did not anticipate, such as model drift as input patterns change, or unusual queries that were not in the test data. You can also run shadow tests or A/B tests, where a new version runs beside the old one. The difficulty is getting reliable labels from live traffic. Teams often rely on user feedback or downstream signals, such as whether the user clicked a result.^[1]

The two should work together. New failure types found online can be added to the offline dataset, and an agent that does well offline can be deployed and monitored with more confidence. Many teams follow this loop: evaluate offline, deploy, monitor online, collect new failure cases, add them to the offline dataset, refine the agent, and repeat.^[1]

Common problems and fixes

• Inconsistent task results. Refine the prompt and state the objective clearly. Consider splitting the task into subtasks handled by several agents.^[1]
• Endless loops. Define clear termination conditions so the agent knows when to stop. For complex reasoning and planning, use a larger model specialized for reasoning.^[1]
• Poor tool calls. Test the tool's output outside the agent system first, then refine the tool's parameters, prompts, and names.^[1]
• Inconsistent multi-agent behavior. Make each agent's prompt specific and distinct from the others. Add a routing or controller agent that decides which agent should handle a request.^[1]

Most of these problems are easier to find with observability in place, because traces and metrics show exactly where in the workflow things go wrong.^[1]

Three ways to manage cost

1. Use smaller models. Small language models (SLMs) can perform well on some agent tasks at much lower cost. They suit simple jobs such as intent classification or parameter extraction, while larger models handle complex reasoning. The best way to know whether an SLM is good enough is to compare its results with a larger model using your evaluation system.^[1]
2. Use a router model. Use models of different sizes. An LLM, an SLM, or a serverless function routes each request to the best-fit model by complexity. Simple queries go to smaller, faster models, and expensive large models handle only complex reasoning.^[1]
3. Cache responses. Identify common requests and return answers before they reach the agent system. A flow can use basic AI models to judge how similar a new request is to cached ones. For frequently asked questions and common workflows, this can cut costs significantly.^[1]

Limits and common questions

• The examples show how to instrument an agent and create spans by hand. To see the full trace view, you need a platform such as Langfuse or Microsoft Foundry.^[1]
• Automated scores are model outputs too. Spot-check them by hand to confirm the scoring standard is sound. This is general practice rather than something the example demonstrates.
• Safe and trustworthy deployment also depends on the design work in Building Trustworthy AI Agents, so read that alongside this.^[1]

Where to go next

To see monitoring and evaluation in practice, open the example notebook 10-expense_claim-demo.ipynb, which shows how observability tools monitor and evaluate an agent.^[1]

Adapted from lesson 10 of Microsoft's open-source course *AI Agents for Beginners*^[1], released under the MIT License.