Deploying scalable agents with Microsoft Foundry

Who this is for, and what you will be able to do

This guide is for developers who can already run an agent on their laptop and want to serve it to real users. By the end you should be able to tell three deployment patterns apart, see where evaluation gates sit in the agent lifecycle, and add smoke tests to a deployed endpoint.

What changes from prototype to production

A prototype agent and a production agent run the same core loop: reason, call tools, respond. What differs is everything wrapped around that loop. The model is maybe 20% of a production agent, and the rest is the operational skeleton^[1].

The two differ in several concrete ways^[1]:

| Concern | Prototype | Production | | --- | --- | --- | | Hosting | Runs in your notebook | Runs as a versioned, rolled-out hosted service | | Identity | Your personal az login token | Managed identity with scoped RBAC | | State | In memory, lost on restart | Externalised to a thread store or memory service | | Failure | You see the traceback | Retries, fallbacks, dead-letter handling and alerts | | Cost | Tracked loosely | Tracked per request, routed, cached and budgeted | | Quality | You check the output by eye | Evaluated automatically before every release | | Trust | You approve every action | Policy plus human-in-the-loop for risky actions |

Three deployment patterns

You will use three patterns, often in combination^[1].

Client-hosted. The agent object lives inside your own application process, and your code calls the model provider directly. The reasoning loop runs in your service. You own scaling, state and resilience yourself^[1].

Hosted Agent. The Foundry Agent Service runs the agent. It is registered as a resource in Microsoft Foundry, which hosts the reasoning loop, stores threads, and enforces content safety and RBAC. The agent is visible in the Foundry portal. Your application becomes a thin client that creates threads and reads responses^[1].

Agent workflow. Several agents and tools are composed into a graph with explicit control flow. The graph can include sequential steps, branches, human approval nodes and durable checkpoints that pause and resume^[1].

The lifecycle: evaluation is a gate

On Microsoft Foundry, an agent moves through six stages: create, version, deploy, evaluate, observe and retire^[1]. Deployment is a loop, much like a software release cycle, rather than a single push.

Offline evaluation works as a gate. A new version ships only if it clears the evaluation thresholds. Online observability then feeds real failures back into the offline test set^[1].

Scaling: statelessness, routing, caching and limits

Scaling an agent is harder than scaling a stateless web API, because each request can trigger several expensive model and tool calls^[1]. Four techniques carry most of the load.

Stateless request handling. Keep no per-user state in process memory. Store conversation threads in the Foundry thread store or a memory service, so any instance can handle any request and you do not need sticky sessions^[1].

Model routing. Send simple requests, such as intent classification and short factual answers, to a small, fast model. Reserve the large model for real reasoning. Foundry's Model Router can do this, or you can build a lightweight classifier yourself^[1].

Response caching. Common questions, such as how to reset a password, can be answered from a cache without calling the model. Even a modest hit rate reduces cost and latency^[1].

Concurrency and backpressure. Model providers enforce rate limits, so bound your concurrency, retry with exponential backoff, and degrade gracefully. A queued response is better than an error^[1].

Observability: every step becomes a span

The Microsoft Agent Framework emits OpenTelemetry traces natively. Every model call, tool invocation and orchestration step becomes a span^[1]. Exporting those spans to Microsoft Foundry, or to any OpenTelemetry-compatible backend, lets you trace one customer complaint end to end, watch p50 and p95 latency and cost per request, and alert on error-rate spikes and cost anomalies^[1].

The following code wraps a request in a span and adds business attributes:

from agent_framework.observability import get_tracer

tracer = get_tracer()

with tracer.start_as_current_span("support_request") as span:
    span.set_attribute("customer.tier", "enterprise")
    span.set_attribute("routed.model", "gpt-5-nano")
    # agent execution is traced automatically inside this span

Attributes such as customer.tier and routed.model make it possible to answer specific questions, such as whether enterprise customers are being routed to the small model too often^[1].

Cost: right-size, route, then cache

Token usage dominates the cost of a production agent. There are three levers, in order of impact^[1]:

1. Right-size the model. A small model that passes your evaluation gate is usually cheaper. Use evaluation to prove the small model is good enough, rather than defaulting to the largest model.
2. Route by complexity, so that only requests that need large-model reasoning pay large-model prices.
3. Cache aggressively. The cheapest model call is the one you never make.

Evaluation and cost control are two sides of one discipline. Evaluation sets the quality floor, and routing and caching keep cost as close to that floor as possible.

Governance, approval and MCP

Hosted Agents inherit Foundry's RBAC, content safety and audit logging. Give each agent a managed identity with only the access it needs, such as read-only access to the knowledge base and scoped access to the ticketing API^[1].

Human approval is for high-risk actions. The Microsoft Agent Framework supports approval-required tools: the agent proposes an action, execution pauses, a person approves or rejects it, and the workflow resumes^[1].

MCP lets an agent use external tools through a standard interface. In production, treat every MCP server as an untrusted boundary. Pin the server version, run it under a scoped identity, validate what it returns, and never expose secrets to it. An MCP server is a dependency, so it needs patching, auditing and rate limits^[1].

Hands-on lab: a customer support agent

The lab builds a Contoso customer support agent. The notebook is code_samples/16-python-agent-framework.ipynb^[1]. It wires in tool calling for order status and support tickets, RAG over a knowledge base on Azure AI Search with an in-memory fallback so the notebook runs without a Search resource, memory across turns, complexity-based model routing, response caching, human approval for refunds above a threshold, an offline evaluation set that acts as a release gate, and OpenTelemetry tracing^[1].

The core of the request handler combines routing and caching:

async def handle_support_request(query: str, customer_id: str) -> str:
    # 1. Serve from cache when we can.
    cached = response_cache.get(normalize(query))
    if cached:
        return cached

    # 2. Route by complexity to control cost.
    model = "gpt-5-nano" if is_simple(query) else "gpt-5-mini"

    # 3. Run the agent inside a trace span for observability.
    with tracer.start_as_current_span("support_request") as span:
        span.set_attribute("routed.model", model)
        span.set_attribute("customer.id", customer_id)
        response = await support_agent.run(query, model=model)

    # 4. Cache and return.
    response_cache.set(normalize(query), response.text)
    return response.text

The release gate is an evaluation function that returns whether the pass rate clears the threshold:

async def evaluation_gate(agent, test_cases, threshold: float = 0.8) -> bool:
    passed = 0
    for case in test_cases:
        result = await agent.run(case["input"])
        if score_response(result.text, case["expected"]) >= 0.8:
            passed += 1
    pass_rate = passed / len(test_cases)
    print(f"Evaluation pass rate: {pass_rate:.0%} (gate: {threshold:.0%})")
    return pass_rate >= threshold  # only deploy if the gate passes

Deploy only when the function returns true^[1].

Smoke tests: a successful deployment is not a working agent

An offline evaluation runs against the agent object. Once the agent is deployed as a hosted endpoint, you need one more check: is the endpoint actually answering^[1]?

A deployment that succeeds only shows that the control plane accepted the definition. A missing dependency, wrong model routing or an expired connection can leave a green deployment that returns nothing. A smoke test checks in seconds that the endpoint answers, and it should run on every deploy^[1].

The lab's repository includes a smoke-test pipeline built on the AI Smoke Test GitHub Action. Its test cases live in tests/lesson-16-smoke-tests.json, with prompts and assertions for the Contoso support agent^[1]. The workflow signs in with Azure OIDC, sends each prompt to the agent's Responses endpoint, and fails the job when any assertion misses^[1]. The step looks like this:

- name: Smoke-test hosted agent
  uses: JFolberth/ai-smoketest@v1
  with:
    project_endpoint: ${{ inputs.project_endpoint }}
    agent_name: ContosoSupportAgent
    tests_file: tests/lesson-16-smoke-tests.json

The federated identity that runs the pipeline needs the Azure AI User role at Foundry project scope^[1].

Three layers of checks

Smoke tests, offline evaluation and online evaluation each answer a different question^[1]:

• Smoke tests ask whether the agent is reachable and responding. They run on every deploy.
• Offline evaluation asks whether the agent is good enough to ship. It runs before promotion.
• Online evaluation asks how the agent performs in the wild. It runs continuously.

Next steps

Get the endpoint answering first. Then add evaluation gates and smoke tests to your release pipeline, so that each deployment is checked before it is trusted.

Adapted from lesson 16 of Microsoft's open-source course *AI Agents for Beginners*^[1], released under the MIT License.