Building a browser-operating agent with Browser-Use and Playwright

Who this is for, and what you will be able to do

This guide is for readers who know a little Python and want an AI agent to operate web pages. By the end you should be able to build a browser agent that opens a site, searches it and extracts structured data, and you will know which actions should wait for a person to confirm.

What a computer use agent is

A computer use agent (CUA) interacts with websites the way a person does. It opens a browser, inspects the page, and takes the next best action from what it sees^[1]. Its strength is tasks where the next step depends on what is visible on the screen.

The example and the stack

The example is a browser agent that searches Airbnb for listings in Stockholm, extracts structured listing data and identifies the cheapest stay^[1].

The stack combines several tools. Browser-Use handles AI-driven navigation, Playwright and the Chrome DevTools Protocol (CDP) control the browser, Azure OpenAI provides vision-enabled reasoning, and Pydantic handles structured extraction^[1].

Prerequisites

You need Python 3.12 or later, an Azure OpenAI deployment, Chrome or Chromium installed locally, the Playwright dependencies, and basic familiarity with async Python^[1].

Step 1: Install the dependencies

Install the packages used in the notebook, then install the Chromium build for Playwright^[1]:

pip install browser_use playwright python-dotenv
playwright install chromium

Step 2: Configure Azure OpenAI

Set the Azure OpenAI environment variables that the notebook reads^[1]:

AZURE_OPENAI_ENDPOINT=...
AZURE_OPENAI_API_KEY=...
AZURE_OPENAI_CHAT_DEPLOYMENT_NAME=...
# Optional: defaults to the latest API version when omitted
AZURE_OPENAI_API_VERSION=...

If you leave out the API version, the latest version is used^[1].

How the workflow fits together

The example is a hybrid workflow with four steps^[1]:

1. Chrome starts with CDP enabled, so Playwright and Browser-Use share the same browser session.
2. A Browser-Use agent handles open-ended navigation, such as opening Airbnb, dismissing pop-ups and searching for Stockholm.
3. A structured Pydantic schema is used to extract listing titles, nightly prices, ratings and URLs from the active page.
4. Python code compares the listings and highlights the cheapest one.

This keeps the flexible, vision-based reasoning that Browser-Use is good at, and still gives deterministic browser control when you need it^[1].

Agent or actor: how to choose

The two modes differ as follows^[1].

• Agents suit dynamic layouts, finding elements from natural language, and complex workflows with unexpected UI states. They are slower, and their timing is less predictable.
• Actor-style code, which drives the page directly, suits known page structures. It needs exact selectors, and brittle selectors can break when the layout changes. In return it gives full control over waits and retries.

A recommended approach is to start with an agent for exploration and dynamic navigation, then switch to direct page control once the interaction is predictable^[1].

Best practices

Practical rules^[1]:

• Use structured output models, such as Pydantic, so extracted data is validated and type-safe.
• Add delays after actions that trigger visible UI changes.
• Capture screenshots while iterating, so failures are easier to debug.
• Expect websites to change, and prepare fallbacks for pop-ups and layout shifts.

Safety guardrails for browser agents

A browser agent works on a live website, so it needs tighter boundaries than a script that only calls a known API^[1].

First, scope the browsing environment. Use a dedicated browser profile or sandbox, and limit the agent to the domains the task requires^[1].

Second, separate observation from action. The agent can search, read and extract data first. Submitting forms, sending messages, booking travel, making purchases, deleting records or changing account settings all require explicit user approval^[1].

Third, keep secrets out of prompts and traces. Passwords, payment details, session cookies and raw personal data should not enter the model context. The user completes authentication, and sensitive fields are redacted from logs^[1].

Fourth, treat page content as untrusted input. A website can contain text meant for the agent. The agent should ignore page text that asks it to change its goal, reveal data, disable safeguards or visit unrelated sites^[1].

Fifth, add deterministic checks before risky steps. Before asking the user to approve the final step, use code to verify the current URL, page title, selected item, price, recipient and action summary^[1].

Sixth, set budgets and stop conditions. Bound the number of actions, retries, tabs and minutes, and stop when the page state is ambiguous instead of continuing to click^[1].

Seventh, record useful evidence and not everything. Keep action summaries, timestamps, URLs, descriptions of the selected elements and screenshot references, so a run can be reviewed without keeping unnecessary sensitive content^[1].

In the Airbnb example, the safe default is to search listings and extract prices. Signing in, contacting a host or completing a booking is a separate action that the user must approve^[1].

Real-world uses

Common applications include: travel booking and price monitoring, e-commerce price comparison and availability checks, structured extraction from dynamic websites, vision-aware UI testing, website monitoring and alerting, and form filling across multi-step flows^[1].

An enterprise example: Microsoft Project Opal

Microsoft Project Opal (Frontier), which brings computer use into Microsoft 365 Copilot. It runs tasks on a secure Windows 365 Cloud PC, works asynchronously in the background, and lets the user guide the work or take control at any time^[1]. Project Opal is available in the Frontier early access program to users with a Microsoft 365 Copilot subscription, and an administrator must complete setup^[1]. Opal pauses for login credentials, sensitive data or ambiguous instructions, and does not enter passwords or submit forms without explicit confirmation^[1].

Next steps

Run the example to get search and extraction working first. Then replace the steps that have become predictable with direct page-control code, and add confirmation steps and logging before anything touches a real account.

Adapted from lesson 15 of Microsoft's open-source course *AI Agents for Beginners*^[1], released under the MIT License.