smolagents 视觉智能体

会读文本的智能体到处都是，会看图的还没那么普及——但网页浏览、文档理解这类真实任务，绕不开对视觉内容的分析。smolagents 给智能体装眼睛的方式是原生接入视觉语言模型（VLM）：图像既能随任务一起给进去，也能在执行中途动态注入 ^[1]。

开局传图：task_images

第一种方式把图像放在执行起点，与任务提示词一起存为 task_images，智能体在整个执行过程中都能引用它们。示例场景是身份核验：手头有历次派对访客的照片库，来了新访客，比对照片判断放不放行。

先取图：

from PIL import Image
import requests
from io import BytesIO

image_urls = [
    "https://upload.wikimedia.org/wikipedia/commons/e/e8/The_Joker_at_Wax_Museum_Plus.jpg", # Joker image
    "https://upload.wikimedia.org/wikipedia/en/9/98/Joker_%28DC_Comics_character%29.jpg" # Joker image
]

images = []
for url in image_urls:
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" 
    }
    response = requests.get(url,headers=headers)
    image = Image.open(BytesIO(response.content)).convert("RGB")
    images.append(image)

再让智能体看：

from smolagents import CodeAgent, OpenAIServerModel

model = OpenAIServerModel(model_id="gpt-4o")

# Instantiate the agent
agent = CodeAgent(
    tools=[],
    model=model,
    max_steps=20,
    verbosity_level=2
)

response = agent.run(
    """
    Describe the costume and makeup that the comic character in these photos is wearing and return the description.
    Tell me if the guest is The Joker or Wonder Woman.
    """,
    images=images
)

运行结果是一段结构化的描述加判定（每次运行措辞可能有差异）：

    {
        'Costume and Makeup - First Image': (
            'Purple coat and a purple silk-like cravat or tie over a mustard-yellow shirt.',
            'White face paint with exaggerated features, dark eyebrows, blue eye makeup, red lips forming a wide smile.'
        ),
        'Costume and Makeup - Second Image': (
            'Dark suit with a flower on the lapel, holding a playing card.',
            'Pale skin, green hair, very red lips with an exaggerated grin.'
        ),
        'Character Identity': 'This character resembles known depictions of The Joker from comic book media.'
    }

判定给出了答案：来者扮成神奇女侠的样子，实则是小丑。形象描述与身份结论分开输出，方便程序直接取用「Character Identity」字段做门禁决策。

边干边看：动态注入 observation_images

数据库里查不到的访客怎么办？那就让智能体自己去网上找图、找资料——图像在执行过程中动态加进智能体的记忆。

理解这条路径要先看 smolagents 的骨架：智能体基于 MultiStepAgent 类，它是 ReAct 框架的抽象，按结构化循环运行，不同阶段记录不同内容——SystemPromptStep 存系统提示词，TaskStep 记录用户查询和输入，ActionStep 捕获动作与结果日志。浏览网页时，智能体可以截图并把它存进 ActionStep 的 observation_images ^[1]。

装依赖：

pip install "smolagents[all]" helium selenium python-dotenv

智能体需要一套像人一样操作浏览器的工具：

@tool
def search_item_ctrl_f(text: str, nth_result: int = 1) -> str:
    """
    Searches for text on the current page via Ctrl + F and jumps to the nth occurrence.
    Args:
        text: The text to search for
        nth_result: Which occurrence to jump to (default: 1)
    """
    elements = driver.find_elements(By.XPATH, f"//*[contains(text(), '{text}')]")
    if nth_result > len(elements):
        raise Exception(f"Match n°{nth_result} not found (only {len(elements)} matches found)")
    result = f"Found {len(elements)} matches for '{text}'."
    elem = elements[nth_result - 1]
    driver.execute_script("arguments[0].scrollIntoView(true);", elem)
    result += f"Focused on element {nth_result} of {len(elements)}"
    return result


@tool
def go_back() -> None:
    """Goes back to previous page."""
    driver.back()


@tool
def close_popups() -> str:
    """
    Closes any visible modal or pop-up on the page. Use this to dismiss pop-up windows! This does not work on cookie consent banners.
    """
    webdriver.ActionChains(driver).send_keys(Keys.ESCAPE).perform()

截图功能是 VLM 浏览智能体的命脉：

def save_screenshot(step_log: ActionStep, agent: CodeAgent) -> None:
    sleep(1.0)  # Let JavaScript animations happen before taking the screenshot
    driver = helium.get_driver()
    current_step = step_log.step_number
    if driver is not None:
        for step_logs in agent.logs:  # Remove previous screenshots from logs for lean processing
            if isinstance(step_log, ActionStep) and step_log.step_number <= current_step - 2:
                step_logs.observations_images = None
        png_bytes = driver.get_screenshot_as_png()
        image = Image.open(BytesIO(png_bytes))
        print(f"Captured a browser screenshot: {image.size} pixels")
        step_log.observations_images = [image.copy()]  # Create a copy to ensure it persists, important!

    # Update observations with current URL
    url_info = f"Current url: {driver.current_url}"
    step_log.observations = url_info if step_logs.observations is None else step_log.observations + "\n" + url_info
    return

这个函数以 step_callback 的身份挂到智能体上，每一步结束都会触发一次。两个细节值得注意：截图存的是 image.copy()——浏览器下一帧会覆盖缓冲区，不复制观察结果就会失效；同时它把两步之前的旧截图清空，图像很占上下文，不清很快就撑爆输入。

组装与执行：

from smolagents import CodeAgent, OpenAIServerModel, DuckDuckGoSearchTool
model = OpenAIServerModel(model_id="gpt-4o")

agent = CodeAgent(
    tools=[DuckDuckGoSearchTool(), go_back, close_popups, search_item_ctrl_f],
    model=model,
    additional_authorized_imports=["helium"],
    step_callbacks=[save_screenshot],
    max_steps=20,
    verbosity_level=2,
)

agent.run("""
I am Alfred, the butler of Wayne Manor, responsible for verifying the identity of guests at party. A superhero has arrived at the entrance claiming to be Wonder Woman, but I need to confirm if she is who she says she is.

Please search for images of Wonder Woman and generate a detailed visual description based on those images. Additionally, navigate to Wikipedia to gather key details about her appearance. With this information, I can determine whether to grant her access to the event.
""" + helium_instructions)

任务提示词末尾拼了一段 helium_instructions，专门用来约束智能体的浏览行为，让它按正确的步骤导航。最终输出：

Final answer: Wonder Woman is typically depicted wearing a red and gold bustier, blue shorts or skirt with white stars, a golden tiara, silver bracelets, and a golden Lasso of Truth. She is Princess Diana of Themyscira, known as Diana Prince in the world of men.

两种注入方式的取舍

开局传图适合「输入已知」的判定类任务——比对照片库、审阅固定文档；动态注入适合开放式任务——要看的页面在执行前根本未知，只能边走边截。后者的代价是上下文管理：图像是 token 大户，保留最近两步、丢弃更早截图是典型的成本控制手段。

把 VLM 和浏览器自动化拼起来，页面交互就变成了智能体动作：Ctrl+F 式的文本查找、后退、关弹窗。这正是 computer-use 类智能体的基本形态——看屏幕、动真实控件。对确定性部分（导航、查找）交给工具，把真正需要视觉判断的部分（这张脸像谁）留给模型，分工明确，系统才稳。

核验场景里还藏着一个要点：模型输出被设计成可解析的结构——描述与结论分开、结论字段固定，这让门禁逻辑不必理解自然语言就能跑。视觉智能体的工程价值不在「能看」，而在把看见的东西变成可执行决策。

本文改编自 Hugging Face agents-course 的 Vision Agents with smolagents 一节^[1]，原文以 Apache-2.0 许可发布。