OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

A real computer as the test bed

OSWorld is described by its authors as the first-of-its-kind scalable, real computer environment for multimodal agents. The paper was submitted to arXiv on April 11, 2024, by Tianbao Xie, Danyang Zhang, Jixuan Chen and 15 other authors, among them Victor Zhong, Tao Yu, Silvio Savarese and Caiming Xiong^[1]. A multimodal agent here means a model that takes in what a person would see on screen — screenshots, usually alongside text — and outputs actions such as clicks and typed input. OSWorld hands such an agent a whole operating system instead of a mocked-up interface, and it runs across Ubuntu, Windows and macOS^[1]. Three things come bundled together: setting up the starting state of a task, judging a run by executing it, and interactive learning inside the environment^[1].

How a task is defined

The benchmark contains 369 computer tasks^[1]. These are not toy puzzles: they involve real web and desktop applications in open domains, file input and output at the operating-system level, and workflows that span several applications^[1]. Every task comes with a detailed initial state setup configuration and a custom execution-based evaluation script^[1]. The setup configuration matters because the same instruction can be trivial or impossible depending on which files, windows and accounts exist when the agent begins; fixing the starting point is what makes runs from different agents comparable. The evaluation script works by execution: rather than reading the agent's reply, it inspects the machine after the agent has acted and decides whether the requested change actually took place. Because each task carries its own script, the success criterion can be tailored to that task instead of being squeezed into one generic check.

The gap between humans and models

The headline result is a wide gap. Humans complete more than 72.36% of the tasks, while the best model succeeds on only 12.24%^[1]. The authors trace the failures mainly to two difficulties: GUI grounding and operational knowledge^[1]. GUI grounding is the step of turning what the model sees in a screenshot into a precise action — which spot to click, where to type. Operational knowledge is knowing how a computer is actually used: which menu hides a given option, what a dialog box is asking for, which sequence of steps a workflow requires. Both are second nature to a person and hard for a model, and a mistake in either place can sink an otherwise sensible plan.

Aids, and their limits

The paper also looks at what can be done to help agents along. Agents could use the accessibility tree and Set-of-Mark prompting — the accessibility tree being a structured description of the elements on an interface, and Set-of-Mark prompting a way of marking up a screenshot with labels so the model can refer to elements by label rather than by coordinate. These aids helped some models, but they could also mislead them^[2]. Vision-language model agents in particular struggled to ground screenshots into precise action coordinates^[2]. That finding is part of why the benchmark is useful: it separates the parts of the problem that better prompting can patch from the parts that remain unsolved.

What is released, and why it matters

The code, the environment, the baseline models and the data are all publicly available^[1]. Because success is scored on what actually happened on the machine rather than on how the agent describes its work, the benchmark measures whether the job got done. Placing agents in open-domain applications on real operating systems puts the tasks close to the way computers are used in practice, and the size of the gap between the human ceiling and the best model marks out how far multimodal agents still have to go on ordinary desktop work.