WebArena: A Realistic Web Environment for Building Autonomous Agents

The problem it addresses

Agents driven by language models are meant to do work on the web: click, type, navigate, fill in forms, look things up. To measure whether they can, research needs an environment where the same task can be run again and again under the same conditions. Live websites change from day to day, and a frozen copy of a page does not react to what an agent does. WebArena was built as an answer to that: a highly realistic and reproducible environment for language-guided agents performing tasks on the web.^[1]

How the environment is put together

The environment is made of fully functional websites from four common domains: e-commerce, social forum discussions, collaborative software development, and content management.^[1] Because the sites work rather than merely display, an agent's actions change the state of the site — a post is created, an order is placed, a repository is edited — and the resulting state can be inspected afterwards.

To push tasks closer to what people actually do online, the environment is enriched with tools such as a map, and with external knowledge bases such as user manuals.^[1] A task can therefore require reading documentation or checking a location before acting, not just clicking through pages in sequence.

The authors said their code, data, environment reproduction resources and video demonstrations were publicly available at webarena.dev.^[2] The reproduction resources are what make comparison possible: another group can rebuild the same environment and run the same tasks, so a score means the same thing across papers.

Tasks and how success is judged

The benchmark tasks are diverse, long-horizon, and designed to emulate tasks that humans routinely perform on the internet, and evaluation focuses on the functional correctness of task completions.^[1] Long-horizon means a single task takes many steps, with intermediate results that later steps depend on. Functional correctness means the grader looks at whether the end state of the site is the one the instruction asked for, rather than at the exact clicks, page paths or wording the agent produced: there may be several valid routes to the same correct result, and the measurement is meant to accept all of them.

The paper and its result

The paper was submitted to arXiv on July 25, 2023 by Shuyan Zhou, Frank F. Xu, Hao Zhu and 11 other authors including Graham Neubig.^[1] Its central measurement is a comparison: the best GPT-4-based agent achieved an end-to-end task success rate of 14.41%, significantly lower than the human performance of 78.24%.^[1]

Why the gap matters

An end-to-end success rate counts a task only when it is fully completed, so the 14.41% figure describes finished work rather than plausible-looking intermediate steps. A model that can describe how to book something, edit a page or open a discussion thread is not thereby able to carry the sequence through in a live interface where each action changes what is available next.

The human figure gives the number a scale. 78.24% is what people reached on the same tasks, so the distance between the two is a statement about how much of this kind of ordinary web work current agents leave undone, and it turns a vague sense that agents struggle into something that can be tracked as models change.^[1]

A reproducible environment with a fixed set of tasks also makes results comparable over time: when a new agent is reported at a higher success rate, the claim rests on the same sites, the same instructions and the same notion of correctness. And because the environment spans e-commerce, a forum, collaborative software development and content management, it exercises quite different kinds of pages and different kinds of goals within one benchmark.^[1]

Scope

A figure like 14.41% is a statement about this benchmark and these tasks, judged by functional correctness on four self-contained websites.^[1] What the paper supplies is a controlled setting in which the question — can a language-guided agent finish a real piece of web work? — has a definite, repeatable answer.