Computer use

What it is

On 22 October 2024, Anthropic released computer use in public beta, alongside an upgraded Claude 3.5 Sonnet and a new, smaller model called Claude 3.5 Haiku ^[1]. A model with this capability does not only answer in text. It looks at what is on the screen, moves the cursor, clicks buttons and types text, operating a computer the way a person does ^[1]. The company described Claude 3.5 Sonnet as the first frontier AI model to offer computer use in public beta ^[1].

How it works

Anthropic built an API — an interface that other programs call — that translates an instruction into computer commands. A developer can describe a task such as filling out a form, and the system turns that instruction into actions: checking a spreadsheet, opening a browser, entering data ^[1]. The model is not following a fixed script for one program; it is choosing steps on screen.

The design choice behind this was deliberate. Rather than building one tool for each task, Anthropic said it was teaching the model general computer skills, so that it can use standard software designed for people ^[1]. That is the difference between a catalogue of narrow utilities and a single skill that can be pointed at whatever program is already installed. A tool built for one job only works for that job; a general skill can be applied to software the model has never been specifically prepared for, because the interface — a screen, a pointer, a keyboard — is the same one people use.

How well it works

The capability was measured on OSWorld, a benchmark that evaluates a model's ability to use computers like people do. There, Claude 3.5 Sonnet scored 14.9% in the screenshot-only category, against 7.8% for the next-best system, and 22.0% when it was given more steps to complete a task ^[1]. Those numbers describe a system that succeeds some of the time and fails often, which is roughly how Anthropic presented it.

Limits

The company called the capability experimental, and at times cumbersome and error-prone. Scrolling, dragging and zooming still challenge the model, and developers were advised to start with low-risk tasks ^[1]. These are ordinary actions for a person and awkward ones for a model that works from what it can see: a long page has to be moved, a window resized, an object dragged from one place to another.

Safety

Because a model that can click and type can also do damage, Anthropic developed new classifiers that identify when computer use is being used and whether harm is occurring ^[1]. Under its Responsible Scaling Policy, the company said the ASL-2 standard remained appropriate for the release, and that joint pre-deployment testing was conducted by the US AI Safety Institute and the UK AI Safety Institute ^[1].

Why it matters

Most software is written for human hands and eyes. Menus, forms, buttons, scroll bars — all of it assumes a person at a screen. Computer use takes that constraint and turns it into an interface for a model: instead of asking developers to rebuild each application as an API, the model reaches the application the way its users do. Anthropic's framing captures the bet — general computer skills rather than one tool per task ^[1]. The published benchmark scores and the company's own warning about error-prone behaviour sit alongside each other, and together they describe what the release was: a working, limited, publicly available first version of a capability rather than a finished product.