Anthropic launches computer use with an upgraded Claude 3.5 Sonnet and Claude 3.5 Haiku

Background

The announcement covered two Claude 3.5 models at once: an upgraded Sonnet and a new Haiku.^[1]

What happened

On 22 October 2024 Anthropic announced three things: an upgraded Claude 3.5 Sonnet, a new Claude 3.5 Haiku, and computer use in public beta.^[1]

With computer use, developers can direct Claude to use a computer much as a person would. Claude looks at a screen, moves a cursor, clicks buttons and types text.^[1]

Anthropic said Claude 3.5 Sonnet was the first frontier AI model to offer computer use in public beta. It also called the capability experimental and, at times, error-prone.^[1]

Key details

The upgraded Sonnet scored 49.0% on SWE-bench Verified, up from 33.4%. SWE-bench Verified is a test built from real software problems. On OSWorld, a test of computer tasks, it scored 14.9% in the screenshot-only category and 22.0% when allowed more steps.^[1]

Claude 3.5 Haiku scored 40.6% on SWE-bench Verified. It matched Claude 3 Opus on many benchmarks.^[1]

Early adopters of computer use included Asana, Canva, Cognition, DoorDash, Replit and The Browser Company.^[1]

The US and UK AI Safety Institutes tested the upgraded Sonnet before release.^[1]

Computer use was offered through the Anthropic API, Amazon Bedrock and Google Cloud Vertex AI.^[1]

Impact

Computer use lets Claude act inside software on a user's behalf, a different kind of task from answering questions or writing text. Anthropic framed the feature as experimental, and named the early adopters listed above.^[1]

SWE-bench Verified tests whether a model can fix real problems from software projects. OSWorld tests whether a model can complete tasks in a computer operating system. Screenshot-only means the model received only images of the screen, with no direct access to the underlying software.

Benchmarks like these are a snapshot taken at one time, on one set of tasks. They show what the model did in Anthropic's testing, and they are not a guarantee of results on a user's own computer.