Anthropic releases Claude 2 and opens claude.ai

Background

Claude 2 succeeded the Claude 1 line, whose last version in the API was Claude 1.3. Anthropic compared Claude 2 directly with Claude 1.3 when it launched.^[1]

What happened

On 11 July 2023 Anthropic released Claude 2, with better performance and longer responses. It was available through the API and through a new public beta website, claude.ai.^[1]

The claude.ai beta opened to the public in the US and UK, with wider availability planned.^[1]

Key details

Claude 2 accepted up to 100K tokens in one prompt. A token is a small piece of text, roughly a word or part of a word.^[1]

Anthropic reported gains over Claude 1.3 on several tests. On the multiple-choice section of the Bar exam, a US legal licensing test, the score rose from 73.0% to 76.5%. On Codex HumanEval, a test of Python coding, it rose from 56.0% to 71.2%. On GSM8K, a set of grade-school maths word problems, it rose from 85.2% to 88.0%.^[1]

In Anthropic's red-teaming evaluation, Claude 2 was twice as good at giving harmless responses as Claude 1.3. Red-teaming means testing a model with deliberately difficult or hostile inputs to find where it fails.^[1]

The API was offered to businesses at the same price as Claude 1.3. Jasper and Sourcegraph were named as partners.^[1]

Impact

Wikipedia calls Claude 2 the first Claude model available to the general public.^[2] The claude.ai beta was therefore the first time people outside the API and business channels could use the model directly.^[2]

A public beta is a version of a product that anyone can try, usually before it is considered finished. Anthropic's beta website let people use the model without going through the developer API, while the API itself was offered to businesses.

For readers new to AI, benchmark scores like these compare models on the same test set. A higher score suggests better performance on that kind of task, but it does not guarantee the same result on every task.