xAI announces Grok-1.5 with a 128K context

Background

Grok-1.5 was an upgrade to Grok-1, the model xAI had released before it.^[1] Upgrades of this kind usually aim at a few things at once: answering harder questions more accurately, keeping more of a conversation or document in view, and handling more types of input. xAI's announcement focused on the first two.

What happened

On 28 March 2024 xAI announced Grok-1.5, with improved reasoning and a context window of 128,000 tokens.^[1] The company reported gains on two tests. On MATH, a set of competition-style maths problems, the score rose from 23.9% for Grok-1 to 50.6%. On HumanEval, a set of Python programming tasks, it rose from 63.2% to 74.1%.^[1]

The model did not reach everyone at once. It went first to early testers and to existing Grok users on X.^[1] It reached X Premium users on 15 May 2024.^[3]

About two weeks later, on 12 April 2024, xAI previewed Grok-1.5V, its first multimodal model, which handles images as well as text.^[2] It reported 68.7% on a new benchmark it had created for the preview, RealWorldQA, which contains over 700 images with questions that have verifiable answers. The benchmark was released under the CC BY-ND 4.0 licence.^[2]

Key details

The preview was a limited one. Grok-1.5V itself was not released publicly.^[3] Anyone who wanted to check its results independently had to wait, while the RealWorldQA images and questions were made available for others to use.^[2]

Impact

The announcement showed xAI moving the Grok line in two directions at once: longer and more reliable text reasoning, and image understanding. For readers, the main lesson is that a model's announced score is a claim made by its maker, and that a benchmark is most useful when its questions and images are published so that others can check the results.