Grok-1.5 and Grok-1.5V

What it is

Grok-1.5 is a large language model (LLM) developed by xAI and announced on 28 March 2024 as an upgrade to Grok-1.^[1] Like other LLMs, it learns from very large amounts of text and writes its answers one token at a time. A token is a small piece of text, often a whole word or part of one. xAI's main claims for the new version were stronger reasoning, meaning better performance on problems that need several steps, and a much longer context window.

The context window

The headline change was the context window. Grok-1.5 can take in 128,000 tokens at once, which xAI said was 16 times the context of Grok-1.^[1] The context window is the amount of text a model can keep in view while it works. A longer window lets it read a long document, a large codebase or a long conversation without losing the earlier parts as quickly.

How it was released

xAI first made Grok-1.5 available to early testers and to existing Grok users on X.^[1] It reached X Premium users on 15 May 2024.^[1][3] The staged release meant that the model was used by a small group before it reached the wider paying audience on the X platform.

Benchmark results

xAI published benchmark results for the model. On MMLU, a test of knowledge across many school and professional subjects, Grok-1.5 scored 81.3% with 5-shot prompting. On MATH, a set of competition-style maths problems, it scored 50.6% with 4-shot prompting. On GSM8K, a set of grade-school word problems, it scored 90% with 8-shot prompting. On HumanEval, a set of Python programming tasks, it scored 74.1% with 0-shot prompting.^[1] The number of shots is the number of worked examples shown to the model before the test question: 5-shot means five examples, and 0-shot means none.

For comparison, xAI reported that Grok-1 had scored 73% on MMLU, 23.9% on MATH, 62.9% on GSM8K and 63.2% on HumanEval.^[1] The largest gain was on MATH, where the score more than doubled. Benchmarks like these are useful for comparing versions of one model, but they measure only the kinds of questions in each test, so a higher score does not guarantee better answers in every situation.

Proprietary release

Unlike Grok-1, Grok-1.5 is proprietary: its weights, the trained numbers that make up the model, were not released.^[3]

Grok-1.5V

On 12 April 2024 xAI previewed Grok-1.5V, which it described as its first multimodal model.^[2] A multimodal model accepts more than one kind of input. Grok-1.5V was built to handle images as well as text, and it can work with documents, diagrams, charts, screenshots and photographs.^[2]

xAI reported zero-shot results for Grok-1.5V on several vision tests: MMMU 53.6%, MathVista 52.8%, AI2D 88.3%, TextVQA 78.1%, ChartQA 76.1%, DocVQA 85.6% and RealWorldQA 68.7%.^[2] Several of these tests ask questions about charts, diagrams and scanned documents, so they show how well a model reads visual information and reasons about it. Zero-shot means the model was given no worked examples before each question.

With the preview, xAI also released RealWorldQA, a benchmark for spatial understanding, which is how well a model reasons about the layout of objects in a real scene. It contains over 700 images, each with questions that have verifiable answers. Many of the images were taken from vehicles. The benchmark was released under the CC BY-ND 4.0 licence, which allows others to share it with credit as long as they do not change it.^[2]

Grok-1.5V was never released to the public.^[3] Its results therefore come from xAI's own reporting, and outside users could not test it directly at the time.

Place in the Grok line

Grok-1.5 was followed by Grok-2 in August 2024. In that step, GPQA, a set of graduate-level science questions, rose from 35.9% to 56.0%, and MMLU rose from 81.3% to 87.5%.^[4] The Grok line therefore kept moving through a sequence of large model releases, each one building on the last.