Grok-1

What it is

Grok-1 is the first large language model (LLM) built by xAI, the company founded by Elon Musk. xAI announced it on 3 November 2023 as the engine of the Grok chatbot.^[1] A large language model is a system trained on large amounts of text to understand and generate language. Grok-1 was the model that answered questions in the first version of Grok.

Origins

Before Grok-1 there was a prototype called Grok-0, a 33-billion-parameter model. xAI said Grok-0 approached the performance of Llama 2 (70B) on standard benchmarks while using about half the training resources.^[1] Grok-1 was then trained from scratch on a custom stack built on JAX, Rust and Kubernetes. Pre-training ended in October 2023.^[1][2]

After its announcement, the model was made available first to a limited number of US users through a waitlist, and later to paid X Premium subscribers.^[1][5]

Main features

Grok-1 is a mixture-of-experts model with 314 billion parameters. It has 8 experts, and 2 of them are active for each token, so about a quarter of its weights are used for any one token.^[3][2] In a mixture-of-experts design, only some of the expert sub-networks run for each input, which keeps the cost of each answer below what the full parameter count would suggest.

The model has 64 layers, 48 query heads and 8 key/value heads, an embedding size of 6,144, and rotary embeddings. A query head and a key/value head are parts of the attention mechanism that lets the model weigh earlier text when producing the next word. Its tokenizer is SentencePiece with a vocabulary of 131,072 tokens, and its maximum context length is 8,192 tokens.^[3][2]

Benchmarks

xAI reported the following results: 62.9% on GSM8k (8-shot), 73.0% on MMLU (5-shot), 63.2% on HumanEval (pass@1) and 23.9% on MATH (4-shot). GSM8k tests grade-school math word problems, MMLU covers exam questions across many subjects, HumanEval tests writing code, and MATH tests competition mathematics. xAI said Grok-1 beat GPT-3.5 and Inflection-1 but trailed GPT-4.^[1]

xAI also reported a test on the Hungarian national high-school mathematics exam from May 2023, which was graded by hand. Grok-1 scored 59%, which corresponds to a C grade. The same exam gave GPT-4 68% and Claude 2 55%.^[1]

Open release

On 17 March 2024 xAI released Grok-1's base weights and architecture under the Apache 2.0 license. The weights were distributed by torrent and through Hugging Face.^[2][3] The release was the raw base-model checkpoint, not a version fine-tuned for dialogue. More detail on that release is given in the separate entry on the open weights.

Where it stands

Grok-1 was xAI's first model, and it was followed by Grok-1.5 in March 2024. That model raised MMLU to 81.3% and extended the context length to 128,000 tokens.^[4] Grok-1 remains the model that started the Grok line of language models.^[1][4]