xAI releases Grok-1's weights under Apache 2.0

Background

Grok-1 was announced in November 2023 as the model behind the Grok chatbot. Its pre-training ended in October 2023, and the model was not yet public in the form of weights, which are the trained numbers that make up a model.^[1]

What happened

On 17 March 2024 xAI released the base-model weights and the network architecture of Grok-1.^[1] The weights and code were both released under the Apache 2.0 license.^[1][2] The weights were distributed by torrent and through the Hugging Face Hub, under the repository name xai-org/grok-1.^[2]

The release was the raw pre-training checkpoint. A checkpoint is a saved set of trained weights, and this one was not fine-tuned for dialogue or any other task. That means it had learned from its pre-training text but had not been adjusted to behave as a chatbot.^[1]

Key details

Grok-1 is a 314-billion-parameter mixture-of-experts model. Of its 8 experts, 2 are used for each token, so about 25% of the weights are active for a given token.^[1][2] Because only part of the network runs at once, the cost of generating each token is well below what the full size would imply.

The GitHub repository gives the configuration. It lists 64 layers, 48 query attention heads and 8 key/value attention heads, an embedding size of 6,144, a SentencePiece vocabulary of 131,072 tokens, and a context length of 8,192 tokens.^[2]

Impact

The release let outside researchers and developers download and study the architecture and weights of a large mixture-of-experts model under a permissive license. Apache 2.0 allows use, modification and redistribution, including for commercial purposes.^[1][2] Because the checkpoint was a raw base model, anyone who wanted a chatbot had to add their own fine-tuning. The release is a fixed point in the Grok line, marking the moment when xAI made its first model's weights public.