DeepSeek LLM (7B and 67B)

What it is

DeepSeek LLM is DeepSeek's first series of general-purpose language models, released on 29 November 2023 ^[3]. It comes in two sizes, 7B and 67B parameters, and each size has a base version and a chat version ^[1]. A base model is trained to predict text and is useful as a foundation. The chat version is the base model further tuned to hold a conversation and answer questions as an assistant.

Origins

The base models were pre-trained on a dataset of 2 trillion tokens ^[1]. The chat models were produced in two steps. First came supervised fine-tuning, in which the model learns from example conversations with suitable answers. Then came Direct Preference Optimization (DPO), which trains the model on pairs of answers marked by preference so that it favors the answers people prefer ^[1].

The accompanying paper studied scaling laws for open models. Scaling laws describe how a model's performance changes as its size, training data and computing budget grow, and they help teams decide how large a model to train for a given budget. The authors noted that earlier literature had reached inconsistent conclusions, and they used their own findings to choose the configurations of the 7B and 67B models ^[1]. The paper was posted to arXiv on 5 January 2024 ^[1].

Key features

DeepSeek reported that the 67B base model outperformed LLaMA-2 70B on code, mathematics and reasoning benchmarks ^[1]. A benchmark is a fixed set of test problems used to compare models. Mathematics and reasoning tests are useful because they require several steps of logic rather than recall alone.

DeepSeek also reported that DeepSeek LLM 67B Chat outperformed GPT-3.5 in open-ended evaluations ^[1]. An open-ended evaluation compares free-form answers, such as explanations or advice, rather than checking a single correct choice, so it is harder to score automatically.

Place in development

DeepSeek's next paper reported that a 145B DeepSeekMoE model matched DeepSeek 67B while using a fraction of the computation ^[4]. The 145B model uses the mixture-of-experts design, which activates only part of the network for each input.The original DeepSeek LLM series is no longer maintained ^[3].

The series matters because it was DeepSeek's first general model, and because its paper documented the scaling choices behind the 7B and 67B sizes. Readers who want to understand how the company's later models were designed can start from these two sizes and the design decisions described in the paper.