DeepSeek releases DeepSeek LLM 7B and 67B

Background

DeepSeek, a Chinese AI company, had already released a family of code models. Its next release was a general-purpose model meant to handle everyday text as well as code.

A general-purpose language model is trained on a broad mix of text so that it can answer questions, summarize, translate and write across many topics. The size of such a model is usually stated as a parameter count. Larger models generally need more computing power to train and to run, so releasing more than one size lets users choose between cost and capability.

What happened

On 29 November 2023 DeepSeek released its first general-purpose language models, named DeepSeek LLM ^[3]. The release included two sizes, 7B and 67B parameters. Each size came in a base version and a chat version ^[1]. The base versions are the foundation models that other developers can fine-tune, while the chat versions are ready for conversation.

The base models were pre-trained on a dataset of 2 trillion tokens ^[1]. A token is a unit of text, and a dataset of this size covers a very large amount of writing.

Key details

The chat models were produced with supervised fine-tuning followed by Direct Preference Optimization (DPO) ^[1]. Supervised fine-tuning teaches the model from example answers, and DPO refines it using pairs of answers that people or judges have ranked. Together these steps shape the model from a text predictor into an assistant.

DeepSeek reported that the 67B base model outperformed LLaMA-2 70B on code, mathematics and reasoning benchmarks ^[1]. It also reported that DeepSeek LLM 67B Chat outperformed GPT-3.5 in open-ended evaluations ^[1]. These are the company's own reported comparisons, and they cover specific test sets rather than every possible use.

Impact

The release gave developers a general-purpose model series in two sizes, with both base and chat versions available for study and use. The series is now described as discontinued ^[3], so its main value today is as a record of the company's early general models and of the reported comparisons with LLaMA-2 70B and GPT-3.5.