DeepSeek-R1-Distill-Qwen with LangChain

LangChain's value is uniform interfaces: chains, prompts, and agents work the same regardless of which model sits underneath. To put a locally deployed DeepSeek-R1-Distill-Qwen-7B into that world, you write one small adapter class. This page builds it, and shows how to split the model's thinking content from its final answer afterward.

Setup

The reference environment is Ubuntu 22.04, Python 3.12, CUDA 12.1, PyTorch 2.3.0, with the CUDA-enabled PyTorch environment preinstalled ^[1]. Install pinned dependencies through a Tsinghua mirror — langchain 0.3.17 alongside transformers 4.48.2 ^[1]:

# 升级pip
python -m pip install --upgrade pip
# 更换 pypi 源加速库的安装
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

pip install transformers==4.48.2
pip install huggingface-hub==0.28.1
pip install accelerate==1.3.0
pip install modelscope==1.22.3
pip install langchain==0.3.17

Download the model with ModelScope's snapshot_download — model name first, cache_dir for the path ^[1]:

from modelscope import snapshot_download

model_dir = snapshot_download('deepseek-ai/DeepSeek-R1-Distill-Qwen-7B', cache_dir='/root/autodl-tmp', revision='master')

The adapter class

The integration itself is a subclass of langchain.llms.base.LLM that overrides two things: the constructor and the _call function ^[1]. The adapter's job is narrow by design — the constructor owns "get the model into memory", _call owns "turn a string into a model call and back" — and that is all LangChain needs from an LLM. The first half loads the model ^[1]:

from langchain.llms.base import LLM
from typing import Any, List, Optional
from langchain.callbacks.manager import CallbackManagerForLLMRun
from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
import torch

class DeepSeek_R1_Distill_Qwen_LLM(LLM):
    # 基于本地 DeepSeek_R1_Distill_Qwen 自定义 LLM 类
    tokenizer: AutoTokenizer = None
    model: AutoModelForCausalLM = None
      
    def __init__(self, mode_name_or_path :str):

        super().__init__()
        print("正在从本地加载模型...")
        self.tokenizer = AutoTokenizer.from_pretrained(mode_name_or_path, use_fast=False)
        self.model = AutoModelForCausalLM.from_pretrained(mode_name_or_path, torch_dtype=torch.bfloat16, device_map="auto")
        self.model.generation_config = GenerationConfig.from_pretrained(mode_name_or_path)
        print("完成本地模型的加载")

Loading happens in the constructor, once, so every later call reuses the model in memory instead of paying the load time again. Three loading details matter: bfloat16 halves memory against fp32 with negligible precision loss for inference; device_map="auto" depends on the accelerate library and distributes model layers across available devices automatically; and the generation config is read from the model directory, carrying the officially recommended generation parameters ^[1]. The second half implements _call, the function LangChain actually invokes ^[1]:

    def _call(self, prompt : str, stop: Optional[List[str]] = None,
                run_manager: Optional[CallbackManagerForLLMRun] = None,
                **kwargs: Any):

        messages = [{"role": "user", "content": prompt }]
        input_ids = self.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
        model_inputs = self.tokenizer([input_ids], return_tensors="pt").to('cuda')
        generated_ids = self.model.generate(model_inputs.input_ids, attention_mask=model_inputs['attention_mask'], max_new_tokens=8192) # 思考需要输出更多的Token数，设为8K
        generated_ids = [
            output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
        ]
        response = self.tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
      
        return response
    @property
    def _llm_type(self) -> str:
        return "DeepSeek_R1_Distill_Qwen_LLM"

Inside _call, the prompt is wrapped as a user message and passed through the chat template — apply_chat_template converts messages into the exact format the model saw during training, and skipping it degrades output quality. Generation runs with max_new_tokens=8192 because thinking needs a large output budget; the generated sequence is sliced to keep only the new tokens and decoded with skip_special_tokens=True ^[1]. Save the file as LLM.py; the rest of the project imports the class from there ^[1].

Calling it and separating the thinking

From here the model behaves like any other LangChain LLM. Instantiate it with your local path — change it to your own download location — and call it with a plain string ^[1]:

from LLM import DeepSeek_R1_Distill_Qwen_LLM
import re

# 文本分割函数
def split_text(text):
    pattern = re.compile(r'<think>(.*?)</think>(.*)', re.DOTALL) # 定义正则表达式模式
    match = pattern.search(text) # 匹配 <think>思考过程</think>回答
  
    if match: # 如果匹配到思考过程
        think_content = match.group(1).strip() # 获取思考过程
        answer_content = match.group(2).strip() # 获取回答
    else:
        think_content = "" # 如果没有匹配到思考过程，则设置为空字符串
        answer_content = text.strip() # 直接返回回答
  
    return think_content, answer_content
  
llm = DeepSeek_R1_Distill_Qwen_LLM(mode_name_or_path = "/root/autodl-tmp/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B")

response = llm("我如何为我学习Python制定目标？")
think, answer = split_text(response) # 调用split_text函数，分割思考过程和回答
print(f"{"-"*20}思考{"-"*20}")
print(think) # 输出思考
print(f"{"-"*20}回答{"-"*20}")
print(answer) # 输出回答

The split_text helper is worth keeping in every reasoning-model project: the raw response contains the thinking between <think> and </think> tags, the regex pulls both parts out, and the fallback branch keeps the answer even when no thinking block appears ^[1]. Showing the two parts separately in logs or UI makes it much easier to see whether a wrong answer came from bad reasoning or bad knowledge.

The adapter pattern also composes: wrap the class with a LangChain prompt template to build domain-specific prompts, chain multiple calls with one link's output feeding the next, and add a parser for structured output — none of it changes when you swap the underlying model later, which is the real payoff for the half hour this adapter takes to write.

A few failures to expect: loading errors about a missing accelerate package mean device_map="auto" cannot work — install it; tight GPU memory calls for shorter max_new_tokens and fewer resident processes; truncated answers mean the thinking phase hit the max_new_tokens ceiling; and split_text finding no <think> block usually means add_generation_prompt was not set to True in the chat template call.

Adapted from the DeepSeek-R1-Distill-Qwen-7B LangChain integration section of Datawhale's open-source guide *self-llm*^[1], released under the Apache-2.0 License.