LoRA fine-tuning Qwen3

Who this is for and what you will be able to do

This is for readers who already use Python and Hugging Face Transformers and want to fine-tune a large model once. After working through it, you will know what supervised fine-tuning (SFT) data looks like, what the main LoRA parameters mean, how to run a training job, how to view its loss curve in SwanLab, and how to load the trained LoRA weights for inference.

What LoRA is

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method. Instead of changing all of the base model's weights, it trains a pair of low-rank matrices and adds them to the base model. This guide fine-tunes Qwen3-8B with data in the supervised fine-tuning (sft) format^[1].

Preparation: install dependencies

Switch pip to the Tsinghua mirror, then install the pinned versions. The guide pins modelscope 1.25.0, transformers 4.51.3, accelerate 1.6.0, datasets 3.5.1, peft 0.15.2 and swanlab 0.5.7^[1]:

# 换清华镜像源
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

pip install modelscope==1.25.0
pip install transformers==4.51.3
pip install accelerate==1.6.0
pip install datasets==3.5.1
pip install peft==0.15.2
pip install swanlab==0.5.7

If the environment gives you trouble, the guide provides a Qwen3 environment image on the AutoDL platform that you can create an instance from^[1].

The model is downloaded with ModelScope. Change cache_dir to your own path^[1]:

# model_download.py
# 注意修改cache_dir为保存的路径
from modelscope import snapshot_download
model_dir = snapshot_download('Qwen/Qwen3-8B', cache_dir='请修改我！！！', revision='master')

print(f"模型下载完成，保存路径为：{model_dir}")

Step 1: prepare SFT data

Each supervised fine-tuning record has three fields: instruction states the task, input holds what the task needs, and output is the answer the model should give^[1]:

{
  "instruction": "回答以下用户问题，仅输出答案。",
  "input": "1+1等于几?",
  "output": "2"
}

Build data for your target task. The guide's example fine-tunes on dialogue data so the model speaks in the style of 甄嬛, and one record looks like this^[1]:

{
  "instruction": "你父亲是谁？",
  "input": "",
  "output": "家父是大理寺少卿甄远道。"
}

Step 2: encode text into model inputs

LoRA training data must be formatted and encoded first. Input text becomes input_ids and output text becomes labels, both as vectors^[1]. The guide's process_func sets the maximum sequence length to 1024 tokens and truncates anything longer. The start of the function is^[1]:

def process_func(example):
    MAX_LENGTH = 1024 # 设置最大序列长度为1024个token
    input_ids, attention_mask, labels = [], [], [] # 初始化返回值
    # 适配chat_template
    instruction = tokenizer(
        f"<s><|im_start|>system\n现在你要扮演皇帝身边的女人--甄嬛<|im_end|>\n"
        f"<|im_start|>user\n{example['instruction'] + example['input']}<|im_end|>\n"
        f"<|im_start|>assistant\n<think>\n\n</think>\n\n",
        add_special_tokens=False
    )
    response = tokenizer(f"{example['output']}", add_special_tokens=False)
    # 将instructio部分和response部分的input_ids拼接，并在末尾添加eos token作为标记结束的token
    input_ids = instruction["input_ids"] + response["input_ids"] + [tokenizer.pad_token_id]
    # 注意力掩码，表示模型需要关注的位置
    attention_mask = instruction["attention_mask"] + response["attention_mask"] + [1]
    # 对于instruction，使用-100表示这些位置不计算loss（即模型不需要预测这部分）
    labels = [-100] * len(instruction["input_ids"]) + response["input_ids"] + [tokenizer.pad_token_id]
    if len(input_ids) > MAX_LENGTH:  # 超出最大序列长度截断
        input_ids = input_ids[:MAX_LENGTH]
        attention_mask = attention_mask[:MAX_LENGTH]
        labels = labels[:MAX_LENGTH]

In this function, the instruction positions in labels are set to -100, so no loss is computed for them, and the model only learns to predict the response. An end token is added after the response to mark where it stops^[1].

Qwen3 is a hybrid reasoning model, so you can choose whether to think. With enable_thinking=False in apply_chat_template, each assistant turn starts with an empty <think> block^[1]:

messages = [
    {"role": "system", "content": "===system_message_test==="},
    {"role": "user", "content": "===user_message_test==="},
    {"role": "assistant", "content": "===assistant_message_test==="},
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False
)
print(text)

The printed output is^[1]:

<|im_start|>system
===system_message_test===<|im_end|>
<|im_start|>user
===user_message_test===<|im_end|>
<|im_start|>assistant
<think>

</think>

===assistant_message_test===<|im_end|>
<|im_start|>assistant
<think>

</think>

With enable_thinking=True, the final assistant turn stops at <|im_start|>assistant with no empty <think> block, so the model can generate its own reasoning^[1]:

messages = [
    {"role": "system", "content": "===system_message_test==="},
    {"role": "user", "content": "===user_message_test==="},
    {"role": "assistant", "content": "===assistant_message_test==="},
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True
)
print(text)

The printed output for that call is^[1]:

<|im_start|>system
===system_message_test===<|im_end|>
<|im_start|>user
===user_message_test===<|im_end|>
<|im_start|>assistant
<think>

</think>

===assistant_message_test===<|im_end|>
<|im_start|>assistant

Step 3: load the model and configure LoRA

Load the tokenizer and base model with AutoTokenizer and AutoModelForCausalLM. The example uses device_map="auto" and torch_dtype=torch.bfloat16, and the path must be replaced with your own^[1]:

tokenizer = AutoTokenizer.from_pretrained('请修改我！！！/Qwen/Qwen3-8B')

model = AutoModelForCausalLM.from_pretrained('请修改我！！！/Qwen/Qwen3-8B', device_map="auto", torch_dtype=torch.bfloat16)

The main LoraConfig parameters are: task_type, the model type, which is CAUSAL_LM for decoder-only models such as Qwen; target_modules, the layers to train; r, the LoRA rank, which sets the size of the low-rank matrices, so a smaller r means fewer parameters; lora_alpha, a scaling parameter; and lora_dropout, the dropout rate on LoRA layers, which helps prevent overfitting^[1].

The effective scale of a LoRA update is lora_alpha/r. In the example, lora_alpha is 32 and r is 8, so the scale is 4 times^[1]. The example's target_modules are q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj and down_proj. The guide says these are mainly attention layers, and layer names differ between models, so check them when you switch models^[1].

config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    inference_mode=False, # 训练模式
    r=8, # Lora 秩
    lora_alpha=32, # Lora alpha
    lora_dropout=0.1 # Dropout 比例
)

Step 4: train and record with SwanLab

In the training arguments, output_dir is the output path, per_device_train_batch_size is the batch size per GPU, gradient_accumulation_steps sets gradient accumulation, and num_train_epochs is the number of epochs^[1]. The example configuration is:

args = TrainingArguments(
    output_dir="./output/Qwen3_8B_LoRA", # 注意修改
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    logging_steps=10,
    num_train_epochs=3,
    save_steps=100,
    learning_rate=1e-4,
    save_on_each_node=True,
    gradient_checkpointing=True,
    report_to="none",
)

SwanLab is an open-source training tracking tool for AI researchers. It offers training visualization, automatic logging, hyperparameter logging, experiment comparison and multi-person collaboration^[1]. The guide suggests registering at https://swanlab.cn/ first, then choosing "(2) Use an existing SwanLab account" when training initializes and logging in with a private API key^[1].

Create the callback and pass it to the Trainer:

import swanlab
from swanlab.integration.transformers import SwanLabCallback

# 实例化SwanLabCallback
swanlab_callback = SwanLabCallback(
    project="Qwen3-Lora",  # 注意修改
    experiment_name="Qwen3-8B-LoRA-experiment"  # 注意修改
)

trainer = Trainer(
    model=model,
    args=args,
    train_dataset=tokenized_id,
    data_collator=DataCollatorForSeq2Seq(tokenizer=tokenizer, padding=True),
    callbacks=[swanlab_callback] # 传入之前的swanlab_callback
)
trainer.train()

The guide's example run finished at global step 699, with a training loss of 2.6425710331557988 and a train runtime of 879.9696 seconds^[1]:

TrainOutput(global_step=699, training_loss=2.6425710331557988, metrics={'train_runtime': 879.9696, 'train_samples_per_second': 12.713, 'train_steps_per_second': 0.794, 'total_flos': 5.190619083415757e+16, 'train_loss': 2.6425710331557988, 'epoch': 2.990353697749196})

After training, SwanLab shows the recorded parameters and the training loss curve^[1].

Step 5: run inference with the LoRA weights

Once you have a checkpoint, load the LoRA weights on top of the base model. The key call is PeftModel.from_pretrained^[1]:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
from peft import PeftModel

mode_path = '请修改我！！！/Qwen/Qwen3-8B'  # 注意修改
lora_path = './output/Qwen3_8B_lora/checkpoint-699' # 注意修改

# 加载tokenizer
tokenizer = AutoTokenizer.from_pretrained(mode_path)

# 加载Qwen3 base model
model = AutoModelForCausalLM.from_pretrained(mode_path, device_map="auto",torch_dtype=torch.bfloat16, trust_remote_code=True)

# 加载lora权重
model = PeftModel.from_pretrained(model, model_id=lora_path)

Generation uses max_length 2500, do_sample True and top_k 1^[1]. In the guide's example, the model answers "你是谁？" in the persona of 甄嬛, and the printed output is^[1]:

我是甄嬛，家父是大理寺少卿甄远道。

Common questions

The model path cannot be found during training. Check the paths in AutoTokenizer.from_pretrained and AutoModelForCausalLM.from_pretrained. The guide's paths are placeholders that must be replaced with your actual download path^[1].

The LoRA path does not match what training saved. The lora_path at inference should point to a checkpoint directory from training, such as the guide's checkpoint-699^[1].

You want thinking mode off at inference. Set enable_thinking=False in apply_chat_template, as the example does^[1].

Limits

This is one small role-play example. Whether the model picks up the target style depends on the amount and quality of your data, and the guide gives no data-size advice. The values for target_modules, learning rate and epochs are example settings to adjust for your model and data. SwanLab needs an account and an API key, and offline use is not covered here^[1].

Next steps

To learn how LoRA works, read this site's entry on LoRA. For the full set of Trainer arguments, see the official Hugging Face Transformers documentation.

Adapted from the "Qwen3-8B LoRA and SwanLab visualization" lesson of Datawhale's open-source self-llm project^[1], released under the Apache-2.0 License.