GRPO reinforcement-learning fine-tuning of Qwen3

Who this is for

This guide is for people who already train large models with Hugging Face tools and want to try reinforcement learning (RL) fine-tuning for a reasoning model. By the end you should understand the GRPO loop, load Qwen3-8B with Unsloth and attach LoRA, write reward functions, run a GRPO training job, and log the run with SwanLab.

What RL and GRPO are

Reinforcement learning deals with sequential decision making: an agent interacts with an environment and adjusts its policy according to the rewards it gets^[1]. GRPO (Group Relative Policy Optimization) can be seen as a variant or improvement of PPO, aimed at training efficiency and stability^[1]. It tries to remove or simplify the value model (the critic) and estimates each output's advantage by comparing outputs within a group^[1].

The GRPO loop works like this. For one prompt, generate several responses; score each with correctness checks or reward functions; compute the group average; compare each response with that average; and reinforce the higher-scoring ones^[1]. Rule-based reward functions or verifiers can take the place of an LLM reward model, which saves training cost and resources^[1]. When DeepSeek researchers trained R1-Zero with pure reinforcement learning, the model learned without human guidance to re-evaluate its approach and think for longer^[1].

Why Unsloth

Common RL frameworks include TRL, developed by Hugging Face; VeRL, from ByteDance; and Unsloth^[1]. VeRL targets distributed training and is most comfortable with two or more GPUs, and Ray has a steep learning curve for beginners^[1]. Unsloth can run RL on an 8B model on a single GPU with very low resource use^[1].

Unsloth says it can make fine-tuning of Llama-3, Mistral, Phi-4 and Gemma 2 times faster, cut memory use by 70%, and do so without any loss of accuracy^[1]. For Qwen3 (8B), it says fine-tuning is 2 times faster, VRAM use drops by 70%, and the context length it supports is 8 times that of environments using Flash Attention 2^[1]. The project pages are docs.unsloth.ai and the unslothai/unsloth repository on GitHub^[1]^[2].

Environment

The test environment is a single A100 with 80 GB of memory^[1]. Install the packages with^[1]:

pip install unsloth==2025.3.19 vllm==0.8.2

Load the model and attach LoRA

Load Qwen/Qwen3-8B with FastLanguageModel.from_pretrained. The example sets max_seq_length to 2048 and fast_inference=True to use vLLM for generation. If memory runs short, lower gpu_memory_utilization^[1]:

from unsloth import FastLanguageModel
import torch

max_seq_length = 2048  # 最大序列长度，可以增加以支持更长的推理轨迹
lora_rank = 32         # LoRA 的秩，秩越大模型可能越智能，但训练和推理速度会变慢

# 从预训练模型加载模型和分词器
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="Qwen/Qwen3-8B",  # 要加载的预训练模型名称(官网版本/离线版本/unsloth版本均可)
    max_seq_length=max_seq_length,        # 设置模型的最大序列长度
    load_in_4bit=False,                   # 是否以4位加载模型，对于LoRA 16位训练，设置为False
    fast_inference=True,                  # 是否启用 vLLM 快速推理
    max_lora_rank=lora_rank,              # 设置 LoRA 的最大秩
    gpu_memory_utilization=0.7,           # GPU显存使用率，如果显存不足 (OOM)，可以降低此值
)

LoRA is added with get_peft_model. The example uses rank 32 and sets lora_alpha to twice the rank^[1]:

# 为模型添加 PEFT (Parameter-Efficient Fine-Tuning) 配置，这里使用 LoRA
model = FastLanguageModel.get_peft_model(
    model,
    r=lora_rank,  # LoRA 的秩 (r)，选择任何大于0的数字！建议值为 8, 16, 32, 64, 128
    target_modules=[  # 需要应用LoRA的模块名称列表
        "q_proj", "k_proj", "v_proj", "o_proj",  # 注意力机制中的查询、键、值、输出投影
        "gate_proj", "up_proj", "down_proj",     # 前馈网络中的门控、上行和下行投影
    ],
    lora_alpha=lora_rank * 2,  # LoRA 的 alpha 参数，设置为秩的2倍可以加速训练
    use_gradient_checkpointing="unsloth",  # 是否使用梯度检查点技术，"unsloth" 表示使用其优化版本以减少显存使用
    random_state=3407,                   # 随机种子，用于确保结果的可复现性
)

Set the output format

Before RL, the model must learn a fixed output format. The example defines four tags: <start_working_out> and <end_working_out> wrap the reasoning, and <SOLUTION> and </SOLUTION> wrap the final answer^[1]:

reasoning_start = "<start_working_out>"     # 用于包裹模型的“思考过程”开始部分
reasoning_end   = "<end_working_out>"       # 用于包裹模型的“思考过程”结束部分
solution_start  = "<SOLUTION>"              # 用于包裹最终解答开始部分
solution_end    = "</SOLUTION>"             # 用于包裹最终解答结束部分

The system prompt tells the model to write its reasoning first and then give the answer^[1]:

system_prompt = \
f"""You are given a problem.
Think about the problem and provide your working out.
Place it between {reasoning_start} and {reasoning_end}.
Then, provide your solution between {solution_start}{solution_end}"""

SFT first, then GRPO

Data preparation has two parts. For SFT, the example uses the cot split of unsloth/OpenMathReasoning-mini and keeps only rows whose expected answer is a number^[1]. It then keeps only samples whose token length is within the 90th percentile^[1]. SFT uses SFTTrainer for 2 epochs, with a learning rate of 2e-4 and a batch size of 1 per device^[1].

The GRPO stage uses the en config and train split of open-r1/DAPO-Math-17k-Processed^[1].

Design the reward functions

Much of GRPO's behavior depends on the reward functions. The example uses four:

match_format_exactly gives 3.0 points when a response matches the full reasoning and solution format^[1]:

def match_format_exactly(completions, **kwargs):
    scores = []  # 用于保存每个 completion 的得分

    for completion in completions:
        score = 0
        response = completion[0]["content"]  # 获取模型输出内容（假设为 messages 列表中的第一个 assistant 回复）

        # 如果输出内容能成功匹配指定格式（即包含完整 <start_working_out>...<SOLUTION>... 标签结构）
        if match_format.search(response) is not None:
            score += 3.0  # 匹配成功得 3 分（用于奖励格式正确的输出）

        scores.append(score)  # 保存该条 completion 的得分

    return scores  # 返回所有 completion 的格式匹配得分列表

match_format_approximately adds 0.5 for each of the end-reasoning and solution tags that appears exactly once, and subtracts 1.0 for each tag that appears a different number of times^[1].
check_answer gives 5.0 for an exact match with the reference answer, 3.5 after stripping spaces, 2.0 for a numeric result within 10%, 1.5 within 20%, and -2.5 otherwise. It gives -4.5 when the answer cannot be parsed and -2.0 when no answer is extracted^[1].
check_numbers gives 3.5 for an exact numeric match and -1.5 otherwise, 0 when parsing fails, and -2.5 when no number is extracted^[1].

GRPO settings

Each prompt gets 4 generated responses, set by num_generations. More responses can help, but they cost more memory, so reduce the value if you run out of memory^[1]. The example uses temperature 1.0, a learning rate of 5e-6, and max_steps of 100, which suits a debugging run^[1]:

training_args = GRPOConfig(
    vllm_sampling_params = vllm_sampling_params,  # 用于生成 completion 的采样策略
    temperature = 1.0,               # 生成的多样性控制（通常设为 0.7 ~ 1.0）
    learning_rate = 5e-6,            # 训练的学习率（较小以保证稳定收敛）
    weight_decay = 0.01,             # 权重衰减，用于防止过拟合
    warmup_ratio = 0.1,              # warmup 步数占总训练步数的比例（通常为 0.05 ~ 0.1）
    lr_scheduler_type = "linear",    # 学习率调度方式为线性下降
    optim = "adamw_8bit",            # 使用 bitsandbytes 的 8bit AdamW 优化器（省显存）

    logging_steps = 1,               # 每一步打印一次日志（适合 debug）
    per_device_train_batch_size = 1, # 每张 GPU 的 batch size
    gradient_accumulation_steps = 1, # 梯度累积步数（设为 4 可等效 batch size=4）

    num_generations = 4,             # 每个 prompt 生成多少个 response（越多越好，但显存消耗更大）
    max_prompt_length = max_prompt_length,         # 提示 token 最大长度（前面计算得出）
    max_completion_length = max_completion_length, # 回答 token 最大长度（确保总长不超过模型限制）

    # num_train_epochs = 1,          # 可选参数：训练轮次，设置后可按 epoch 控制训练终止
    max_steps = 100,                 # 最大训练步数（适合调试用，正式训练时可加大）
    save_steps = 100,                # 每 100 步保存一次模型（训练短时可以不存）

    report_to = "none",              # 不上传训练日志（可设置为 "wandb" 使用可视化工具）
    output_dir = "outputs",          # 模型输出路径

The four reward functions are passed to GRPOTrainer together^[1]:

trainer = GRPOTrainer(
    model = model,  # 需要训练的语言模型（必须支持 causal LM 格式，如 GPT、LLaMA 等）

    processing_class = tokenizer,  # 用于生成 prompt、解码 response 的 tokenizer（需支持 chat_template）

    reward_funcs = [  # 自定义奖励函数列表，用于计算每条样本的得分
        match_format_exactly,       # 检查是否严格符合 <start_working_out>...<SOLUTION> 格式，匹配得分高
        match_format_approximately, # 检查是否大致有格式标签，宽松评分
        check_answer,               # 与参考答案对比，进行精确匹配、模糊匹配、近似数值匹配
        check_numbers,              # 提取数值进行比较（用于数学题）
    ],

    args = training_args,  # 训练参数配置（使用前面定义好的 GRPOConfig 对象）

    train_dataset = dataset,  # 实际用于训练的数据集

    # 可选：如启用训练 + 验证评估，可替换为如下配置
    # train_dataset = new_dataset["train"],
    # eval_dataset = new_dataset["test"],
)

Save and test

After training, save the LoRA weights with model.save_lora("grpo_saved_lora"), then check that no saved tensor is entirely zero^[1]:

model.save_lora("grpo_saved_lora")

The example also includes merged 16-bit and 4-bit saving, LoRA-only saving, and GGUF formats such as Q8_0, f16 and q4_k_m. All of these sit in if False: blocks, so they do not run unless you change the condition^[1].

Log the run with SwanLab

SwanLab is an open-source tool for recording training runs, with visualization, automatic logging, hyperparameter logging, experiment comparison and team collaboration^[1]^[3]. Register at swanlab.cn first, then set report_to to "swanlab" in the SFT and GRPO configurations^[1]:

        report_to = "swanlab", # 这里改成swanlab

Next steps

For background, the DeepSeek-R1 paper is at arxiv.org/abs/2501.12948^[1]. For more training options, see Unsloth's documentation at docs.unsloth.ai.

Adapted from the tutorial "Qwen3-8B GRPO微调及通过swanlab可视化" in Datawhale's open-source project self-llm^[1], released under the Apache-2.0 License.

Cite this page: GRPO reinforcement-learning fine-tuning of Qwen3. TellWorld, https://www.tellworld.net/practice/practice-qwen3-grpo. Updated 2026-11-14