Direct Preference Optimization: Your Language Model is Secretly a Reward Model

The paper

*Direct Preference Optimization: Your Language Model is Secretly a Reward Model* was submitted to arXiv on 29 May 2023 as 2305.18290, by Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning and Chelsea Finn ^[1]. It proposes a way to fine-tune a language model towards human preferences without the reinforcement-learning machinery that such work usually requires ^[1].

The problem

Large-scale unsupervised language models learn broad world knowledge and some reasoning skills from their training data, but precisely controlling their behaviour is difficult, because that training is completely unsupervised ^[1]. The standard remedy is to collect human labels of the relative quality of model generations and fine-tune the model to match those preferences, typically with reinforcement learning from human feedback, or RLHF ^[1]. RLHF works, but the authors describe it as a complex and often unstable procedure ^[1].

The usual RLHF pipeline

RLHF is normally run in two stages. First a reward model is fitted so that it reflects human preferences over pairs of generations. Then the language model is fine-tuned with reinforcement learning to maximise that estimated reward, subject to a constraint that keeps it from drifting too far from the original model ^[1]. Both stages involve large models, sampling and tuning; the reinforcement-learning step in particular is the part that makes the whole procedure awkward to run.

Rewriting the objective

The paper's contribution is a change of variables. The authors introduce a new parameterisation of the reward model in RLHF, one under which the corresponding optimal policy can be extracted in closed form ^[1]. The relation between reward and policy is turned around: instead of searching for the policy that maximises a given reward, the reward is expressed through the policy, so the best policy for any reward has an explicit formula. Once the two are tied together this way, the standard RLHF problem can be solved with only a simple classification loss ^[1]. The language model being trained is, in effect, also the reward model, which is what the subtitle means.

The algorithm that falls out of this is called Direct Preference Optimization (DPO). The authors describe it as stable, performant and computationally lightweight, and single out two things it removes: there is no need to sample from the language model during fine-tuning, and no need to tune many hyperparameters ^[1].

Experiments

The experiments compare DPO with existing preference-alignment methods. The authors report that DPO aligns language models with human preferences as well as or better than existing methods ^[1]. Two comparisons appear in the abstract: it exceeds PPO-based RLHF at controlling the sentiment of generations, and it matches or improves response quality in summarisation and single-turn dialogue, while being substantially simpler to implement and train ^[1].

Why it matters

The engineering burden of the reward-model route has always been the barrier to preference alignment: a separate reward model has to be fitted and maintained, and reinforcement learning has to be run on top of it. DPO leaves the data alone — it still uses human judgements of which of two responses is better — but rewrites the objective as a classification problem over pairs of responses, the kind of loss ordinary supervised training already handles ^[1]. In practice that means the same preference data can be consumed by a conventional training loop rather than a reinforcement-learning pipeline, and the reported results indicate the simplification does not cost alignment quality ^[1]. The evaluations described in the paper cover sentiment control, summarisation and single-turn dialogue ^[1].