Direct preference optimization

What it is

Direct preference optimization (DPO) is a method for fine-tuning a language model on data that records which of two answers people prefer. What distinguishes it is the shape of the training objective: it works like supervised learning rather than a reinforcement-learning loop, and it removes the intermediate step of training a separate reward model.^[1] It is an alternative to the standard recipe of reinforcement learning from human feedback (RLHF).^[1]

How it works

The usual RLHF pipeline has two stages. First, human comparisons are used to fit a reward model that scores answers; then the policy — the language model being trained — is optimised with PPO so that it scores highly under that reward while not drifting too far from the model it started from.^[1][2] The authors of the DPO paper describe this pipeline as complex and often unstable.^[2]

DPO keeps the preference data but rewrites the problem. By a change of variables, the reward is expressed implicitly as the log ratio between the policy being trained and a supervised fine-tuned reference model. The paper introduces a new parameterization of the reward model in RLHF that allows the corresponding optimal policy to be extracted in closed form, and this reduces the standard RLHF problem to a simple classification loss.^[1][2]

Two consequences follow. No explicit reward model has to be trained, and no reinforcement-learning loop has to be run.^[1] And because the reward is defined through the model's own probabilities, the language model itself is, in effect, treated as the reward model.^[1]

In practice the training signal comes from pairs: a prompt, an answer people preferred, and one they did not. The loss compares how much more probable the model makes each answer than the reference model does, and adjusts the parameters so that the preferred answer gains relative likelihood and the rejected one loses it. That is why the procedure feels closer to supervised learning than to optimising against a separate scorer.

Where it came from

Rafailov, Sharma, Mitchell, Ermon, Manning and Finn published the method in 2023 as arXiv:2305.18290, under the title "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"; the paper was first submitted in May 2023.^[1][2] The title states the central observation directly: with the right change of variables, the language model already carries the reward inside it, so a separate scorer is not needed.^[1]

Trade-offs

Its advantages are simpler implementation and training, closer to supervised learning, with results comparable to and sometimes better than RLHF.^[1] On some benchmarks, such as truthfulness, it performs worse than RLHF.^[1] The DPO loss is also not H-consistent, so it may not reduce ranking errors; this has motivated variants such as SA-DPO.^[1] In short, DPO trades the machinery of RLHF for a simpler objective, and gets much of the benefit — but not on every measure.