DPO Derived: Preference Optimization Without RL

Last time we looked at RLHF: a reward model, a PPO loop, and four models fighting for your GPU memory. It works, but it is fragile and expensive. In this lesson we will show that the whole thing collapses into one simple loss function. Pure algebra, no reinforcement learning, and you can derive it yourself.

• One loss, no reward model
• No PPO loop

Here is the plan. We start from the RLHF objective, solve it exactly, and flip the solution upside down to express reward through policies. Then we plug that into the Bradley-Terry model and watch a nasty term cancel. After that comes a worked example with real numbers, gradient intuition, code in PyTorch and TRL, and honest limitations.

• Derive the DPO loss
• Numbers, gradients, code
• Limits and mistakes

Quick recap. After supervised fine-tuning, we collect preference data: for a prompt x, humans pick a chosen answer y w and a rejected answer y l. We fit a reward model r of x and y on those pairs. Then PPO tunes the policy to score high reward while staying close to a frozen reference model.

• Pretrain, then SFT
• Learn reward from preferences
• Optimize policy with RL

The objective is this. Maximize, over policy pi, the expected reward of sampled answers, minus beta times the KL divergence between pi and the reference policy pi ref. Pi is the model we train. Pi ref is the frozen SFT model. Beta is a positive number controlling how tightly we are leashed to the reference.

• Maximize expected reward
• Penalize drift from reference

Why have the KL term at all? Without it, the policy would exploit quirks in the reward model and drift into gibberish that scores well. With a large beta the model barely moves from the reference. With a small beta it chases reward aggressively. Everything that follows is about solving this one objective exactly.

• Prevents reward hacking
• Keeps fluent language
• Beta sets the leash length

Let's solve it. Expand the KL as the expected log of pi over pi ref. Factor out beta, and the objective becomes minus beta times the expectation of log pi over pi ref minus reward over beta. Maximizing that is the same as minimizing the expectation of log pi over pi ref, minus r over beta.

• Expand the KL
• Factor out beta
• Flip max to min

Now the key trick. Define Z of x as the sum over all possible answers y of pi ref times e to the power r over beta. It depends only on the prompt x, not on y. Using Z, we can define a new distribution pi star: pi ref times e to the r over beta, divided by Z. It is a valid probability distribution because Z normalizes it.

• Z(x) = sum of pi_ref exp(r/beta)
• Normalizes to a distribution

Plug pi star into the minimization. Inside the log, pi over pi ref divided by e to the r over beta equals pi over pi star times one over Z. So the expression becomes the KL divergence from pi to pi star, minus log Z of x. The log Z term does not depend on pi at all, so it is just a constant.

• Insert pi star into the log
• Get KL(pi || pi_star) minus log Z

So we are minimizing a KL divergence plus a constant. A KL is never negative, and it hits zero exactly when the two distributions match. Therefore the optimal policy is pi star of y given x, proportional to pi ref of y given x times e to the r over beta. Good answers get exponentially upweighted relative to the reference.

• KL is zero only when pi = pi_star
• pi*(y|x) ∝ pi_ref · exp(r/β)

Notice what beta does here. With a tiny beta, r over beta becomes huge, and the optimal policy piles onto the highest-reward answer. With a large beta, the exponent is nearly zero and pi star hugs the reference. For example, a reward gap of 1 with beta 0.1 multiplies the odds by e to the 10, about twenty-two thousand.

• Small beta: sharp tilt
• Large beta: stay near reference

There is a catch. Z of x sums over every possible sequence of tokens, an astronomically large set. We cannot compute it, so we cannot just write down pi star. That is why RLHF uses PPO to approximate the solution by sampling. DPO takes a different route: keep Z around, and make it cancel.

• Sum over every possible text
• Cannot compute directly

Here is the clever flip. Take the log of the optimal policy equation, then solve for the reward. We get r of x and y equals beta times log of pi star over pi ref, plus beta times log Z of x. The reward is now written in terms of policies. The reward model has disappeared into the policy itself.

• Take log of both sides
• Solve for r

Read this carefully. The main part of the reward is beta times the log ratio: how much more likely the policy makes an answer than the reference does. If the policy makes an answer more likely than the reference, that answer is implicitly being rewarded. The leftover term, beta log Z, depends only on the prompt.

• Reward = beta × log ratio
• Plus a prompt-only term

Now bring in preferences. The Bradley-Terry model says the probability that humans prefer y w over y l is the sigmoid of the reward difference, r of x and y w minus r of x and y l. Sigma is the sigmoid, one over one plus e to the minus its input. Only the difference matters, never the absolute reward.

• P(y_w ≻ y_l) = σ(r_w − r_l)
• Only reward differences matter

Now substitute our expression for reward into that difference. The reward of y w contains beta log Z of x. The reward of y l contains exactly the same beta log Z of x, because both answers answer the same prompt. Subtract them, and the intractable term vanishes. This is the entire magic of DPO.

• Substitute implicit reward
• beta log Z appears twice
• Subtraction removes it

What remains is that the probability of preferring y w equals sigma of beta times the quantity: log of pi star over pi ref for the chosen answer, minus the same for the rejected answer. Preferences are now modeled directly by policy probabilities. All we have to do is train our policy pi to make the observed human choices likely.

• P = σ(β[log ratio_w − log ratio_l])
• No reward model required

We replace the unknown pi star with our trainable policy pi, and do maximum likelihood on the preference dataset, just as the ML course did for classification. The loss is minus the log of sigma of beta times the difference of log ratios, averaged over all pairs. That is the DPO loss. It is binary cross-entropy in disguise.

• Replace pi* with trainable pi
• Minimize negative log-likelihood

Let's define every symbol. Pi is the policy being trained. Pi ref is a frozen copy, usually the SFT model. X is the prompt, y w the chosen response, y l the rejected one. Log pi of y given x is the sum of token log-probabilities over the whole response. Beta, often around 0.1, scales the implicit reward.

• pi: trainable policy
• pi_ref: frozen reference
• y_w, y_l: chosen, rejected
• beta: strength of the leash

It helps to name the pieces. Let the chosen log ratio be log pi over pi ref for y w, and the rejected log ratio the same for y l. Their difference times beta is the margin, call it z. The loss is minus log sigma of z. Big positive margin means small loss. Negative margin means the model prefers the wrong answer, so loss is large.

• z = β × (ratio_w − ratio_l)
• Loss = −log σ(z)

Time for numbers. Let beta be 0.1. For the chosen answer, the policy's total log-probability is minus 12, while the reference gives minus 13. For the rejected answer, the policy gives minus 15 and the reference minus 14. Remember these are sums over all tokens in each response, so they are quite negative.

• β = 0.1
• Policy vs reference log-probs

Step one: log ratios. For the chosen answer, minus 12 minus minus 13 equals plus 1. The policy has already made it a bit more likely than the reference. For the rejected answer, minus 15 minus minus 14 equals minus 1, so the policy made it less likely. The model is already moving in the right direction.

• Chosen: −12 − (−13) = +1
• Rejected: −15 − (−14) = −1

Step two: the difference is 1 minus minus 1, which is 2. Multiply by beta 0.1 to get a margin z of 0.2. Sigma of 0.2 is about 0.55. The loss is minus the natural log of 0.55, roughly 0.598. For comparison, when the policy equals the reference, the margin is zero, sigma is one half, and the loss is log 2, about 0.693.

• Difference = 1 − (−1) = 2
• z = 0.1 × 2 = 0.2
• Loss = −log σ(0.2) ≈ 0.598

As training continues, suppose the chosen log ratio grows to plus 10 and the rejected falls to minus 10. The difference is 20, times beta is a margin of 2. Sigma of 2 is about 0.88, and the loss falls to roughly 0.127. Notice that the model never needed a reward score. Only the relative shift against the reference mattered.

• Margin grows, loss shrinks
• z = 2 gives loss ≈ 0.127

Now the gradient. Differentiating the loss gives two pushes. One pushes up the log-probability of the chosen response, and one pushes down the log-probability of the rejected response. Both are scaled by beta. So DPO is like supervised learning on the winner, plus unlikelihood training on the loser, anchored to the reference.

• Raise chosen log-prob
• Lower rejected log-prob

Each update is multiplied by a weight: sigma of the implicit reward of the rejected answer minus that of the chosen one. When the model ranks the pair wrongly, the weight is near one and learning is strong. When the margin is already large, the weight shrinks. At our margin of 0.2, the weight is about 0.45. At a margin of 3, it is under 0.05.

• Weight = σ(r̂_l − r̂_w)
• Big when the model is wrong
• Fades when it is right

Why not simply raise the chosen likelihood and call it done? Because the weighting focuses effort on pairs the model still gets wrong, and stops pushing once the preference is learned. Without that, probabilities would keep sliding toward extremes. The reference ratio also keeps every change measured relative to the starting point, which acts like the KL leash from RLHF.

• Weighting prevents overfitting
• Focus on hard pairs

Let's code it. The function takes four tensors of summed sequence log-probs: policy chosen, policy rejected, reference chosen, reference rejected, plus beta. Compute the chosen ratio and rejected ratio by subtraction, take their difference times beta, and return the negative mean of torch dot nn dot functional dot logsigmoid. Using logsigmoid is more numerically stable than taking the log of sigmoid.

• Inputs: four log-prob tensors
• logsigmoid for stability

Test it with our example. Pass minus 12 and minus 15 for the policy, minus 13 and minus 14 for the reference, with beta 0.1. The function returns about 0.598, matching our hand calculation. Always do this kind of sanity check. Also remember the reference log-probs are computed with no gradient, and the sequence log-prob must sum only over response tokens, not the prompt.

• Feed in −12, −15, −13, −14
• Expect loss ≈ 0.598

In practice, use the TRL library. Build a small dataset with three columns: prompt, chosen, and rejected. Create a DPOConfig with beta 0.1, a small learning rate like five times ten to the minus seven, and a batch size. Pass the model, config, dataset, and tokenizer to DPOTrainer, then call train. If you give no reference model, the trainer makes a frozen copy for you.

• Dataset: prompt, chosen, rejected
• DPOConfig with beta
• Trainer builds the reference

While training, watch the logged metrics. Reward accuracy is the fraction of pairs where the chosen implicit reward beats the rejected. Reward margin is the average gap. Both should rise. Also watch the chosen log-probs themselves. If they plunge steadily, the model may be degrading, even while the loss improves. We will see why in a moment.

• Reward accuracy rising
• Reward margin growing
• Chosen log-probs trend

How does this compare with PPO? DPO needs no separate reward model, no value network, and no sampling during training. It is offline: it learns from a fixed dataset of pairs, with one stable supervised-style loss. PPO is more complex and sensitive to hyperparameters, but it generates fresh samples from the current policy and can explore beyond the data.

• No reward model
• No sampling loop
• Offline, stable, cheap

Researchers have proposed many variants. IPO changes the objective to reduce overfitting to the preference labels. KTO learns from simple good or bad labels rather than pairs. ORPO folds preference optimization into supervised fine-tuning without a reference model. We only name them here; each one tweaks the same basic idea of optimizing preferences directly.

• IPO
• KTO
• ORPO

Now the limitations. Because DPO is offline, it only learns from the responses in your dataset. If those came from a different model, they may look nothing like what your policy would generate, so the signal can be weak or misleading. Preferences cannot adapt as the policy improves. Online or iterative variants try to refresh the data to fix this.

• Only learns from given pairs
• Off-distribution responses
• Stale preferences

Two more known issues. First, DPO often drifts toward longer responses, especially when chosen answers in the data are longer than rejected ones. Second, the loss only cares about the gap. The rejected likelihood can fall faster than the chosen one, so both drop and the chosen answer becomes less likely in absolute terms. Watch for it, and consider adding an SFT term.

• Tends to favor longer outputs
• Chosen likelihood can drop

Now common mistakes. The first is reference mismatch. The derivation assumes the policy starts as the reference, so the reference must be the same model you start from, usually your SFT checkpoint. If you use a different base or a different chat template, the log ratios begin far from zero, and the loss is meaningless noise. Always initialize the policy from the reference.

• Reference must equal the SFT starting model
• Same tokenizer and template

The second mistake is noisy pairs. DPO trusts every label. If chosen and rejected are nearly identical in quality, or the labels are simply wrong, the model learns arbitrary patterns. Filter ambiguous pairs and check samples by hand. Also use a small learning rate, and tune beta: too small lets the policy drift far, too large barely changes it.

• Label noise teaches wrong signals
• Tune beta and learning rate

Let's recap. We began with the KL-regularized reward objective and solved it: the optimal policy is the reference reweighted by e to the reward over beta. Inverting gave reward as beta times a log ratio plus a prompt-only term. In Bradley-Terry, that term cancels, leaving a simple loss. The gradient raises the chosen, lowers the rejected, weighted by how wrong the model is.

• KL objective → optimal policy
• Invert → reward from policies
• Bradley-Terry → Z cancels

DPO is a powerful tool, but it is one option among several. In the next lesson we step back and ask the practical question: when should you fine-tune, when should you use retrieval, and when is a better prompt enough?

• Fine-tune vs RAG vs prompt
• Next lesson: how to decide