RLHF needs a reward model, PPO, and a lot of moving parts. DPO deletes most of that and trains with one simple loss.
Introduction
What Did DPO Delete?
RLHF needs four models. DPO needs two. So what did DPO delete?
- RLHF: four models
- DPO: two models
Human Preferences
First, the RLHF recipe. Humans compare two answers and pick the better one. Those picks are the raw material.
- Pick the better answer
- Chosen versus rejected
Step One: Reward Model
Step one: train a reward model. It learns to give the chosen answer a higher score than the rejected one.
- Score chosen above rejected
Step Two: PPO Loop
Step two: run PPO. The model generates answers, the reward model scores them, and a penalty keeps it close to the original.
- Generate, score, update
- KL penalty keeps it close
Heavy and Fiddly
This loop works, but it is heavy. Sampling, scoring, and unstable training make it expensive and hard to tune.
- Costly sampling
- Unstable training
Skip the Middlemen
DPO makes a bold move. Skip the reward model. Skip the sampling loop. Train directly on the preference pairs.
- No reward model
- No RL loop
One Simple Loss
The single loss does one thing. It raises the probability of the chosen answer and lowers the rejected one, relative to the reference model.
- Chosen up, rejected down
- Relative to reference
Reward Hidden Inside
It is just supervised-style training. Same data, far fewer moving parts. The reward model is hidden inside the policy itself.
- Policy is its own reward model
Why It Works: Next
Why does that work? The derivation is next lesson. Follow so you do not miss it.
- The DPO derivation