RLHF Explained: Reward Models, KL Penalty and PPO Step by Step

A pretrained language model is a brilliant autocomplete engine. Ask it a question and it might answer, or it might continue with five more questions, because that is what the internet often looks like. So how does a raw predictor turn into an assistant that feels helpful? A big part of the answer is reinforcement learning from human feedback, or RLHF.

• Pretrained model: autocomplete
• Assistant: helpful and safe

Last lesson we saw how QLoRA lets us fine-tune big models on modest hardware. Today we change what we optimise. We will build a reward model from human comparisons, define the reinforcement learning objective, and understand PPO, the algorithm that classically powered this stage. We will finish with PyTorch code for the two key losses.

• Lesson 23: QLoRA
• Today: RLHF
• Rewards, KL, PPO

Here is the core idea in one sentence. It is much easier for a person to say which of two answers is better than to write the perfect answer from scratch. RLHF turns that cheap judgement into a training signal. Let us see why that matters.

• Judging is easier than writing
• Learn a score from comparisons

Imagine asking someone to write the ideal summary of a long article. That takes minutes, and every writer produces something different. Now show them two candidate summaries and ask which is better. That takes seconds, and people tend to agree more often. Think of judging a cooking contest: few of us can cook like a chef, but most of us can taste two dishes and pick a favourite.

• Writing ideal answers is slow
• Ranking two answers is fast
• Humans agree more on rankings

Supervised fine-tuning, from lesson twenty-one, teaches the model to imitate demonstrations. That has two limits. It only shows what good looks like, never what is worse, and it caps the model near the quality of the people who wrote the examples. Preferences let us push beyond imitation by rewarding answers that are better, even among outputs the model produced itself.

• SFT imitates demonstrations
• Cannot say what is worse
• Capped by demonstrator quality

Let us define symbols. A prompt is x and a response is y. A reward model is a function r of x and y that returns a single number, a score for how much humans would like that response. The policy, written pi, is the language model itself: it defines the probability of each response given the prompt. The goal is a policy whose responses earn high reward.

• Prompt x, response y
• Reward r(x, y) is a number
• Higher means preferred

The classic RLHF pipeline has three stages. First, supervised fine-tuning gives us a model that follows instructions reasonably well. Second, we collect human comparisons and train a reward model. Third, we use reinforcement learning to push the model towards responses the reward model scores highly.

• 1. SFT
• 2. Reward model
• 3. RL against reward

To collect data, we take a prompt, sample two or more responses from the SFT model, and ask annotators which they prefer. Each judgement is stored as a triple: the prompt, the winning response, and the losing response. If annotators rank four responses, that yields six pairs. The reward model never sees a score from a human, only these relative judgements.

• Sample several responses
• Annotators pick the better one
• Store (x, winner, loser)

The reward model is usually a transformer initialised from the SFT model. We remove the vocabulary output layer and add a small head that produces one number, read at the final token of the response. So the model reads the prompt and the response together and compresses its judgement into a scalar. The question is how to train that scalar from comparisons alone.

• Start from the SFT transformer
• Replace the output head
• Scalar score at the last token

We use the Bradley-Terry model, borrowed from ranking players in games. It says the probability that response A beats response B depends on their score gap. Formally, the probability that the winner y w is preferred over the loser y l equals e to the r w, divided by e to the r w plus e to the r l. A little algebra turns that into the sigmoid of the difference, r of y w minus r of y l.

• P(A beats B) from scores
• Higher score, higher win chance
• Uses the sigmoid

Training is maximum likelihood. We want the model to assign high probability to the choices humans actually made. Taking the negative log of that probability, and averaging over the dataset, gives the loss: minus log sigmoid of r of x and y w, minus r of x and y l. Notice it only depends on the gap between the two scores, so adding a constant to every reward changes nothing.

• Maximize likelihood of human choices
• Take negative log
• Loss = -log sigmoid(r_w - r_l)

A worked example. Suppose the reward model scores the winner at two point zero and the loser at zero point five. The gap is one point five. The sigmoid of one point five is about zero point eight one eight, so the model gives the human choice an eighty-two percent chance. The loss is minus the log of that, about zero point two zero. Small loss, because the model agrees with the human.

• r_w = 2.0, r_l = 0.5
• gap = 1.5
• sigma(1.5) = 0.818
• loss = 0.20

Now flip the scores: the winner gets zero point five and the loser two point zero. The gap is minus one point five, the sigmoid is only about zero point one eight two, and the loss jumps to about one point seven zero. If the two scores are equal, the probability is exactly one half and the loss is zero point six nine three, the log of two. The gradient is largest when the model is confidently wrong, which is exactly where we want learning to happen.

• r_w = 0.5, r_l = 2.0
• gap = -1.5
• sigma(-1.5) = 0.182
• loss = 1.70

Be careful about what we have built. The reward model is a learned proxy for human preference, trained on a limited set of comparisons. It is reasonably accurate on responses that look like its training data, and unreliable far outside it. Remember this, because the next stage is going to push the policy as hard as it can into exactly the places where the proxy might be wrong.

• A proxy, not truth
• Trained on limited comparisons
• Imperfect off-distribution

Now the reinforcement learning view. The state is the prompt plus the tokens generated so far. The action is the next token. When the response ends, the reward model gives a score. The policy generates a whole response, gets a reward, and we adjust its weights so that high-reward responses become more likely.

• State: prompt + tokens so far
• Action: next token
• Reward: scored at the end

The naive objective is simply to maximise expected reward. It fails badly. The policy will find quirks in the reward model, such as odd phrases, excessive length, or flattery, that earn high scores without being genuinely good. This is reward hacking, and it is a case of Goodhart's law: when a measure becomes a target, it stops being a good measure.

• Maximize E[r] alone
• Policy drifts to exploit the scorer
• Text becomes gibberish that scores well

The fix is to add a leash. We maximise the expected reward minus beta times the KL divergence between the current policy pi and a frozen reference policy, usually the SFT model. KL divergence measures how far the policy's distributions have moved from the reference. Beta is a coefficient: large beta keeps the model close to the reference, small beta lets it roam.

• Maximize E[r] - beta * KL(pi || pi_ref)
• pi_ref = frozen SFT model
• beta sets the leash length

In practice the KL is estimated from samples. For each generated token we compute the log probability under the policy minus the log probability under the reference, and sum across the response. Suppose a response earns reward five, and its summed log ratio is twenty nats. With beta of zero point one, the penalty is two, so the net reward is three. A response that earns reward six but drifts forty nats pays a penalty of four, netting only two. The drift is not worth it.

• Per-token log pi - log pi_ref
• Sum over the response
• Reward 5.0, KL 20, beta 0.1
• Net reward 3.0

Why does this prevent hacking? The reward model is only trustworthy near the kind of text it was trained on, and the reference model produces exactly that kind of text. Exploits tend to live far away, in strange regions of response space. Those regions cost a lot of KL, so the penalty cancels the inflated reward. The KL term keeps the policy inside the region where the proxy still means something.

• Hacks live far from the reference
• Far means large KL
• Proxy is only trustworthy nearby

Now, how do we actually optimise this? The basic tool is the policy gradient. Sample some responses, look at how good each one was, and nudge the model to increase the log probability of good responses and decrease it for bad ones. The size of the nudge is proportional to how good or bad. Simple, but noisy, and a single oversized step can wreck the model.

• Sample responses
• Raise probability of good ones
• Lower probability of bad ones

To reduce noise we subtract a baseline. The advantage is the actual return minus the value, where the value is a prediction of how much reward we expected from this state. If we expected two point two and got three, the advantage is zero point eight: better than expected, so reinforce it. A separate value model, often a copy of the reward model backbone with a value head, learns these predictions. In practice, generalised advantage estimation blends them across tokens.

• A = return - V(state)
• V predicts expected reward
• Positive A: better than expected

Proximal policy optimisation, or PPO, reuses a batch of samples for several update steps, so it needs to track how much the policy has changed. It does this with the probability ratio: the new policy's probability of a token divided by the old policy's probability when the sample was generated. A ratio of one means no change. A ratio of one point five means the token became fifty percent more likely.

• ratio = pi_new / pi_old
• 1.0 means unchanged
• 1.5 means 50 percent more likely

The PPO objective takes the minimum of two terms: the ratio times the advantage, and the ratio clipped to the range one minus epsilon to one plus epsilon, times the advantage. Epsilon is commonly zero point two. The effect is that once a token has moved more than twenty percent in the helpful direction, there is no further reward for pushing it, so the gradient vanishes.

• min(r*A, clip(r, 1-eps, 1+eps)*A)
• eps is typically 0.2
• Removes incentive for big steps

Four cases with epsilon zero point two. Advantage plus two, ratio one point five: the raw term is three, the clipped term is one point two times two, which is two point four, and the minimum is two point four, so clipped and no gradient. Advantage plus two, ratio zero point seven: raw one point four, clipped one point six, minimum one point four, so the gradient flows. Advantage minus one, ratio zero point six: raw minus zero point six, clipped minus zero point eight, minimum minus zero point eight, clipped. Advantage minus one, ratio one point five: raw minus one point five, clipped minus one point two, minimum minus one point five, so the gradient still flows and corrects the mistake.

• A=+2, r=1.5: min(3.0, 2.4) = 2.4
• A=+2, r=0.7: min(1.4, 1.6) = 1.4
• A=-1, r=0.6: min(-0.6, -0.8) = -0.8
• A=-1, r=1.5: min(-1.5, -1.2) = -1.5

Here is the practical pain. PPO-based RLHF needs four models at once. The policy, which is trained. The reference, frozen, for the KL term. The reward model, frozen, for scoring. And the value model, trained, for advantages. Two of them also carry optimiser states. For a large model that means a lot of GPU memory, plus a slow generation step in every iteration. Techniques like LoRA and sharing backbones help.

• Policy (trained)
• Reference (frozen)
• Reward model (frozen)
• Value model (trained)

Putting it together, one iteration looks like this. Sample prompts and generate responses with the current policy. Score each response with the reward model and subtract the KL penalty against the reference. Use the value model to compute advantages. Then run a few gradient steps on the clipped objective, update the value model, and repeat with fresh samples.

• Generate responses
• Score and compute KL
• Compute advantages
• Several clipped updates

On to code, on toy tensors rather than full training. For the reward loss, we have two tensors of scores, one for chosen and one for rejected responses. The loss is the negative mean of log sigmoid of the chosen minus the rejected. Use the logsigmoid function, not the log of the sigmoid, because it is numerically stable. With the scores from our example, this prints about zero point two zero.

• F.logsigmoid(r_w - r_l)
• Negative mean
• Toy tensors

For PPO, we store log probabilities from the old policy and compute new ones. The ratio is the exponential of the difference of log probabilities. We build the unclipped term, ratio times advantage, and the clipped term using torch clamp between zero point eight and one point two. Take the elementwise minimum, average it, and negate it because optimisers minimise. Always compute ratios in log space; dividing raw probabilities is numerically fragile.

• ratio = exp(logp_new - logp_old)
• clip with torch.clamp
• Negate the min for a loss

The KL shaping is just as short. Per token, compute the policy log probability minus the reference log probability, sum over the response, multiply by beta, and subtract from the reward. The reference model runs under no-grad, since it is frozen. In many implementations this penalty is applied per token, with the reward model's score added at the final token, which gives the value model a denser signal.

• kl = logp_policy - logp_ref
• shaped = reward - beta * kl.sum()
• Detach the reference

Now the common mistakes. The first is a weak reward model. If it agrees with held-out human comparisons only slightly better than chance, RL will happily optimise its noise. Always measure pairwise accuracy on a held-out set, inspect the highest-scored responses by hand, and fix the reward model first. Reinforcement learning amplifies whatever signal you give it, including the flaws.

• Low accuracy on held-out pairs
• Policy exploits its blind spots
• Check before running RL

The second mistake is setting the KL penalty too small. The reward curve climbs beautifully while real quality collapses: responses get longer, repetitive, or oddly flattering. Watch the KL value during training, read samples every few hundred steps, and compare against a held-out evaluation. Many implementations adapt beta to hit a target KL. Setting beta too large is also wasteful, because the model barely moves from the SFT starting point.

• Beta too low: policy drifts
• Reward climbs, quality falls
• Watch KL and sample outputs

Human feedback has its own biases. Annotators may prefer confident, longer, or agreeable answers even when a shorter or more critical one is more correct. The reward model learns those tastes, and the policy amplifies them. One visible result is sycophancy: telling users what they want to hear. Diverse annotators, clear guidelines, length controls, and adversarial evaluation can reduce this, but not eliminate it.

• Humans like agreement and length
• Reward model learns the bias
• Policy amplifies it

Researchers have developed variations. Reinforcement learning with verifiable rewards, or RLVR, replaces the learned reward model with a checker, such as unit tests for code or answer matching for maths, which is much harder to hack. GRPO, group relative policy optimisation, drops the value model: it samples a group of responses per prompt and uses each response's reward relative to the group average as its advantage. That removes one of the four models and cuts memory.

• RLVR: verifiable rewards
• GRPO: no value model
• Group-relative advantages

Let us recap. Preferences are easier to collect than ideal answers, so we train a reward model on pairwise comparisons using the Bradley-Terry loss, minus log sigmoid of the score gap. We then optimise the policy to maximise reward minus beta times KL to a reference, so it cannot wander into reward hacking territory. PPO does this stably with advantages from a value model and a clipped ratio, at the cost of four models in memory. And remember the failure modes: weak reward models, small KL, and biased annotators.

• Compare, don't write
• Bradley-Terry reward model
• Reward minus beta KL
• PPO: advantage + clipping

RLHF works, but it is heavy: a reward model, a value model, sampling loops, and finicky hyperparameters. Next lesson, we derive direct preference optimisation, which uses the same KL-regularised objective but solves it in closed form, so we can train straight from preference pairs with a simple loss.

• Next: DPO
• Skip the reward model and RL loop
• Subscribe for the series