Reinforcement Learning from Human Feedback

Chapter 22 turned a base model into an instruction-follower by imitation: we showed it good demonstrations and trained it to copy them. This works remarkably well, but it has a fundamental ceiling. This chapter is about breaking through that ceiling with a different kind of training signal — human PREFERENCES rather than demonstrations.

Supervised fine-tuning has two deep limitations that no amount of additional demonstrations can fully fix:

Here is the insight that launched RLHF. For many qualities we care about — helpfulness, tone, harmlessness, nuance — it is HARD for a human to write the ideal response, but EASY to compare two responses and say which is better. Writing a perfect, balanced answer to a sensitive question is difficult; judging which of two answers is more balanced is quick and reliable.

Almost anyone can instantly see that the left response is more helpful and empathetic than the right one — but writing the left response from scratch is a real skill. RLHF exploits this asymmetry: it collects easy human JUDGMENTS (which is better?) at scale, and uses them to push the model toward responses humans prefer, beyond what demonstrations alone could teach.

RLHF, as introduced by InstructGPT (Ouyang et al., 2022) and used to build ChatGPT, Claude, and others, is a three-stage pipeline. Before diving into the math of each stage, it helps to see the whole shape, so you always know where you are.

RLHF involves several models with confusingly similar names. Let us name them all up front so the later sections are clear:

The fuel for RLHF is preference data. The process is straightforward: take a prompt, have the model generate two (or more) responses, and ask a human to pick the better one. Each judgment produces a preference pair — a 'chosen' response and a 'rejected' response for the same prompt. This is the same structure you saw in Chapter 22's prefPair visual, now the central training signal.

Why compare two responses instead of just rating each on a scale of 1–10? Because humans are far more consistent at relative judgments than absolute ones. My '7/10' and your '7/10' may mean different things, and my own ratings drift over a long session. But 'A is better than B' is stable and consistent across annotators. Pairwise comparison sidesteps the calibration problem of absolute scores.

Preference data is expensive and its quality is paramount. Annotators must be trained, given clear guidelines (what counts as 'better'?), and monitored for agreement. Disagreement is inevitable — reasonable people prefer different responses — so the data is inherently noisy. The guidelines encode the values the model will learn: if annotators are told to prefer concise answers, the model becomes concise. The preference data IS the specification of desired behaviour.

We cannot run reinforcement learning against a human — humans are far too slow to provide feedback on every one of the millions of responses RL generates. So we train a REWARD MODEL: a model that LEARNS to predict human preferences, then provides instant reward signals during RL. The reward model is the bridge from slow, expensive human judgments to fast, scalable training signal.

A reward model takes a prompt and a response and outputs a single number — a scalar reward — representing how much a human would prefer that response. It is usually built by taking a pretrained model (often the SFT model), removing its token-prediction head, and adding a small head that outputs one number instead of a vocabulary distribution.

How do we train a model to output rewards when our data is only PAIRWISE preferences (A is better than B), not absolute scores? The answer is the Bradley-Terry model, a classic statistical model of pairwise comparisons. It says: the probability that response A is preferred over B is the logistic function of the DIFFERENCE in their rewards.

This gives us a training objective. For each preference pair (chosen y_w, rejected y_l), we want the reward model to assign a HIGHER reward to the chosen response. We maximize the probability of the observed preference, which means minimizing the following loss:

Now we have a reward model that scores responses. The RL stage uses that reward to improve the policy. But many readers have never studied reinforcement learning, so this section builds up just enough RL from scratch to understand RLHF. We will keep it concrete and tied to language models throughout.

In RL terms, generating a response is a sequence of DECISIONS. At each step, the model is in a STATE (the prompt plus the tokens generated so far), takes an ACTION (choosing the next token), and eventually receives a REWARD (the reward model's score of the finished response). The model's strategy for choosing actions is called its POLICY — which is exactly the language model's probability distribution over next tokens.

The objective of RL is simple to state: adjust the policy so that the responses it generates get high reward, ON AVERAGE. Formally, we want to maximize the expected reward over the responses the policy produces:

How do we increase expected reward by gradient descent? The trick — the policy gradient theorem — gives a beautifully intuitive answer. To make high-reward responses more likely, increase the probability of the actions that led to them, weighted by how much reward they earned. The simplest version is the REINFORCE algorithm:

Plain REINFORCE works but is extremely noisy. The reward r(y) varies wildly from sample to sample, so the gradient estimate jumps around, making training slow and unstable. The standard fix is a BASELINE: instead of weighting by the raw reward, weight by how much BETTER than average a response was. This 'advantage' — reward minus a baseline — has much lower variance and is the key idea connecting REINFORCE to PPO.

REINFORCE with a baseline is the conceptual foundation, but the algorithm actually used in classic RLHF is PPO (Proximal Policy Optimization; Schulman et al., 2017). PPO adds two crucial ingredients that make policy-gradient RL stable enough to train language models: a learned value model for the baseline, and a CLIPPED objective that prevents the policy from changing too fast.

PPO learns the baseline with a VALUE MODEL — a network (often sharing the body of the policy or reward model) that predicts the expected reward from a given state. The advantage is then the actual reward minus the value model's prediction: how much better the outcome was than expected. The value model is trained alongside the policy to predict rewards accurately.

The danger in policy-gradient RL is taking too large a step — a big update can collapse the policy into producing garbage, from which it never recovers. PPO prevents this by CLIPPING: it limits how much the policy's probability for an action can change in a single update. If an update would change a token's probability by more than a small factor, the change is clipped.

The min-of-two-terms looks cryptic but is doing something simple: it takes the more pessimistic (smaller) of the unclipped and clipped objectives, which removes the incentive to push the policy ratio far beyond 1±ε. The effect is that PPO improves the policy in small, safe steps — 'proximal' means it stays close to the previous policy each update.

There is a serious danger lurking in RLHF, and the KL penalty is the defense against it. Recall from Section 23.4 that the reward model is an imperfect proxy for human preferences. If we optimize against it with no constraint, the policy will discover responses that score HIGH reward but are actually BAD — it exploits the reward model's flaws. This is called REWARD HACKING, and it is the central failure mode of RLHF.

Imagine the reward model has a quirk: it slightly over-rewards responses that include the word 'certainly' or that are very long. With unconstrained optimization, the policy will discover this and produce responses stuffed with 'certainly' or padded to absurd length — high reward, terrible quality. The policy is optimizing the PROXY (reward model) instead of the true GOAL (human preference). This is Goodhart's law again (Chapter 21): when the reward becomes the target, it stops measuring quality.

The KL penalty constrains the policy to stay CLOSE to the original SFT model (the frozen reference). It adds a penalty proportional to the KL divergence (Chapter 4) between the current policy and the reference — a measure of how much the policy's distribution has drifted. The policy is rewarded for high reward-model score, but PENALIZED for straying far from the sensible SFT model.

In practice the KL term is folded into the per-token reward: each token gets the reward-model score (at the end) minus a per-token penalty for the policy assigning that token a much higher probability than the reference does. This keeps the policy anchored to fluent, sensible language while still letting it improve toward higher reward.

We can now assemble the complete RLHF loop. This is where the four models from Section 23.2 — policy, reference, reward, value — all come together. Seeing them interact in one place makes the whole pipeline concrete.

RLHF is powerful but notoriously difficult. It combines the instability of reinforcement learning with the imperfection of a learned reward model and the complexity of juggling four models. Understanding the failure modes is essential — they explain why so much of the craft of alignment is about taming RLHF.

A characteristic RLHF phenomenon: as you optimize against the reward model, the reward-model SCORE keeps rising, but the TRUE quality (measured by held-out humans) rises, peaks, and then FALLS. Beyond the peak, the policy is exploiting the reward model rather than improving — over-optimization. The art is to stop near the peak, before the proxy and the goal diverge. This 'Goodhart curve' is one of the most important empirical findings in RLHF (Gao et al., 2022).

GRPO (Group Relative Policy Optimization), introduced by DeepSeek (Shao et al., 2024) and central to the DeepSeek-R1 reasoning models, is a streamlined alternative to PPO that has become very popular — especially for training reasoning models. Its key idea is elegant: eliminate the value model entirely by computing the baseline from a GROUP of sampled responses to the same prompt.

Recall that PPO needs a value model to estimate the baseline for advantages (Section 23.6). This value model doubles a chunk of the memory and adds training complexity — it must be learned accurately, or the advantages are wrong. GRPO asks: what if we could get a good baseline WITHOUT a separate model? Its answer: sample several responses per prompt, and use their AVERAGE reward as the baseline.

For each prompt, GRPO samples a GROUP of G responses (say, 8 or 16). It scores all of them with the reward model, then computes each response's advantage as its reward relative to the group — normalized by subtracting the group mean and dividing by the group standard deviation. A response that beats its group-mates gets positive advantage; one that loses gets negative. No value model needed.

GRPO then uses the same clipped PPO-style objective from Section 23.6, but with the group-relative advantage and a KL penalty to the reference model included directly. Because the advantage is the same for all tokens in a response, the algorithm is simpler than PPO's per-token value estimation.

Beyond the core algorithms, getting RLHF to work in practice involves a set of variants and practical lessons. This section collects the most important ones.

Before reaching for full PPO, it is worth knowing the simplest preference-based method: rejection sampling (or best-of-N). Generate N responses per prompt, score them with the reward model, keep the best one, and add it to an SFT dataset. Then fine-tune on these best responses. This 'distills' the reward model's preferences into the policy via plain SFT — no RL machinery at all. It is surprisingly effective and is often used as a first step or a strong baseline.

Today, RLHF (or a preference-optimization stand-in like DPO) is a standard stage in building frontier assistants. The typical modern recipe is: pretrain (Part IV), SFT (Chapter 22), then preference optimization — increasingly DPO or GRPO rather than classic PPO — then safety tuning (Chapter 26). RLHF was the breakthrough that made models genuinely helpful and aligned; its descendants continue to do that work with less fragility.

Exercises 1–11 are pen-and-paper or derivations; 12–22 require code.

Further reading: “Training language models to follow instructions with human feedback” (Ouyang et al., 2022, InstructGPT) — the canonical RLHF paper. “Deep Reinforcement Learning from Human Preferences” (Christiano et al., 2017) for the original idea. “Proximal Policy Optimization Algorithms” (Schulman et al., 2017) for PPO. “DeepSeekMath” (Shao et al., 2024) and the DeepSeek-R1 report for GRPO. “Scaling Laws for Reward Model Overoptimization” (Gao et al., 2022) for the Goodhart curve. “Constitutional AI” (Bai et al., 2022) for RLAIF. The Hugging Face TRL library for PPO, GRPO, and DPO implementations.

Next → Chapter 24: Direct Preference Optimization

RLHF works, but you have now seen how hard it is: a reward model, a value model, a delicate RL loop, reward hacking, and a tug-of-war over the KL coefficient. Chapter 24 introduces a remarkable simplification — Direct Preference Optimization (DPO) — which achieves the goal of preference alignment WITHOUT a separate reward model and WITHOUT reinforcement learning. DPO shows, through an elegant derivation, that the RLHF objective can be reframed as a simple classification loss on preference pairs, trainable with ordinary supervised learning. We will derive it from the RLHF objective of this chapter, meet its variants (IPO, KTO, ORPO), and understand when to choose DPO over the RL methods you just learned.