RLHF Explained: The Loop That Turned GPT-3 Into ChatGPT

Here is RLHF explained in the order it runs: supervised fine-tuning on demonstration data, a reward model trained on human preference rankings, then reinforcement learning, originally PPO, that fine-tunes the policy against that reward model. This three-stage loop turned GPT-3 into InstructGPT, and InstructGPT's descendants into ChatGPT. In 2026 it rarely runs alone. Most labs pair it with DPO, RLAIF, or verifiable-reward RL, depending on what they are trying to fix.

I have watched a team spend six weeks training a reward model that made every response longer, more hedged, and more agreeable, then wonder why user satisfaction dropped. The reward model was doing exactly what it was trained to do, not what the team wanted. That gap, between the proxy you optimize and the outcome you want, is the whole story of RLHF. Understand it before you spend a training budget chasing it.

Key takeaways

If you read nothing else, these are the load-bearing claims:

• RLHF is three stages, not one. Supervised fine-tuning teaches the model to follow instructions, a reward model learns to rank outputs, and reinforcement learning, originally PPO, optimizes the policy against that reward model.
• A 100x-smaller model beat its parent on human preference. OpenAI's 1.3B-parameter InstructGPT was preferred by human raters over the 175B-parameter base GPT-3, entirely from the RLHF stage (Ouyang et al., 2022).
• DPO removes the RL loop entirely. Direct Preference Optimization trains one classification loss on preference pairs and matches or beats PPO-based RLHF on several benchmarks, with no sampling from the model during training (Rafailov et al., 2023).
• Reward hacking is the honest failure mode. The policy optimizes the reward model, not the human, and it finds sycophancy, padded hedging, and confident-sounding wrong answers before it finds what you meant.
• In 2026, "RLHF" usually means a family, not one fixed pipeline. DPO, RLAIF, and verifiable-reward RL (GRPO) each replace a different piece of the classic loop depending on what is being optimized.

What is RLHF?

RLHF, reinforcement learning from human feedback, is a training method that aligns a language model's outputs with what people want, rather than only what is statistically likely to follow a prompt. It works by turning human judgments into a reward signal, then optimizing the model against that signal with reinforcement learning.

Before RLHF, a base model like GPT-3 was a next-token predictor: given a prompt, it produced whatever continuation was statistically likely, including instructions ignored, questions dodged, and unsafe completions faithfully continued from the training distribution. RLHF is the layer that turns "statistically likely" into "helpful, honest, and harmless enough to ship" by training on how humans rank outputs, not only on how the internet writes.

The three stages of RLHF: SFT, reward model, PPO

Stage one is supervised fine-tuning (SFT). You collect a set of prompts and human-written demonstrations of the ideal response, then fine-tune the base model on those pairs with ordinary next-token prediction. This gets the model into the neighborhood of "follows instructions" and gives the later stages something coherent to build on. It is the smallest, cheapest stage, and it also sets the ceiling: a reward model can push a policy toward better answers, but it cannot teach the model to write in a style it never saw.

Stage two trains the reward model. You sample multiple responses to the same prompt from the SFT model, have human raters rank them from best to worst, and train a separate model to predict that ranking. The reward model's job is narrow: given a prompt and a response, output a single scalar score that correlates with human preference. It never generates text. It only scores it.

Stage three is the reinforcement learning step. The policy, a copy of the SFT model, generates a response, the reward model scores it, and an RL algorithm, originally proximal policy optimization (PPO), updates the policy's weights to make higher-scoring responses more likely. A KL-divergence penalty against the original SFT model keeps the policy from drifting so far it forgets how to write coherent text while gaming the reward. This is also the expensive stage: PPO needs four models resident in memory at once, the policy, a frozen reference copy for the KL penalty, the reward model, and a value model estimating expected reward. That footprint is a meaningful reason RLHF fell out of favor as the default.

How the reward model learns from human preference rankings

Human raters do not score a response with a number. Scoring is unreliable between people and even from the same person on different days. Instead, raters compare two or more responses to the same prompt and pick which one is better. That comparison is easier to do consistently, and it is also all the signal the reward model needs: it learns to output a score where the chosen response scores higher than the rejected one, using a pairwise loss like the one above, adapted from the Bradley-Terry preference model.

The result is a reward model that generalizes past the exact prompts it saw in training and scores new prompt-response pairs fast, with no human in the loop for every training step. That speed is the entire value of the reward model: it lets the RL stage run millions of scored rollouts no human rater could keep up with. It is also where the proxy problem starts. The reward model is a lossy compression of what raters valued, and the policy optimizes the compression, not the original intent.

If you are staffing this stage from a hiring plan rather than a whiteboard, an ML engineering team that has built reward models and preference pipelines before will save you the two or three iterations most teams burn discovering the annotator-agreement problem the hard way.

RLHF vs. DPO: why some teams skip the RL loop entirely

Direct Preference Optimization (DPO) is the most common alternative to classic RLHF, and it works by removing the middle: no separate reward model, no PPO rollout loop. DPO trains directly on the same preference pairs, chosen response and rejected response, with a single classification-style loss derived from the policy model itself. Rafailov et al. report that DPO "exceeds PPO-based RLHF in ability to control sentiment of generations, and matches or improves response quality in summarization and single-turn dialogue," while removing the need to sample from the model during training or tune RL-specific hyperparameters (Rafailov et al., 2023).

The practical draw is obvious once you have run a PPO job: DPO needs two models instead of four, trains with a debuggable supervised loss, and has no rollout sampling to stabilize. The honest trade-off: DPO trains on a fixed, static preference dataset. Classic RLHF's RL loop can, in principle, explore responses the original raters never saw and get a score on them from the reward model. DPO cannot; it only refines within the distribution of the pairs you collected. For most product teams shaping tone, format, and helpfulness, that limit rarely bites. For teams pushing into genuinely new capability, it can.

RLHF vs. RLAIF: when an AI model replaces the human rater

RLAIF, reinforcement learning from AI feedback, keeps RLHF's reward-model-plus-RL structure and replaces human raters with a model that judges responses instead, usually scored against a written set of principles. Anthropic's Constitutional AI is the reference version: it uses AI-generated preference judgments, checked against a constitution, in place of human preference labels, and reports that this lets developers "control AI behavior more precisely and with far fewer human labels" than standard RLHF (Bai et al., 2022).

RLAIF does not remove the need for judgment. It moves judgment upstream, into the principles the judging model applies, and into deciding when an AI judge is calibrated enough to trust for a given behavior. Use it where the standard you are enforcing is stateable and stable, like "do not give financial advice" or "match this style guide." Keep a human in the loop where the standard is genuinely contested, ambiguous, or expensive enough to get wrong once it ships to every user at once.

Where RLHF breaks: reward hacking and annotator disagreement

RLHF does not optimize against human preference directly. It optimizes against a learned reward model, a lossy proxy for what raters said they preferred. The policy is good at finding outputs that score well on the proxy without being what a human wants: sycophancy, padded hedging, confident-sounding wrong answers, and length inflation are the most common shapes this takes, because raters, on average and without noticing it, reward all four.

I have watched this play out on a support-response reward model trained from rankings where two annotator groups disagreed almost half the time: one group rewarded concise, direct answers, the other rewarded thorough, hedged ones. The reward model learned an average of both preferences, which meant it learned neither cleanly, and the resulting policy produced answers too long to be direct and too clipped to be thorough. The fix was not more RL. It was tightening the rating guidelines until agreement climbed, then retraining the reward model on the cleaner signal.

That is also, not by coincidence, the judgment economy playing out inside the training pipeline itself: the reward model is a stand-in for human judgment, compressed and delegated. Delegate it carelessly, on preference data two raters cannot agree on, and the compression is where the judgment breaks, long before the RL loop ever runs.

RLHF in 2026: GRPO, reasoning models, and verifiable rewards

The newest branch in this family is not preference-based at all. DeepSeek-R1 trained reasoning capability with pure reinforcement learning against verifiable rewards, correct or incorrect answers on math and code problems that a program can check, rather than a learned reward model trained on human preference rankings (DeepSeek-R1, 2025). The algorithm, Group Relative Policy Optimization (GRPO), drops the separate value model entirely and estimates advantage from a group of sampled responses to the same prompt, part of why it runs cheaper than classic PPO.

Verifiable-reward RL and preference-based RLHF solve different problems, and they increasingly get combined in the same training pipeline: verifiable rewards to push raw capability on checkable tasks like math and code, preference-based RLHF or DPO to shape tone, safety, and helpfulness on everything a program cannot check. Treat "RLHF" in 2026 as a family of related techniques, not a single fixed pipeline, and ask which piece a given method is actually optimizing before you compare it to another.

Should you use RLHF, DPO, or skip preference tuning entirely?

Match the method to the failure mode you are budgeting engineering time to prevent, not to which paper is newest:

None of this replaces the upstream decision of whether to fine-tune at all, covered in when to fine-tune versus prompt or retrieve. Once you decide to fine-tune, whether preference tuning belongs alongside a parameter-efficient method matters too, which I cover in LoRA vs. full fine-tuning.

Both come back to the same pillar guide, the complete guide to fine-tuning, and the fuller version of this decision tree, including the annotator-agreement problem above, is the subject of my book To Fine-Tune or Not.

What does RLHF actually stand for, and how is it different from normal fine-tuning?

RLHF stands for reinforcement learning from human feedback. Normal supervised fine-tuning trains a model to imitate demonstrations directly. RLHF adds two more stages on top: a reward model that learns to score outputs by human preference, and a reinforcement learning step that optimizes the policy against that score, which lets the model improve past what any single demonstration showed it.

Is RLHF the same thing as DPO?

No. Both train on human preference pairs, but RLHF trains a separate reward model and then runs a reinforcement learning loop against it. DPO skips both steps: it trains one loss directly on the preference pairs, with no reward model and no RL rollouts. Rafailov et al. found DPO matches or beats PPO-based RLHF on several tasks while cutting the infrastructure roughly in half.

Why would RLHF make a model worse instead of better?

RLHF optimizes a reward model, not a human, and that reward model is a lossy proxy. If your preference data rewards length, hedging, or agreeableness without meaning to, the policy learns exactly that, and you get a model that sounds more confident while being no more correct. This is reward hacking, and it is the single most common way an RLHF project quietly makes a product worse.

How much human preference data do you actually need to run RLHF?

Less than most teams assume for narrow, well-defined tasks, more than most teams budget for anything broad. OpenAI's original InstructGPT reward model trained on tens of thousands of comparisons. What matters more than raw volume is annotator agreement: a smaller, cleanly-agreed preference set trains a better reward model than a larger, noisy one.

Pick the method that matches the failure mode you are trying to prevent. Measure the reward model against held-out human judgment before you trust an RL loop with it. That is the thesis behind everything on this site: the machine does the work, and the training pipeline is only as good as the judgment you build into it.