What are RLHF, DPO and GRPO?
The three techniques by which models learn to behave after pretraining: from reinforcement with human feedback to direct preferences.
RLHF, DPO and GRPO are the three main acronyms of post-training: the phase where a model, after learning language from data, learns to behave as intended: helpful, honest, in the right format. RLHF (Reinforcement Learning from Human Feedback) is the historical technique behind ChatGPT: humans compare answers, a "reward model" learns their tastes, and the model is optimized via reinforcement learning to maximize them. DPO (Direct Preference Optimization) achieves a similar effect skipping the detour: it optimizes directly on preference pairs, with no reward model and no reinforcement loop. GRPO (Group Relative Policy Optimization), made famous by DeepSeek, samples several answers to the same prompt and scores each against the group average: above-average ones are reinforced, below-average ones discouraged, with no separately trained reward model. In reasoning models the reward is often an automatic verifier (the maths answer is right or wrong) rather than a human preference, which makes it a different lever from RLHF and DPO.
See also: Q-learning · The method by which an agent learns by trial and error which action pays off in each state, estimating each move's value.
How to choose among the three
In practice: DPO is the simple, cheap route when you have preference pairs ("this answer beats that one") and want to steer style and behavior. Classic RLHF gives more control but costs in complexity and training instability. GRPO shines where quality is automatically measurable (math, code, verifiable tasks), which is why it dominates the reasoning-model lineage.
Why you should care even if you never train models
Two practical reasons. First: the behavior of every model you use (why it refuses certain requests, why it answers in a certain format) is the product of these techniques, and understanding them helps explain the odd behaviors. Second: the same ideas are coming down in scale: aligning a small model to your company's preferences with DPO on a few thousand examples is now a feasible project, not research. The prerequisite, as always, is not the algorithm: it is quality examples and an evals suite to measure whether behavior improved. One variant that cuts the human labor is constitutional AI, where the preferences are written out as explicit principles and a model does the judging.
Related terms
- Q-learning · The method by which an agent learns by trial and error which action pays off in each state, estimating each move's value.
- Fine-tuning · Further training a pretrained model on your own examples: excellent for style, format and recurring tasks, wrong for facts that change daily.
- Reasoning models · Models that reason step by step before answering, spending more compute at inference time on complex problems.
- RLVR · RLVR is a reinforcement learning method in which the model's reward comes from an automatic verifier, such as a passed test or an exact answer, not a human.
- Sycophancy · Sycophancy is an AI model's tendency to agree with the user even when wrong, instead of correcting them.