Training & Learning
Direct Preference Optimization
DPOA simpler alternative to RLHF that tunes a model on human preferences directly, skipping the separate reward model and reinforcement learning.
DPO reaches the same goal as RLHF, aligning a model with human preferences, but in one step instead of two. Rather than training a separate reward model and then running reinforcement learning against it, DPO optimizes the model directly on pairs of preferred and rejected responses. That makes it simpler, more stable, and cheaper in memory and compute, which is why it became a widely used alignment method, though RLHF can still have an edge on complex reasoning.
Related terms
Reinforcement Learning from Human Feedback
A training method that tunes a model to be helpful by rewarding the responses human reviewers prefer.
Reinforcement Learning
Training a model through trial and error, rewarding good outcomes and penalizing bad ones until it learns a strategy.
AI Alignment
The work of making AI systems pursue what people actually intend, including their values and safety.
Fine-Tuning
Taking a pretrained model and training it further on a smaller, focused dataset to specialize it for a task or style.