Training & Learning

Direct Preference Optimization

DPO

A simpler alternative to RLHF that tunes a model on human preferences directly, skipping the separate reward model and reinforcement learning.

DPO reaches the same goal as RLHF, aligning a model with human preferences, but in one step instead of two. Rather than training a separate reward model and then running reinforcement learning against it, DPO optimizes the model directly on pairs of preferred and rejected responses. That makes it simpler, more stable, and cheaper in memory and compute, which is why it became a widely used alignment method, though RLHF can still have an edge on complex reasoning.