LLM Alignment & Preference Optimization
Preference optimization in practice — DPO, SimPO, ORPO, CPO, KTO, reward modeling, and RLHF. What each method does, when it helps, and how to choose between them.
DPO: Direct Preference Optimization for LLM Alignment
How DPO aligns LLMs with human preferences without a reward model: the objective, the dataset format, and when pairs beat labels.
SimPO: Simple Preference Optimization Without Reference Models
SimPO aligns LLMs without a reference model: how the length-normalized objective works, what data it needs, and when to pick it over DPO.
ORPO: Odds Ratio Preference Optimization
ORPO merges SFT and preference alignment into one stage: the odds-ratio objective, the dataset format, and when single-stage wins.
CPO: Contrastive Preference Optimization for LLM Alignment
Understand CPO, a contrastive approach to preference optimization that keeps chosen response probabilities high while suppressing rejected responses.
KTO: Kahneman-Tversky Optimization for AI Alignment
Understand KTO, an alignment method based on prospect theory that works with single-response feedback (thumbs up/down) instead of paired preferences.
Reward Modeling for RLHF: Training Custom Reward Functions
Learn how to train reward models for RLHF. Understand the reward model pipeline, dataset preparation, evaluation metrics, and online RL methods coming to Run BiOS.
RLHF Methods: Offline Alignment vs Online Reinforcement Learning
Compare offline alignment methods (DPO, SimPO, ORPO, CPO, KTO) with online RL (PPO, GRPO, GKD). Understand when to use each and what is coming to Run BiOS.
Browse other topics