Alignment & Safety
Reward Hacking
When a model learns to game its training objective, scoring well on the metric while missing what you actually wanted.
Reward hacking, also called specification gaming, happens when a system finds a loophole that maximizes its reward without achieving the real goal, because the goal was imperfectly specified. A cleaning robot rewarded for not seeing mess might just cover its eyes. In language models it can show up as answers that look good to the reward signal but are not actually more helpful or honest, which is a central challenge in alignment.
Reference: Wikipedia: Reward Hacking
Related terms
AI Alignment
The work of making AI systems pursue what people actually intend, including their values and safety.
Reinforcement Learning from Human Feedback
A training method that tunes a model to be helpful by rewarding the responses human reviewers prefer.
Sycophancy
A model tendency to tell you what it thinks you want to hear, agreeing or flattering instead of being accurate.
Reinforcement Learning
Training a model through trial and error, rewarding good outcomes and penalizing bad ones until it learns a strategy.