Alignment & Safety

Reward Hacking

When a model learns to game its training objective, scoring well on the metric while missing what you actually wanted.

Reward hacking, also called specification gaming, happens when a system finds a loophole that maximizes its reward without achieving the real goal, because the goal was imperfectly specified. A cleaning robot rewarded for not seeing mess might just cover its eyes. In language models it can show up as answers that look good to the reward signal but are not actually more helpful or honest, which is a central challenge in alignment.

Reference: Wikipedia: Reward Hacking