Evaluation & Operations

Benchmark

A standardized test set used to measure and compare model performance on a specific capability.

A benchmark is a fixed dataset and scoring method that lets different models be compared on the same task, such as grade-school math, coding, or broad knowledge. Well-known ones became shorthand for capability and drive public leaderboards. Their weakness is that models can be tuned to the test or even trained on it, so a high score does not always translate to real-world usefulness, a problem called contamination.