Evaluation & Operations
Benchmark
A standardized test set used to measure and compare model performance on a specific capability.
A benchmark is a fixed dataset and scoring method that lets different models be compared on the same task, such as grade-school math, coding, or broad knowledge. Well-known ones became shorthand for capability and drive public leaderboards. Their weakness is that models can be tuned to the test or even trained on it, so a high score does not always translate to real-world usefulness, a problem called contamination.
Related terms
Evaluation
Measuring how good a model actually is, from accuracy on tests to human judgments of quality and safety, often called evals.
Leaderboard
A public ranking of models by their scores on benchmarks or head-to-head human preference votes.
Training Data
The examples a model learns from, whose size, quality, and bias largely determine how the model turns out.