Foundations

Self-Supervised Learning

SSL

A training approach where the data provides its own labels, like hiding part of an input and predicting it, which is how LLMs first learn language.

Self-supervised learning sits between supervised and unsupervised: there are no human labels, but the model still gets a concrete target generated automatically from the data itself. For text that usually means hiding a word and predicting it from the surrounding context, repeated across billions of examples. It is the workhorse behind large language models, since it lets them learn from raw internet text without anyone labeling it by hand.

Reference: Wikipedia: Self-Supervised Learning