Models & Architecture

Transformer

The neural network design behind almost every modern language model, built around a mechanism called attention.

The transformer is the architecture introduced in 2017 that made todays large language models possible. Its key trick is attention, which lets the model weigh how much every word in a sequence relates to every other word, in parallel, instead of reading strictly left to right. That parallelism made it practical to train on far more data than earlier designs, and it now underpins text, image, and audio models alike.

Reference: Wikipedia: Transformer