Models & Architecture
Attention
A mechanism that lets a model focus on the most relevant parts of its input when producing each piece of output.
Attention lets a model decide, for every element it is processing, which other elements matter most and how much to weigh them. In a sentence, it is how the model connects a pronoun back to the noun it refers to, even when they are far apart. It is the core idea inside the transformer, and self-attention, where a sequence attends to itself, is what gives these models their grasp of context.
Reference: Wikipedia: Attention
Related terms
Transformer
The neural network design behind almost every modern language model, built around a mechanism called attention.
Context Window
The maximum amount of text, measured in tokens, that a model can consider at once.
Large Language Model
A model trained on massive amounts of text that predicts and generates language, powering tools like ChatGPT and Claude.