Models & Architecture

Attention

A mechanism that lets a model focus on the most relevant parts of its input when producing each piece of output.

Attention lets a model decide, for every element it is processing, which other elements matter most and how much to weigh them. In a sentence, it is how the model connects a pronoun back to the noun it refers to, even when they are far apart. It is the core idea inside the transformer, and self-attention, where a sequence attends to itself, is what gives these models their grasp of context.

Reference: Wikipedia: Attention