Attention Mechanisms Learning Path

Definition of attention in machine learning

In machine learning, attention is a mechanism for deciding how important each element in an input sequence is relative to the others. In NLP, this importance is often represented as “soft” weights over words in a sentence, which are used to form a weighted combination of token embeddings. These weights are computed during the forward pass and can change with every input step. More generally, attention operates over a fixed-width sequence of token embeddings (from tens to millions of tokens) and allows each token to access information from any other token directly, rather than only through a previous hidden state as in older recurrent approaches. This capability helps models capture long-range dependencies and reduces biases toward later tokens that can occur in RNNs. Modern architectures, especially Transformers, rely heavily on (self-)attention: each element attends to all others, enabling parallel computation and global context modeling. Attention weights can also be interpreted as alignment or correlation patterns (e.g., which source words influence which target words in translation), and they are widely used across tasks in language, vision, and speech.

0/2 modes complete

Soft attention weights across token sequences

0/2 modes complete