Definition of attention in machine learning

Summary

In machine learning, attention is a mechanism for deciding how important each element in an input sequence is relative to the others. In NLP, this importance is often represented as “soft” weights over words in a sentence, which are used to form a weighted combination of token embeddings. These weights are computed during the forward pass and can change with every input step. More generally, attention operates over a fixed-width sequence of token embeddings (from tens to millions of tokens) and allows each token to access information from any other token directly, rather than only through a previous hidden state as in older recurrent approaches. This capability helps models capture long-range dependencies and reduces biases toward later tokens that can occur in RNNs. Modern architectures, especially Transformers, rely heavily on (self-)attention: each element attends to all others, enabling parallel computation and global context modeling. Attention weights can also be interpreted as alignment or correlation patterns (e.g., which source words influence which target words in translation), and they are widely used across tasks in language, vision, and speech.

Key Points

• 1

  Attention assigns importance weights to elements in a sequence relative to one another.

• 2

  “Soft” attention weights are computed in the forward pass and determine weighted combinations of token embeddings.

• 3

  Self-attention lets each token directly attend to all other tokens, enabling global dependency modeling and parallel computation.

• 4

  Attention weights can be interpreted as alignment/correlation patterns and are used for interpretability (e.g., attention maps).