Learning Attention Mechanisms in Deep Learning

Definition of attention and soft weights

In machine learning, attention is a mechanism that determines the importance of each component in a sequence relative to other components. In natural language processing, this importance is expressed using “soft” weights over words, which are derived from correlations between token embeddings (vectors representing each position in a fixed-width sequence). These weights are used to form context representations as weighted combinations of token embeddings. Soft weights are computed during the forward pass and are typically continuous values (often produced via a softmax), so they can distribute probability mass across multiple tokens rather than selecting a single token. Because they are not fixed parameters learned once, but instead depend on the current input, the soft weights change at every step of processing. This contrasts with “hard” attention, which would make discrete selections (e.g., one weight set to 1 and others to 0) and is not computed in the same way during the forward pass.

0/2 modes complete

Token embeddings and sequence-wide attention

0/2 modes complete