Definition of attention and soft weights

Summary

In machine learning, attention is a mechanism that determines the importance of each component in a sequence relative to other components. In natural language processing, this importance is expressed using “soft” weights over words, which are derived from correlations between token embeddings (vectors representing each position in a fixed-width sequence). These weights are used to form context representations as weighted combinations of token embeddings. Soft weights are computed during the forward pass and are typically continuous values (often produced via a softmax), so they can distribute probability mass across multiple tokens rather than selecting a single token. Because they are not fixed parameters learned once, but instead depend on the current input, the soft weights change at every step of processing. This contrasts with “hard” attention, which would make discrete selections (e.g., one weight set to 1 and others to 0) and is not computed in the same way during the forward pass.

Key Points

• 1

  Attention assigns importance to tokens in a sequence relative to other tokens.

• 2

  Soft weights are continuous, input-dependent values computed in the forward pass.

• 3

  Soft attention forms context vectors as weighted sums of token embeddings, allowing multiple tokens to contribute.

• 4

  Soft attention differs from hard attention, which makes discrete single-token selections.