Attention as token importance weighting

Summary

In attention-based models, “attention weights” act as token importance scores: each token in a sequence is assigned a soft weight that reflects how relevant it is to the current token’s representation. These weights are computed during the forward pass (not as discrete hard choices), and they determine how much each token’s embedding contributes when forming a context vector—typically as a weighted sum of other token representations. In the common scaled dot-product attention used in Transformers, similarity between query and key vectors produces raw scores, which are then normalized with a softmax to yield the attention weights. A scaling factor (e.g., dividing by \(\sqrt{d_k}\)) helps prevent high-variance scores that would otherwise cause the softmax to collapse toward a single token (similar to a hard max). As a result, attention weights provide a graded, interpretable distribution over tokens rather than selecting only one. Interpreting these weights is especially useful in sequence tasks like translation: high attention weight on particular source tokens indicates alignment to specific target tokens. Because multiple-to-multiple alignments can be necessary (e.g., phrases), soft attention weights are often more effective than hard attention that forces a single token to receive all probability mass.

Key Points

• 1

  Attention weights are soft token-importance scores computed in the forward pass and used to form context vectors as weighted sums of token embeddings.

• 2

  In scaled dot-product attention, query-key similarity scores are softmax-normalized; scaling (e.g., by \(\sqrt{d_k}\)) reduces variance so attention doesn’t collapse to a single token.

• 3

  Attention weights can be interpreted as alignment signals in tasks like machine translation, supporting both one-to-one and multi-to-multi relationships.