Learning Attention Mechanisms in Deep Learning

Attention as token importance weighting

In attention-based models, “attention weights” act as token importance scores: each token in a sequence is assigned a soft weight that reflects how relevant it is to the current token’s representation. These weights are computed during the forward pass (not as discrete hard choices), and they determine how much each token’s embedding contributes when forming a context vector—typically as a weighted sum of other token representations. In the common scaled dot-product attention used in Transformers, similarity between query and key vectors produces raw scores, which are then normalized with a softmax to yield the attention weights. A scaling factor (e.g., dividing by \(\sqrt{d_k}\)) helps prevent high-variance scores that would otherwise cause the softmax to collapse toward a single token (similar to a hard max). As a result, attention weights provide a graded, interpretable distribution over tokens rather than selecting only one. Interpreting these weights is especially useful in sequence tasks like translation: high attention weight on particular source tokens indicates alignment to specific target tokens. Because multiple-to-multiple alignments can be necessary (e.g., phrases), soft attention weights are often more effective than hard attention that forces a single token to receive all probability mass.

0/2 modes complete

Soft versus hard attention weights

0/2 modes complete