Transformer overview and core idea

Summary

Transformers are a family of neural network architectures built around the multi-head attention mechanism. Input data (such as text, images, or audio) is converted into tokens, each token is mapped to a vector using an embedding lookup, and then—at every layer—tokens are contextualized by letting them attend to other tokens within the context window. This attention is computed in parallel across tokens and uses multiple attention heads so the model can learn different notions of relevance simultaneously. A core idea is that transformers do not rely on recurrence (unlike RNNs), so they can process all tokens in parallel, making training more efficient than sequential architectures. Because self-attention is permutation-invariant, transformers add positional information (typically via positional encodings or learned positional embeddings) so token order affects the output. The original 2017 transformer introduced an encoder–decoder design, and later variants are commonly grouped into encoder-only, decoder-only, and encoder–decoder models depending on whether they target representation learning, autoregressive generation, or conditional sequence-to-sequence tasks.

Key Points

• 1

  Transformers contextualize tokens using parallel multi-head self-attention rather than recurrence.

• 2

  Positional information is added because attention alone does not capture token order.

• 3

  Different transformer variants (encoder-only, decoder-only, encoder–decoder) support different learning and generation objectives.