Transformer Learning Path

Transformer family overview and variants

The transformer is a family of neural network architectures built around the multi-head attention mechanism. Inputs (e.g., text, images, or audio) are converted into token sequences, embedded into vectors, and then processed layer by layer. Within each layer, self-attention contextualizes every token by letting it attend to other tokens in the context window (typically using parallel multi-head attention). Because self-attention is permutation-invariant, transformers add positional information via positional encodings or learned positional embeddings so that token order affects the output. Transformers are commonly categorized into three main variants based on their structure and training/generation behavior: encoder-only, decoder-only, and encoder–decoder. Encoder-only transformers are optimized for representation learning and tasks like masked prediction (e.g., BERT). Decoder-only transformers are optimized for autoregressive generation (e.g., GPT). Encoder–decoder transformers support conditional sequence-to-sequence tasks (e.g., translation), using a decoder with causal (masked) self-attention plus cross-attention to incorporate encoder representations. The original architecture was introduced in the 2017 paper "Attention Is All You Need," and later variants and training techniques (such as pretraining + fine-tuning) have made transformers central to modern large language models and many multimodal applications.

0/2 modes complete

Tokenization and embedding

0/2 modes complete