Loading your learn pack…
Read the summary, then start training!
Transformers are defined by multi-head attention plus positional information, enabling parallel contextualization of tokens without recurrence.
The transformer is a family of neural network architectures built around the multi-head attention mechanism. Inputs (e.g., text, images, or audio) are converted into token sequences, embedded into vectors, and then processed layer by layer. Within each layer, self-attention contextualizes every token by letting it attend to other tokens in the context window (typically using parallel multi-head attention). Because self-attention is permutation-invariant, transformers add positional information via positional encodings or learned positional embeddings so that token order affects the output. Transformers are commonly categorized into three main variants based on their structure and training/generation behavior: encoder-only, decoder-only, and encoder–decoder. Encoder-only transformers are optimized for representation learning and tasks like masked prediction (e.g., BERT). Decoder-only transformers are optimized for autoregressive generation (e.g., GPT). Encoder–decoder transformers support conditional sequence-to-sequence tasks (e.g., translation), using a decoder with causal (masked) self-attention plus cross-attention to incorporate encoder representations. The original architecture was introduced in the 2017 paper "Attention Is All You Need," and later variants and training techniques (such as pretraining + fine-tuning) have made transformers central to modern large language models and many multimodal applications.
Transformers are defined by multi-head attention plus positional information, enabling parallel contextualization of tokens without recurrence.
Main architecture variants are encoder-only, decoder-only, and encoder–decoder, each suited to different task types (representation learning, autoregressive generation, and conditional seq2seq).
The original 2017 transformer introduced the encoder–decoder design with attention-based parallel processing, and later work expanded training and architectural variants for large-scale models.
A family of neural network architectures that use multi-head attention to contextualize token sequences, typically with added positional information.
An attention mechanism that runs multiple attention heads in parallel, allowing the model to learn different notions of relevance and combine their outputs.
Attention where queries, keys, and values come from the same sequence, letting each token incorporate information from other tokens in the context window.
A method for injecting token order information into a transformer so that sequence order influences attention and outputs.
A transformer variant that uses only an encoder stack, commonly used for masked prediction and representation learning (e.g., BERT).
A transformer variant that uses only a decoder stack with causal (masked) self-attention, commonly used for autoregressive generation (e.g., GPT).
A transformer variant with both encoder and decoder stacks, where the decoder uses causal self-attention and cross-attention to condition generation on encoder representations.
A masking strategy in decoder self-attention that prevents a token from attending to future tokens, preserving autoregressive behavior.
“Can you explain what "Transformers are defined by multi-head attention plus positional information, enabling parallel contextualization of tokens without recurrence." means in simple terms?”