Word Embeddings Learning Path

Development and history of word embeddings

Word embeddings are vector representations of words (and sometimes phrases or documents) used in natural language processing to capture meaning. Typically, each word is mapped to a real-valued vector such that words with similar usage patterns appear close in vector space. This idea is grounded in distributional semantics—famously summarized as “a word is characterized by the company it keeps”—and builds on earlier distributional and vector-space approaches from information retrieval. Historically, early semantic space models used sparse, high-dimensional word co-occurrence vectors, which motivated dimensionality reduction methods such as singular value decomposition and latent semantic analysis (late 1980s), along with approaches like random indexing for collecting co-occurrence contexts. Around 2000, neural probabilistic language models (e.g., Bengio and colleagues) introduced learning distributed word representations to reduce dimensionality in a data-driven way. After foundational neural work by Bengio and colleagues, most techniques from roughly 2005 onward increasingly relied on neural architectures. A major milestone was word2vec (2013, by Mikolov and Google), which enabled much faster training and helped popularize embeddings for broad experimentation and practical NLP. A key limitation of early “static” embeddings is that a single word vector conflates multiple senses (polysemy and homonymy). This led to multi-sense and context-aware developments, including multi-sense skip-gram variants and, later, token-level contextual embeddings such as ELMo and BERT, where each word occurrence gets its own embedding based on context. Embeddings have also expanded beyond text to domains like biological sequences (BioVectors) and have been used in tasks such as sentence representation (e.g., skip-thought vectors, Sentence-BERT). Finally, ethical research shows that embeddings can encode and amplify biases present in training data, motivating debiasing and careful oversight.

0/2 modes complete

Polysemy and homonymy in static embeddings

0/2 modes complete