Word Embeddings Learning Path

Development and history of word embeddings

Word embeddings are vector representations of words (and sometimes phrases or documents) used in NLP to capture meaning. The core idea comes from distributional semantics: words that appear in similar contexts tend to have similar meanings, often formalized as “a word is characterized by the company it keeps.” Early work included the vector space model for information retrieval, which produced sparse high-dimensional representations; later techniques like singular value decomposition led to latent semantic analysis and related approaches such as random indexing for building co-occurrence-based representations. Modern neural approaches accelerated progress. In 2000, Bengio and colleagues introduced neural probabilistic language models to learn distributed word representations, reducing dimensionality and enabling better generalization. Around the early 2000s, research also explored combining word and document embeddings (e.g., via kernel CCA) and different ways to define semantic spaces (co-occurring words vs. linguistic contexts). After roughly 2005, many methods shifted toward neural architectures, and in 2013 Google’s team led by Tomas Mikolov released word2vec, a toolkit that trained embeddings much faster and helped popularize embeddings for broad experimentation and practical use. A major historical limitation of early “static” embeddings is that a single word vector conflates multiple meanings (polysemy and homonymy). This motivated multi-sense embedding methods that split a word into multiple vectors, including both unsupervised and knowledge-based approaches. By the late 2010s, contextually meaningful embeddings such as ELMo and BERT addressed this by producing token-level embeddings that vary with context, better reflecting word sense in real usage. The development of embeddings also extended beyond general NLP into areas like bioinformatics (e.g., BioVectors for biological sequences), sentence/document embeddings, and applications in software ecosystems (e.g., word2vec, GloVe, fastText, Gensim, and BERT).

0/2 modes complete

Distributional semantics and semantic spaces

0/2 modes complete