Definition and purpose of language models

Summary

A language model is a computational model that predicts sequences in natural language. Its core purpose is to estimate what text (or the next token/word) is likely to come next given prior context, enabling systems to generate, transform, and interpret language. Because of this predictive capability, language models are used across many applications such as speech recognition, machine translation, natural language generation, optical character recognition, handwriting recognition, grammar induction, information retrieval, and even disaster response. Modern language models—especially large language models (LLMs)—are typically built using transformer architectures trained on very large datasets (often including large amounts of internet text). Earlier approaches included purely statistical models such as word n-gram models, which predict the next word using a fixed-size window of previous words. Over time, neural approaches introduced continuous word representations (embeddings) to address issues like data sparsity and high-dimensionality, and transformers later became the dominant architecture for state-of-the-art performance.

Key Points

• 1

  A language model predicts sequences in natural language, typically estimating the probability of the next word/token given previous context.

• 2

  Language models support many tasks including speech recognition, translation, text generation, OCR/handwriting recognition, grammar induction, and information retrieval.

• 3

  Earlier statistical methods (e.g., word n-gram models) used fixed context windows, while neural methods introduced embeddings and later transformers, leading to modern LLMs.