Loading your learn pack…
Read the summary, then start training!
Information theory quantifies information using probability, with information interpreted as the resolution or reduction of uncertainty.
Information theory is the mathematical study of how information is quantified, stored, compressed, transmitted, and recovered in the presence of uncertainty and noise. Claude Shannon established the field formally in the 1940s, building on earlier work by Harry Nyquist and Ralph Hartley. The central idea is that information can be understood as the reduction of uncertainty: an unlikely event conveys more information than a likely one. Information is commonly measured in bits using base-2 logarithms; for example, learning the result of a fair coin toss provides one bit of information. Entropy is the foundational measure of uncertainty in a random variable and represents the average amount of information produced by its possible outcomes. Related concepts include joint entropy, conditional entropy, mutual information, and Kullback–Leibler divergence, which describe combined uncertainty, remaining uncertainty after observation, shared information, and the difference between probability distributions. Shannon's source-coding and noisy-channel coding theorems establish fundamental limits for data compression and reliable communication, while channel capacity specifies the maximum achievable information rate over a noisy channel. These foundations support applications such as compression, error correction, cryptography, statistics, signal processing, artificial intelligence, and communications engineering.
Information theory quantifies information using probability, with information interpreted as the resolution or reduction of uncertainty.
Self-information is defined as -log2(p), so rarer events carry greater surprisal; the unit based on logarithm base 2 is the bit or shannon.
Entropy is the expected surprisal of a random variable and is maximized when all possible outcomes are equally probable.
Conditional entropy measures remaining uncertainty after observing another variable, while mutual information measures the information shared between variables.
Kullback–Leibler divergence compares probability distributions and measures the additional information or coding cost caused by using an incorrect distribution.
Shannon's source-coding and noisy-channel coding theorems define fundamental limits for compression and reliable communication over noisy channels.
Information theory originated in telecommunications but now informs fields including computer science, physics, biology, neuroscience, cryptography, and artificial intelligence.
A quantitative measure of the reduction of uncertainty about the outcome of a random event.
The information or surprisal associated with an event of probability p, defined as -log2(p) bits.
The expected self-information of a random variable and a measure of its average uncertainty.
The standard unit of information based on the binary logarithm, representing the information in one fair binary choice.
The entropy of two or more random variables considered together.
The average uncertainty remaining about one random variable after another variable is known.
A measure of how much knowing one random variable reduces uncertainty about another.
A measure of the difference between a true probability distribution and an assumed distribution, often interpreted as excess coding cost or information gain.
The process of compressing data to represent it efficiently with as few symbols or bits as possible.
The use of structured redundancy to detect and correct errors introduced during communication over a noisy channel.
The maximum rate at which information can be transmitted reliably through a communication channel.
Shannon's result stating that reliable communication is possible below channel capacity and generally impossible above it.
“Can you explain what "Information theory quantifies information using probability, with information interpreted as the resolution or reduction of uncertainty." means in simple terms?”