Dashboard

Now you are in the subtree of Lecture Notes public knowledge tree. 

[SCI] Information Theory

Information Theory (Shannon, 1948) is the mathematical theory of communication, defining entropy as a measure of uncertainty, and proving fundamental limits on data compression and error-free communication.

Overview

Claude Shannon's 1948 paper "A Mathematical Theory of Communication" introduced the bit as the unit of information, defined information entropy H = −Σ pᵢ log₂ pᵢ, and proved two theorems: (1) data can be compressed to at most H bits/symbol (source coding theorem); (2) any noisy channel has a capacity C such that reliable communication is possible at all rates below C (channel coding theorem). Both theorems are tight. Shannon's entropy is mathematically identical to Boltzmann's thermodynamic entropy — a deep connection Shannon acknowledged.

Information theory underpins all of data compression (ZIP, MP3, JPEG), error-correcting codes (CDs, space probes, 5G), cryptography, and statistical machine learning.

Key Figures & Recognition

  • Claude Shannon (1916–2001): Father of information theory. Turing Award 1966, Kyoto Prize 1985, National Medal of Science. No Nobel (not physics or chemistry).
  • Norbert Wiener (1894–1964): Cybernetics, 1948 (related but distinct work on feedback).

Seminal Papers

What This Enables

  • [TECH] Digital Computing — Shannon's channel capacity and error-correcting codes (Hamming, 1950) provide the theoretical foundation for digital communication.
  • [SCI] Machine Learning Theory — Mutual information, entropy, maximum likelihood, and the information bottleneck are core ML concepts from information theory.
  • [SCI] Genomics & Computational Biology — DNA is an information-storage medium; sequence alignment, compression, and variant calling use information-theoretic methods.

Discovery Character

Surprise level: Extreme — Shannon proved that there is a hard mathematical limit (channel capacity) to reliable communication, and that it can be approached arbitrarily closely with the right coding — regardless of what the channel looks like. Nobody expected information to have a rigorous mathematical definition, let alone hard quantitative limits. The result that H = −Σpᵢ log pᵢ is the same formula as Boltzmann's entropy — derived entirely independently — remains one of the most astonishing cross-domain coincidences in science.

Mode: Systematic-theoretical, solo. Shannon worked alone at Bell Labs for several years, driven by wartime cryptography and communication engineering. No serendipity: pure mathematical insight of the highest order, published in a form so complete that it founded a new field fully formed.