# [SCI] Information Theory

**Information Theory** (Shannon, 1948) is the mathematical theory of communication, defining entropy as a measure of uncertainty, and proving fundamental limits on data compression and error-free communication.

## Overview

Claude Shannon's 1948 paper "A Mathematical Theory of Communication" introduced the bit as the unit of information, defined information entropy H = −Σ pᵢ log₂ pᵢ, and proved two theorems: (1) data can be compressed to at most H bits/symbol (source coding theorem); (2) any noisy channel has a capacity C such that reliable communication is possible at all rates below C (channel coding theorem). Both theorems are tight. Shannon's entropy is mathematically identical to Boltzmann's thermodynamic entropy — a deep connection Shannon acknowledged.

Information theory underpins all of data compression (ZIP, MP3, JPEG), error-correcting codes (CDs, space probes, 5G), cryptography, and statistical machine learning.

## Key Figures & Recognition

- **Claude Shannon** (1916–2001): Father of information theory. Turing Award 1966, Kyoto Prize 1985, National Medal of Science. No Nobel (not physics or chemistry).
- **Norbert Wiener** (1894–1964): Cybernetics, 1948 (related but distinct work on feedback).

## Seminal Papers

- Shannon, C.E. ["A Mathematical Theory of Communication." *Bell Syst. Tech. J.* 27 (1948)](https://doi.org/10.1002/j.1538-7305.1948.tb01338.x)
- Shannon, C.E. & Weaver, W. *The Mathematical Theory of Communication*. Illinois, 1949.

## What This Enables

- **[TECH] Digital Computing** — Shannon's channel capacity and error-correcting codes (Hamming, 1950) provide the theoretical foundation for digital communication.
- **[SCI] Machine Learning Theory** — Mutual information, entropy, maximum likelihood, and the information bottleneck are core ML concepts from information theory.
- **[SCI] Genomics & Computational Biology** — DNA is an information-storage medium; sequence alignment, compression, and variant calling use information-theoretic methods.

## Discovery Character
⏎
**Surprise level**: Extreme — Shannon proved that there is a hard mathematical limit (channel capacity) to reliable communication, and that it can be approached arbitrarily closely with the right coding — regardless of what the channel looks like. Nobody expected information to have a rigorous mathematical definition, let alone hard quantitative limits. The result that H = −Σpᵢ log pᵢ is the same formula as Boltzmann's entropy — derived entirely independently — remains one of the most astonishing cross-domain coincidences in science.
⏎
**Mode**: Systematic-theoretical, solo. Shannon worked alone at Bell Labs for several years, driven by wartime cryptography and communication engineering. No serendipity: pure mathematical insight of the highest order, published in a form so complete that it founded a new field fully formed.
⏎
# Parents

* [SCI] Statistical Mechanics
* [SCI] Statistical Mechanics
* [TECH] Telegraph & Telephone