# [SCI] Information Theory

**Information Theory** (Shannon, 1948) is the mathematical theory of communication, defining entropy as a measure of uncertainty, and proving fundamental limits on data compression and error-free communication.

## Overview

Claude Shannon's 1948 paper "A Mathematical Theory of Communication" introduced the bit as the unit of information, defined information entropy H = −Σ pᵢ log₂ pᵢ, and proved two theorems: (1) data can be compressed to at most H bits/symbol (source coding theorem); (2) any noisy channel has a capacity C such that reliable communication is possible at all rates below C (channel coding theorem). Both theorems are tight. Shannon's entropy is mathematically identical to Boltzmann's thermodynamic entropy — a deep connection Shannon acknowledged.

Information theory underpins all of data compression (ZIP, MP3, JPEG), error-correcting codes (CDs, space probes, 5G), cryptography, and statistical machine learning.

## Key Figures & Recognition

- **Claude Shannon** (1916–2001): Father of information theory. Turing Award 1966, Kyoto Prize 1985, National Medal of Science. No Nobel (not physics or chemistry).
- **Norbert Wiener** (1894–1964): Cybernetics, 1948 (related but distinct work on feedback).

## Seminal Papers

- Shannon, C.E. ["A Mathematical Theory of Communication." *Bell Syst. Tech. J.* 27 (1948)](https://doi.org/10.1002/j.1538-7305.1948.tb01338.x)
- Shannon, C.E. & Weaver, W. *The Mathematical Theory of Communication*. Illinois, 1949.

## What This Enables

- **[TECH] Digital Computing** — Shannon's channel capacity and error-correcting codes are(Hamming, 1950) provide the theoretical foundation for all digital communication.
- **[SCI] Machine Learning Theory** — Mutual information, entropy, maximum likelihood, and the information bottleneck are core ML concepts drawn from information theory.
- **[SCI] Genomics & Computational Biology** — DNA is literally an information-storage medium; sequence alignment, compression, and variant calling use information-theoretic methods.

# Parents

* [SCI] Statistical Mechanics
* [SCI] Statistical Mechanics
* [TECH] Telegraph & Telephone