Square h5 · Physics · Kingside
Shannon
Shannon entropy of a discrete distribution.
Meaning
Shannon entropy measures the average information content, or uncertainty, of a discrete probability distribution . When the logarithm is base 2, is in bits; with natural log it is in nats. Certain outcomes () contribute zero; a fair coin yields bit; a uniform distribution over symbols yields , the maximum for that alphabet. The formula is concave, extensive for independent sources, and the unique (up to scaling) functional satisfying Shannon’s natural axioms for information.
It is the communication-theoretic cousin of Boltzmann–Gibbs entropy : the mathematics coincides, the interpretation shifts from thermodynamic multiplicity to coding length. Source coding theorems say that is the asymptotic lower bound on average lossless compressed length.
Context
Claude Shannon defined this entropy in his 1948 paper “A Mathematical Theory of Communication,” founding information theory. The work quantified channel capacity, introduced the bit as a unit, and separated information from meaning—a move that transformed engineering, cryptography, and statistical inference. Jaynes later championed the formal parallel with statistical mechanics, while Kolmogorov and others developed algorithmic information alternatives.
On a physics board, Shannon’s earns its square because modern thermodynamics, quantum information (), and black-hole entropy debates all speak its dialect. Whether one is designing an error-correcting code or arguing about Landauer’s principle, the sum is the starting measure of ignorance and surprise.
Relative entropy (Kullback–Leibler divergence) and mutual information extend Shannon’s to comparisons between distributions and to shared information between variables. Those descendants power modern machine learning objectives and statistical-mechanics proofs of the second law’s information form.
Sources
- Grokipedia: Entropy (information theory); Information theory
- Wikipedia: Entropy (information theory)
- Britannica: Information theory