Entropy, Relative Entropy, and Information

Entropy is not one undifferentiated substance. Several quantities use “minus an average logarithm,” but their inputs, units, invariances, and operational questions differ. Relative entropy is more distinct still: it compares two probability assignments and quantifies distinguishability rather than the uncertainty of one assignment.

Required background. Calculus and Linear Algebra supplies eigenvalues and linear-map language; logarithms, probability, and entropy conventions are defined below. Helpful background. Classical and Variational Mechanics supplies phase-space evolution; Quantum States, Observables, and Measurement supplies state and observable language, while Density Operators, Entanglement, and Reduced States supplies density operators, traces, and partial traces.

Four entropy notions and their jobs

Thermodynamic entropy $S(U,V,N,\ldots)$ is an equilibrium state function. Reversible heat exchange defines differences through $dS=\delta Q_{\mathrm{rev}}/T$, while the second law constrains irreversible processes. No microscopic model is required to use it.

For a discrete classical ensemble with probabilities $p_i$,

$$S_{\mathrm G}[p]=-\sum_i p_i\ln p_i.$$

Terms with $p_i=0$ are assigned $0\ln0=0$. With $k_{\mathrm B}=1$ this Gibbs entropy is dimensionless; restoring physical units multiplies it by $k_{\mathrm B}$. A uniform microcanonical distribution over $\mathcal N_E$ states gives $S_{\mathrm G}=\ln\mathcal N_E$. Identifying that quantity with thermodynamic entropy additionally requires an equilibrium interpretation, a specified shell convention, and the appropriate large-system assumptions. Tong develops the microcanonical count in Tong, current notes, §§1.2–1.2.2, printed pp. 2–10.

For a discrete random variable $X$ with law $p_X$, Shannon entropy applies the same functional. Base-two logarithms give bits; natural logarithms give nats. It controls average lossless coding uncertainty under a specified probability law Médard 2010, Lecture 1, printed pp. 3–4. Differential entropy for a continuous variable is a different, reference-measure-dependent object: it may be negative, infinite, or coordinate dependent. No continuous formula is being used on this page.

For a density operator $\rho$ in finite dimension,

$$S_{\mathrm vN}(\rho)=-\operatorname{Tr}(\rho\ln\rho).$$

Here $\rho\ln\rho$ is defined spectrally with the same convention $0\ln0=0$. The von Neumann entropy is the Shannon entropy of the eigenvalues of $\rho$, not generally the entropy of an arbitrary preparation ensemble representing the same operator. A reduced state may have nonzero von Neumann entropy even when the total bipartite state is pure; in that setting it measures entanglement across the chosen bipartition. Preskill develops these properties in Preskill, Chapter 5, §5.2, equations (5.42)–(5.58), printed pp. 14–18.

Quantity	Input	Units here	Operational role	Typical evolution statement
Thermodynamic $S$	equilibrium macrostate	dimensionless with $k_{\mathrm B}=1$	heat, work, and allowed macroscopic processes	nondecreasing for an isolated spontaneous process
Gibbs $S_{\mathrm G}$	discrete classical ensemble	nats	uncertainty in a microscopic probability law	unchanged by a bijective relabeling of discrete microstates
Shannon $H(X)$	discrete random variable	bits or nats	coding uncertainty	depends on the channel and reported variable
von Neumann $S_{\mathrm vN}$	density operator	nats	mixedness; for a pure bipartite state, entanglement of a chosen part	constant under unitary evolution of the closed state

The table prevents a shared formula from erasing physical meaning. A Shannon entropy becomes thermodynamic only after one specifies a physical ensemble, coarse graining, constraints, and units. Conversely, thermodynamic entropy need not be interpreted as an observer’s subjective ignorance.

Coarse graining must name the map

A coarse graining discards specified distinctions: phase-space cells, unresolved observables, a subsystem, or a macroscopic projection. Replacing a fine state by the maximum-entropy state compatible with retained data can increase an assigned coarse-grained Gibbs or von Neumann entropy. Calling that increase thermodynamic requires an additional equilibrium argument.

Other operations behave differently. If a reported discrete label deterministically merges outcomes, its Shannon entropy can decrease. The continuous phase-space Gibbs functional, when defined relative to Liouville measure, is conserved by Hamiltonian flow, while von Neumann entropy is constant under unitary evolution. A monotonicity claim is therefore incomplete until it names the state, the map, the reference measure when one is needed, and the entropy functional.

Relative entropy measures distinguishability

Let $p$ and $q$ be distributions on the same finite or countable alphabet $\mathcal A$. Their relative entropy is

$$D(p\lVert q)=\sum_{i\in\mathcal A}p_i\ln\frac{p_i}{q_i}.$$

Terms with $p_i=0$ contribute zero. If $p_i\gt 0$ for some $i$ with $q_i=0$, then $D(p\lVert q)=+\infty$. Thus finiteness requires $\operatorname{supp}p\subseteq\operatorname{supp}q$. These conventions appear in Polyanskiy and Wu 2016, §1.2, printed pp. 15–17; Kullback and Leibler introduced the hypothesis-discrimination interpretation in Kullback and Leibler 1951, §2, pp. 80–82.

When the support inclusion holds, apply $-\ln x\geq1-x$ only for indices with $p_i\gt 0$:

$$\begin{aligned} D(p\lVert q) &=-\sum_{i\in\operatorname{supp}p}p_i\ln\frac{q_i}{p_i}\\ &\geq\sum_{i\in\operatorname{supp}p}(p_i-q_i)\\ &=1-q(\operatorname{supp}p)\geq0. \end{aligned}$$

Equality in every step holds exactly when $p=q$ as distributions. Relative entropy is nevertheless asymmetric and fails the triangle inequality, so it is not a distance metric.

Its decisive monotonicity is data processing. If the same stochastic channel $K$ is applied to both hypotheses, then

$$D(Kp\lVert Kq)\leq D(p\lVert q).$$

Discarding observations cannot make two hypotheses easier to distinguish. Polyanskiy and Wu prove positivity and channel data processing in Polyanskiy and Wu 2016, §§2.1–2.2, printed pp. 20–22.

For a finite alphabet, the proof makes the information loss explicit. Write $K(y|i)\geq0$ for the probability of reporting $y$ given $i$, with $\sum_yK(y|i)=1$. Define $P_{iy}=p_iK(y|i)$ and $Q_{iy}=q_iK(y|i)$. The log-sum inequality applied separately to each reported outcome gives

$$\sum_i P_{iy}\ln\frac{P_{iy}}{Q_{iy}} \geq \left(\sum_iP_{iy}\right) \ln\frac{\sum_iP_{iy}}{\sum_iQ_{iy}}.$$

Summing over $y$, the left side is $D(p\lVert q)$ because the same channel occurs in both hypotheses; the right side is $D(Kp\lVert Kq)$. The log-sum inequality itself follows from the positivity proof above after normalizing the two nonnegative lists. The support conventions handle zero entries. Thus the theorem describes precisely the loss caused by hiding the input label $i$.

Mutual information is a useful consequence:

$$I(X;Y)=D(p_{XY}\lVert p_Xp_Y)\geq0.$$

It vanishes exactly when $X$ and $Y$ are independent, and it cannot increase when either variable is passed through a local channel. This gives “information” an operational meaning: correlation available for inference, rather than a synonym for thermodynamic entropy.

Exercise: a biased coin and an erasing channel

Let $p=(3/4,1/4)$ describe a biased coin and $q=(1/2,1/2)$ a fair one. Compute $D(p\lVert q)$, then pass both laws through a channel that reports only “a toss occurred.”

Solution

Direct substitution gives

$$D(p\lVert q) =\frac34\ln\frac32+\frac14\ln\frac12 \approx0.1308\ \text{nats}.$$

The erasing channel maps both outcomes to one symbol, so the two processed distributions are both $(1)$. Their relative entropy is zero, consistent with data processing. The loss is not a change in either physical coin; it is a loss of data with which to discriminate the hypotheses.

The gravitational boundary

Bekenstein–Hawking entropy, Wald entropy, generalized entropy, fine-grained radiation entropy, and the thermodynamic entropy of matter are distinct gravitational or semiclassical constructions. Their definitions, hypotheses, and regimes are supplied in later volumes; no identification among them is made here. This page provides only the probability and quantum-information language. Gravitational definitions belong to Black-Hole Thermodynamics and Semiclassical Gravity, while claims about unitary recovery belong to Black-Hole Information.

Fluctuations, Response, and Susceptibilities next shows how an equilibrium probability law turns uncertainty in an observable into measurable static response.

References

• Kullback, Solomon, and Richard A. Leibler. “On Information and Sufficiency.” Annals of Mathematical Statistics 22 (1951): 79–86, §2, pp. 80–82. doi:10.1214/aoms/1177729694.
• Médard, Muriel. “Introduction, Entropy, and Mutual Information.” MIT 6.441 Information Theory (Spring 2010), Lecture 1, printed pp. 3–6. MIT OpenCourseWare PDF.
• Polyanskiy, Yury, and Yihong Wu. Information Theory: From Coding to Learning. MIT 6.441 course notes (Spring 2016), §§1.2 and 2.1–2.2, printed pp. 15–22. MIT OpenCourseWare PDF.
• Preskill, John. Quantum Information Theory, Chapter 5, §5.2, equations (5.42)–(5.58), printed pp. 14–18. California Institute of Technology lecture notes. Official PDF.
• Tong, David. Statistical Physics. University of Cambridge Part II Mathematical Tripos lecture notes, current official undated-cover PDF, §§1.2–1.2.2, printed pp. 2–10. Open PDF.