The information inequality is a lower bound on the variability of a statistic; the idea is very closely related to that of the Cramér-Rao lower bound.
See also: Gibbs inequality, which is unrelated but also sometimes referred to as the “information inequality”.
One dimension
Theorem (Information inequality): Let $\gh$ be a statistic with finite expectation $g(\theta) = \Ex \gh$. Suppose $X \sim p( \cdot \mid \ts)$ and $d/d\theta$ can be passed under the integral sign with respect to both $\int dP$ and $\int \gh dP$. Finally, suppose $\mathscr{I}(\ts) > 0$. Then
\[\Var \gh \ge \frac{\dot{g}(\ts)^2}{\mathscr{I}(\ts)}.\]Proof: Special case of multivariate proof given below.
The information inequality pertains to a single observed $X$; for the case involving repeated sampling from a distribution, see Cramér-Rao lower bound. This is perhaps slightly misleading, since $X$ could be a “single observation” from the $n$-dimensional joint distribution of all the observations. The information inequality still applies to this case, but is expressed more succinctly in the language of the CRLB.
Multiple dimensions
Theorem (Information inequality): Suppose $X \sim p(x \mid \bts)$, with $\fI(\bts)$ positive definite. Let $\bgh$ be an estimator with finite expected value $g(\bt)$. If $\nabla_{\bt} f(x \mid \bts)$ exists and can be passed under the integral sign with respect to $\int dP$ and $\int \bgh dP$, then
\[\Var \bgh \gge \nabla g(\bts) \tp \fI(\bts)^{-1} \nabla g(\bts)\](the symbol $\A \gge \B$ is defined here).
Proof: Let’s begin by labeling our assumption for easy reference later:
- $\fI(\bts)$ is positive definite
- $g(\theta) = \Ex \bgh$ exists
- $\nabla_{\theta}$ passable with respect to $\int \, dP$
- $\nabla_{\theta}$ passable with respect to $\int \bgh \, dP$
Furthermore, let $k$ denote the dimension of $\bgh$ and $d$ denote the dimension of $\bt$. Now,
\[\begin{alignat*}{2} \tag*{$\tcirc{1}$} \nabla g(\theta) &= \nabla_\theta \int \bgh \, dP &\hspace{4em}& \text{ii.} \\ &= \int \nabla_\theta \bgh \, dP && \text{iv.} \\ &= \int \nabla_\theta L(\bt) \bgh \tp \, dx && \href{vector-calculus.html}{\text{Chain rule}}: \pf{\bgh}{\bt} \, dP= \pf{p(x)}{\bt} \pf{\bgh \, p(x)}{p(x)} \, dx \\ &= \int \u(\bt) \bgh \tp \, dP && \u = \nabla \log p = \nabla L / p \\ &= \cov(\u, \bgh) && \Ex \u = \zero \text{ by iii.} \end{alignat*}\]Now, $\Var \u = \fI$, which is positive definite by i., so we can apply the covariance inequality with $\x = \u$ and $\y = \bgh$:
\[\begin{alignat*}{2} \Var \bgh &\gge \cov(\bgh, \u) \fI^{-1} \cov(\u, \bgh) &\hspace{4em}& \href{covariance-inequality.html}{\text{Covariance inequality}} \\ &= \nabla g \tp \fI^{-1} \nabla g && \tcirc{1} \end{alignat*}\]Intuition
The proof has two steps. Step $\tcirc{1}$ establishes the somewhat surprising result that even if an estimator is not based on likelihood in any way, it is still related to the score in the sense that the covariance between the estimator and the score is equal to the rate which its expected value changes as a function of $\bt$.
The second step is generic: a random vector can’t vary less than the part of it explained by its correlation with something else. Here $\nabla g \tp \fI^{-1} \u$ is the best linear predictor of $\bgh - g$ from the score, and the bound is its variance. Anything left over in $\bgh$ is uncorrelated with the score, which makes it noise that carries no information about $\bt$ and only adds variance.
The equality condition in the covariance inequality tells us when the bound is attained: exactly when $\bgh$ is an affine function of the score, $\bgh = g(\bt) + \nabla g \tp \fI^{-1} \u$. For example, if $X_1, \ldots, X_n \iid \Normal(\mu, \sigma^2)$ with $\sigma^2$ known, the score is $n(\bar{x} - \mu)/\sigma^2$, so $\bar{x}$ attains the bound; the sample median is not a linear function of the score and does not.