The covariance inequality says that a random vector can’t vary less than the part of it explained by its (linear) relationship with another random vector. It is the matrix version of the Cauchy-Schwarz inequality for covariances, and is the main tool behind the information inequality.
Theorem (Covariance inequality): Let $\x$ and $\y$ be random vectors of dimension $d$ and $k$, respectively, with finite second moments, and suppose $\bS = \Var \x$ is positive definite. Then
\[\Var \y \gge \cov(\y, \x) \bS^{-1} \cov(\x, \y),\]with equality if and only if $\y = \a + \B \tp \x$ with probability 1 for some constant vector $\a$ and matrix $\B$.
(the symbol $\gge$ is defined here).
Proof: Let $\B = \bS^{-1} \cov(\x, \y)$, a $d \times k$ matrix, and let $\r = \y - \B \tp \x$. Then
\[\begin{alignat*}{2} \Var \r &= \Var \y - \cov(\y, \x) \B - \B \tp \cov(\x, \y) + \B \tp \bS \B &\hspace{4em}& \href{random-vector.html}{\text{Variance of a sum}} \\ &= \Var \y - 2 \cov(\y, \x) \bS^{-1} \cov(\x, \y) + \cov(\y, \x) \bS^{-1} \bS \bS^{-1} \cov(\x, \y) && \text{Definition of } \B \\ &= \Var \y - \cov(\y, \x) \bS^{-1} \cov(\x, \y) \\ &\gge \zero, \end{alignat*}\]since $\Var \r$ is a variance and is therefore positive semidefinite.
Equality holds if and only if $\Var \r = \zero$; that is, if and only if $\r$ is equal to a constant $\a$ with probability 1, in which case $\y = \a + \B \tp \x$. Conversely, if $\y = \a + \B \tp \x$ for some $\B$, then $\cov(\x, \y) = \bS \B$, so the right hand side equals $\B \tp \bS \B = \Var \y$.
Interpretation
The matrix $\B$ in the proof is not a trick: it is the coefficient matrix from the population least squares regression of $\y$ on $\x$, and $\B \tp \x$ is the best linear predictor of $\y$ given $\x$. The residual $\r$ is uncorrelated with $\x$, since
\[\cov(\x, \r) = \cov(\x, \y) - \bS \B = \zero,\]so the variance of $\y$ splits into an “explained” part and a “residual” part:
\[\Var \y = \underbrace{\cov(\y, \x) \bS^{-1} \cov(\x, \y)}_{\Var(\B \tp \x)} + \Var \r.\]The covariance inequality simply says that the residual part can’t be negative. If you didn’t know which $\B$ to use, any $\B$ would give a valid inequality from $\Var(\y - \B \tp \x) \gge \zero$; choosing $\B$ to make the residual uncorrelated with $\x$ (equivalently, to minimize $\Var(\y - \B \tp \x)$) gives the sharpest one.
One dimension
When $X$ and $Y$ are scalars, the inequality becomes
\[\Var Y \ge \frac{\cov(X, Y)^2}{\Var X},\]or equivalently $\cov(X, Y)^2 \le \Var X \Var Y$; in other words, correlations are bounded by 1 in absolute value. This is the Cauchy-Schwarz inequality applied to the centered random variables $X - \Ex X$ and $Y - \Ex Y$, with covariance playing the role of the inner product.
Applications
- Information inequality: Take $\x$ to be the score $\u$, so that $\bS = \fI$. Any estimator $\bgh$ with mean $g(\bt)$ satisfies $\cov(\u, \bgh) = \nabla g$, and the covariance inequality immediately gives $\Var \bgh \gge \nabla g \tp \fI^{-1} \nabla g$. The bound is attained exactly when $\bgh$ is an affine function of the score.
- Nuisance parameters: Partitioning the score as $(\u_1, \u_2)$, the variance of the residual from regressing $\u_1$ on $\u_2$ is $\fI_{11} - \fI_{12} \fI_{22}^{-1} \fI_{21}$, the “efficient information” for $\bt_1$ that appears in the Cramér-Rao lower bound with nuisance parameters.