The Bernstein-von Mises Theorem is the name given to a variety of results that are essentially Bayesian versions of the central limit theorem. Two such results are presented below.
Posterior density
Let $\bth$ denote the MLE and $\bd = \sqrt{n}(\bt - \bth)$, with $\pi_n(\bd \mid \x)$ denoting the posterior density of $\bd$. For each data set $\x$, the total variation distance between this posterior and a target density $\phi$ is
\[D_n(\x) = \tfrac12 \int \abs{\pi_n(\bd \mid \x) - \phi(\bd)} \, d\bd.\]The integral is over $\bd$, so $D_n(\x)$ is a function of the data, and therefore a random variable in the frequentist sense.
Theorem (Bernstein-von Mises): Suppose the prior $\pi(\bt)$ is proper and continuous with $\pi(\bt)>0$ for all $\bt \in \bT$. If regularity conditions (A)-(D) hold, then
\[\tfrac12 \int\abs{\pi_n(\bd \mid \x) - \phi(\bd)}\,d\bd \inP 0,\]where $\phi(\cdot)$ is the $\Normal(\zero, \fI(\bts)^{-1})$ density.
Posterior mean
The following regularity conditions are involved in the theorem:
- Standard likelihood regularity conditions (A), (B), and (C) are met.
-
For all $\eps > 0$, there exist $\delta > 0$ such that in the expansion
\[\begin{align*} \ell(\bt) = \, &\ell(\bt^*) + (\bt - \bts) \tp \u(\bts) - \\ &\tfrac{1}{2}(\bt - \bts) \tp [\oI(\bts) + \R(\bts)] (\bt - \bts), \end{align*}\]the probability of the event
\[\sup_{\norm{\bt - \bts} < \delta} \abs{\tfrac{1}{n}R_{ij}(\bt)} > \eps\]tends to 0 as $n \to \infty$ for all $i$ and $j$.
-
For all $\eps > 0$, there exist $\delta > 0$ such that the probability of the event
\[\sup_{\norm{\bt - \bts} \ge \delta} \tfrac{1}{n}\{\ell(\bt) - \ell(\bts)\} < -\eps\]tends to 1 as $n \to \infty$.
- The prior density $\pi(\bt)$ is continuous and positive for all $\bt \in \bT$.
- The prior expectation exists: $\int \norm{\bt} \pi(\bt) \, d\bt < \infty$.
Condition 3 is a “separation” condition: away from $\bts$, the likelihood is uniformly smaller than it is at $\bts$. It is not needed if the model is log-concave (regularity condition (D)), since concavity implies it: the supremum outside a ball around $\bts$ is controlled by the values on the boundary of the ball, where $\tfrac{1}{n}\lbrace \ell(\bt) - \ell(\bts) \rbrace$ converges uniformly to a negative limit.
Theorem (Bernstein-von Mises): Let $\btt$ denote the posterior mean. If (1)-(5) hold, then
\[\sqrt{n}(\btt-\bts) \inD \Normal(\zero, \fI^{-1}(\bts)).\]Proof: See Lehmann’s Theory of Point Estimation (Theorem 8.3), which he attributes to Peter Bickel.