Fisher metric vs KL-divergence

16 Oct 2016

Let $P$ and $Q$ be probability measures over a set $X$, and let $P$ be absolutely continuous with respect to $Q$. If $\mu$ is any measure on $X$ for which ${\displaystyle p=\frac{\mathrm{d}P}{\mathrm{d}\mu }}$ and ${\displaystyle q=\frac{\mathrm{d}Q}{\mathrm{d}\mu }}$ exist, then the Kullback-Leibler divergence from $Q$ to $P$ is given as

$$D_{\mathrm{KL}}(P\Vert Q)={\int }_{X}p\log \frac{p}{q}\mathrm{d}\mu .$$

Let the density $q=q(x,\theta )$ be parameterized by a vector $\theta$ and let $p$ be a variation of $q$, i.e., $p=q+\delta q$, where ${\displaystyle \delta q=\frac{\partial q}{\partial {\theta }_m}\delta {\theta }_m}$. Then

$$\begin{array}{rl}{D}_{\mathrm{KL}}(P\Vert Q)& ={\int }_{X}p\log \frac{p}{q}\mathrm{d}\mu ={\int }_X(q+\delta q)\log \frac{q+\delta q}{q}\mathrm{d}\mu \\ & ={\int }_{X}q\log (1+\frac{\delta q}{q})\mathrm{d}\mu +{\int }_X\delta q\log (1+\frac{\delta q}{q})\mathrm{d}\mu \\ & \approx {\int }_{X}q(\frac{\delta q}{q}-\frac{(\delta q{)}^2}{2q^2})\mathrm{d}\mu +{\int }_X\delta q\frac{\delta q}{q}\mathrm{d}\mu \\ & ={\int }_X\delta q\mathrm{d}\mu +\frac{1}{2}{\int }_{X}q\frac{(\delta q{)}^2}{q^2}\mathrm{d}\mu \\ & =\delta {\theta }_m\frac{\partial }{\partial {\theta }_m}{\int }_{X}q\mathrm{d}\mu +\frac{1}{2}\delta {\theta }_k\delta {\theta }_j{\int }_{X}q\left(\frac{1}{q}\frac{\partial q}{\partial {\theta }_k}\right)\left(\frac{1}{q}\frac{\partial q}{\partial {\theta }_j}\right)\mathrm{d}\mu \\ & =0+\frac{1}{2}\delta {\theta }_k\delta {\theta }_j\mathrm{E}\left\{\frac{\partial \log q}{\partial {\theta }_k}\frac{\partial \log q}{\partial {\theta }_j}\right\}\\ & =\frac{1}{2}\delta {\theta }_k\delta {\theta }_j{g}_{jk}(\theta ),\end{array}$$

where we recognize $g_{jk}$, the Fisher information metric,

$$g_{jk}(\theta )=\mathrm{E}\left\{\frac{\partial \log q}{\partial {\theta }_k}\frac{\partial \log q}{\partial {\theta }_j}\right\}.$$

Thus, the Fisher information metric is the second derivative of the Kullback-Leibler divergence,

Bonus: one prominent equality for the Fisher information¶

Let’s prove the following useful equality:

$$\mathrm{E}\frac{\partial \log p}{\partial {\theta }_k}\frac{\partial \log p}{\partial {\theta }_j}=-\mathrm{E}\frac{{\partial }^2\log p}{\partial {\theta }_k\partial {\theta }_j}.$$

Consider the argument on the right-hand side:

$$\frac{\partial }{\partial {\theta }_k}\left(\frac{\partial \log p}{\partial {\theta }_j}\right)=\frac{\partial }{\partial {\theta }_k}\left(\frac{1}{p}\frac{\partial p}{\partial {\theta }_j}\right)=-\frac{1}{p^2}\frac{\partial p}{\partial {\theta }_k}\frac{\partial p}{\partial {\theta }_j}+\frac{1}{p}\frac{{\partial }^{2}p}{\partial {\theta }_k\partial {\theta }_j}.$$

Compute its expectation:

$$-\mathrm{E}\frac{{\partial }^2\log p}{\partial {\theta }_k\partial {\theta }_j}={\int }_{X}p\frac{\partial \log p}{\partial {\theta }_k}\frac{\partial \log p}{\partial {\theta }_j}\mathrm{d}\mu -\frac{{\partial }^2}{\partial {\theta }_k\partial {\theta }_j}{\int }_{X}p\mathrm{d}\mu .$$

The second term on the right equals zero, which concludes the proof. Derivations in this post closely follow the book by Kullback.