All smooth $f$-divergences are locally the same

22 Jul 2017

For discrete distributions $p$ and $q$, the $f$-divergence is defined as

$$D_f(p\Vert q)=\sum _i{q}_{i}f\left(\frac{p_i}{q_i}\right),$$

where $f$ is a convex function $f:(0,\mathrm{\infty })\to \mathbb{R}$ satisfying the condition $f(1)=0$. If $p$ is a variation of $q$, then

$$D_f(q+dq\Vert q)=\sum _i{q}_{i}f\left(\frac{q_i+d{q}_i}{q_i}\right)=\sum _i{q}_{i}f(1+\frac{d{q}_i}{q_i}).$$

Provided $f$ is twice differentiable, we can develop it into Taylor series

$$f(1+\frac{d{q}_i}{q_i})\approx f(1)+f^{\prime }(1)\frac{d{q}_i}{q_i}+\frac{1}{2}{f}^{\prime \prime }(1){\left(\frac{d{q}_i}{q_i}\right)}^2$$

and thus approximate the $f$-divergence by a quadratic function

$$D_f(q+dq\Vert q)\approx \frac{1}{2}{f}^{\prime \prime }(1)\sum _{i}d{q}_i\frac{1}{q_i}d{q}_i.$$

Comparing it with the Fisher metric, we see that it is the same quadratic form scaled by a constant factor $f^{\prime \prime }(1)$.

Note that not all $f$-divergences are locally the same, only the smooth ones. For example, the total variation distance corresponds to

$$f_{TV}(x)=\frac{1}{2}|x-1|,$$

which is not quadratic around $x=1$.

Special case: $\alpha$-divergence¶

The $f$-divergence with $f$ having the form

$$f_{\alpha }(x)=\frac{(x^{\alpha }-1)-\alpha (x-1)}{\alpha (\alpha -1)},\alpha \in \mathbb{R}$$

is known as the $\alpha$-divergence. Noting that $f_{\alpha }$ has the properties $f_{\alpha }^{\prime \prime }(x)=x^{\alpha -2}$ and $f_{\alpha }^{\prime \prime }(1)=1$, we obtain the approximation of the $\alpha$-divergence

$$D_{\alpha }(q+dq\Vert q)\approx \frac{1}{2}\sum _{i}d{q}_i\frac{1}{q_i}d{q}_i,$$

which directly generalizes the result of Kullback for the KL divergence and its reverse, corresponding to $\alpha =1$ and $\alpha =0$ respectively.

Local approximation is exact for Pearson ${\chi }^2$ divergence¶

Pearson ${\chi }^2$ divergence is the $\alpha$-divergence with $\alpha =2$, corresponding to the generating function

$$f_2(x)=\frac{1}{2}{(x-1)}^2.$$

We established the following quadratic approximation of the $f$-divergence

$$D_f(q+dq\Vert q)\approx {\mathbb{E}}_q\left[\frac{f^{\prime \prime }(1)}{2}{\left(\frac{dq}{q}\right)}^2\right]$$

that is valid for small $dq$. However, if we allow big deviations $dq=p-q$, then we obtain Pearson ${\chi }^2$ divergence (scaled by $f^{\prime \prime }(1)$),

$${\mathbb{E}}_q\left[\frac{f^{\prime \prime }(1)}{2}{\left(\frac{p-q}{q}\right)}^2\right]=f^{\prime \prime }(1)D_2(p\Vert q).$$

Thus, Pearson ${\chi }^2$ divergence is the linear extension of the Fisher metric from a local neighborhood to the whole space. Consequently, local quadratic approximation is exact for Pearson ${\chi }^2$ divergence, since $f_2^{\prime \prime }(1)=1$.