All smooth $f$-divergences are locally the same

22 Jul 2017

For discrete distributions $p$ and $q$, the $f$-divergence is defined as

$$D_f(p\|q) = \sum_i q_i \,f\left( \frac{p_i}{q_i} \right),$$

where $f$ is a convex function $f: (0, \infty) \to \mathbb{R}$ satisfying the condition $f(1) = 0$. If $p$ is a variation of $q$, then

$$D_f(q+dq\|q) = \sum_i q_i \,f\left( \frac{q_i+dq_i}{q_i} \right) = \sum_i q_i \,f\left( 1 + \frac{dq_i}{q_i} \right).$$

Provided $f$ is twice differentiable, we can develop it into Taylor series

$$f(1 + \frac{dq_i}{q_i}) \approx f(1) + f^\prime(1) \frac{dq_i}{q_i} + \frac{1}{2} f^{\prime\prime}(1) \left( \frac{dq_i}{q_i} \right)^2$$

and thus approximate the $f$-divergence by a quadratic function

$$D_f(q+dq\|q) \approx \frac{1}{2} f^{\prime\prime}(1) \sum_i dq_i \frac{1}{q_i} dq_i.$$

Comparing it with the Fisher metric, we see that it is the same quadratic form scaled by a constant factor $f^{\prime\prime}(1)$.

Note that not all $f$-divergences are locally the same, only the smooth ones. For example, the total variation distance corresponds to

$$f_{TV}(x) = \frac{1}{2} | x - 1 |,$$

which is not quadratic around $x = 1$.

Special case: $\alpha$-divergence

The $f$-divergence with $f$ having the form

$$f_\alpha(x) = \frac{(x^\alpha-1) - \alpha(x-1)}{\alpha(\alpha-1)}, \; \alpha \in \mathbb{R}$$

is known as the $\alpha$-divergence. Noting that $f_\alpha$ has the properties $f_\alpha^{\prime\prime}(x) = x^{\alpha-2}$ and $f_\alpha^{\prime\prime}(1) = 1$, we obtain the approximation of the $\alpha$-divergence

$$D_\alpha(q+dq\|q) \approx \frac{1}{2} \sum_i dq_i \frac{1}{q_i} dq_i,$$

which directly generalizes the result of Kullback for the KL divergence and its reverse, corresponding to $\alpha = 1$ and $\alpha = 0$ respectively.

Local approximation is exact for Pearson $\chi^2$ divergence

Pearson $\chi^2$ divergence is the $\alpha$-divergence with $\alpha = 2$, corresponding to the generating function

$$f_2(x) = \frac{1}{2}\left( x-1 \right)^2.$$

We established the following quadratic approximation of the $f$-divergence

$$D_f(q+dq\|q) \approx \mathbb{E}_q \left[ \frac{f^{\prime\prime}(1)}{2} \left( \frac{dq}{q} \right)^2 \right]$$

that is valid for small $dq$. However, if we allow big deviations $dq = p - q$, then we obtain Pearson $\chi^2$ divergence (scaled by $f^{\prime\prime}(1)$),

$$\mathbb{E}_q \left[ \frac{f^{\prime\prime}(1)}{2} \left( \frac{p-q}{q} \right)^2 \right] = f^{\prime\prime}(1) D_2(p\|q).$$

Thus, Pearson $\chi^2$ divergence is the linear extension of the Fisher metric from a local neighborhood to the whole space. Consequently, local quadratic approximation is exact for Pearson $\chi^2$ divergence, since $f_2^{\prime\prime}(1) = 1$.