Smoothing and differentiation08 Oct 2017
The problem addressed in this post came up during a discussion with Samuele Tosatto. The motivation was to understand the often used reparameterization trick (presented here with Gaussians for concreteness)
from the point of view of smoothing and differentiation operators. A particularly curious observation can be made if the statement above is written in the language of operators, using $S$ for smoothing (with the Gaussian kernel) and $D$ for differentiation with respect to $\theta$. Exploiting commutativity of operators $S$ and $D$, one obtains
which says that differentiation acts by multiplication under the smoothing operator—a property somewhat reminiscent of the Fourier transform. This and other related ideas are explored in a bit more depth in this pdf attachment, although many open questions remain.