← Notes

Parameter Updates in Distribution Space

From Parameter Space to Distribution Space

Suppose a parameterized distribution \(p_\theta(x)\) is updated from \(\theta\) to

\[ \theta' = \theta + \Delta\theta. \]

The question is not merely how large \(\Delta\theta\) is in Euclidean parameter space, but how much that update changes the represented probability distribution.

Since KL divergence measures a difference between two distributions, define

\[ f(\Delta\theta) = D_{\mathrm{KL}} \left( p_\theta \,\middle\|\, p_{\theta+\Delta\theta} \right). \]

This function maps a displacement in parameter space to the resulting displacement in distribution space.

Why We Need a Local Approximation

For a neural-network policy, the mapping from parameters to the induced distribution can be complicated. Rather than reason about it exactly, we approximate the KL divergence locally around \(\Delta\theta=0\) using a Taylor expansion.

The supporting concepts are:

Second-Order Approximation of KL

A multivariate Taylor expansion of \(f(\Delta\theta)\) around \(\Delta\theta=0\) gives

\[ f(\Delta\theta) \approx f(0) + \nabla f(0)^\top \Delta\theta + \frac{1}{2} \Delta\theta^\top \nabla^2 f(0) \Delta\theta. \]

At \(\Delta\theta=0\), the two distributions are identical, so

\[ f(0) = D_{\mathrm{KL}}(p_\theta\|p_\theta) = 0. \]

KL divergence is locally minimized when the two distributions are equal, so the first-order term also vanishes:

\[ \nabla f(0)=0. \]

Therefore, for a small update,

\[ D_{\mathrm{KL}} \left( p_\theta \,\middle\|\, p_{\theta+\Delta\theta} \right) \approx \frac{1}{2} \Delta\theta^\top F(\theta) \Delta\theta, \]

where the local Hessian of KL is the Fisher information matrix \(F(\theta)\). This quadratic form is the local measure of how large a parameter update is in distribution space.

The Local KL Metric

The derivation can now be summarized in one line:

\[ D_{\mathrm{KL}} \left( p_\theta \,\middle\|\, p_{\theta+\Delta\theta} \right) \approx \frac{1}{2} \Delta\theta^\top F(\theta) \Delta\theta. \]

The Fisher information matrix is therefore the local curvature of KL divergence. It converts a parameter-space displacement into an approximation of the corresponding change in distribution space.

See the full derivation in Fisher Information Matrix.