Parameter Updates in Distribution Space
From Parameter Space to Distribution Space
Suppose a parameterized distribution \(p_\theta(x)\) is updated from \(\theta\) to
The question is not merely how large \(\Delta\theta\) is in Euclidean parameter space, but how much that update changes the represented probability distribution.
Since KL divergence measures a difference between two distributions, define
This function maps a displacement in parameter space to the resulting displacement in distribution space.
Why We Need a Local Approximation
For a neural-network policy, the mapping from parameters to the induced distribution can be complicated. Rather than reason about it exactly, we approximate the KL divergence locally around \(\Delta\theta=0\) using a Taylor expansion.
The supporting concepts are:
Second-Order Approximation of KL
A multivariate Taylor expansion of \(f(\Delta\theta)\) around \(\Delta\theta=0\) gives
At \(\Delta\theta=0\), the two distributions are identical, so
KL divergence is locally minimized when the two distributions are equal, so the first-order term also vanishes:
Therefore, for a small update,
where the local Hessian of KL is the Fisher information matrix \(F(\theta)\). This quadratic form is the local measure of how large a parameter update is in distribution space.
The Local KL Metric
The derivation can now be summarized in one line:
The Fisher information matrix is therefore the local curvature of KL divergence. It converts a parameter-space displacement into an approximation of the corresponding change in distribution space.
See the full derivation in Fisher Information Matrix.