Fisher Information
Overview
Fisher information measures how much information observable data carries about the parameters of a probability distribution. Equivalently, it measures how sensitively the log-likelihood changes when the parameters move.
This gives us a principled way to reason about parameter uncertainty and, later, about how large a parameter update is in distribution space rather than ordinary Euclidean parameter space.
The Statistical Setup
Suppose observations \(x\) are generated according to a parameterized probability density
The density accepts an observation \(x\) and a parameter vector \(\theta\), and tells us how plausible that observation is under those parameters.
The goal is to understand how strongly the observed data identifies the true parameter value. If small changes in \(\theta\) produce large changes in likelihood, then the data is highly informative about \(\theta\). If the likelihood barely changes, then the data provides weak information about the parameter.
Intuition: Narrow vs. Broad Likelihoods
Consider a Gaussian model whose unknown parameter is its mean:
Compare a low-variance Gaussian with a high-variance Gaussian. When the variance is small, observations cluster tightly around the true mean, so the likelihood as a function of \(\mu\) is narrow and the parameter is easier to identify. When the variance is large, many values of \(\mu\) explain the same observation reasonably well, so the likelihood is broad.
Low-Variance Model
Different observations produce likelihood curves whose peaks cluster in a narrow region. The data localizes the mean precisely.
High-Variance Model
The likelihood curves are much broader. A single observation gives weak evidence about where the mean should be.
The Score Function
The derivative of the log-likelihood with respect to the parameters is called the score:
It answers:
If I slightly change \(\theta\), in which direction and how strongly does the model assign more probability to the observed sample \(x\)?
Large score magnitude means the observation strongly favors moving the parameter in some direction. A score near zero means the observation provides little local evidence for moving the parameter.
The Expected Score Is Zero
Under standard regularity conditions, the score has zero expectation when samples are drawn from the same model:
This follows from normalization:
Moving \(\theta\) can redistribute probability mass among outcomes, but it cannot change the total probability mass.
For a finite sample, the empirical mean score is generally not exactly zero:
but its expectation is zero.
Fisher Information
Since the score is centered at zero, the natural measure of how strongly it fluctuates is its second moment. The Fisher information matrix is
Substituting the score definition,
In one dimension this reduces to
Because the expected score is zero, the scalar Fisher information is also the variance of the score.
Gaussian Example
Let
where the unknown parameter is the mean \(\mu\). The score of one sample is
Therefore,
So the Fisher information is
A low-variance Gaussian has large Fisher information. Even small deviations of a sample from the mean create a comparatively large score, so each observation gives a strong recommendation about how the mean should move.
A high-variance Gaussian has small Fisher information. Large deviations are normal, so the same observation gives much weaker evidence about how the mean should change.
Distribution of Scores
Narrow data distributions produce wide score distributions; noisy data produces scores concentrated closer to zero.
From Scalar Fisher Information to the Matrix
For a single scalar parameter \(\theta\), the score is a scalar,
and, because the expected score is zero, the Fisher information is both its second moment and its variance:
For a vector parameter \(\theta\in\mathbb R^d\), the score becomes a vector:
The Fisher information matrix is then the covariance matrix of that score vector:
The covariance form and the expected outer-product form are equal because \(\mathbb E[s_\theta(X)]=0\).
What Each Matrix Entry Means
The \((i,j)\) entry of the Fisher matrix is
The diagonal terms measure sensitivity to individual parameter directions:
The off-diagonal terms measure how changes in two parameter directions interact in their effect on the distribution.
This gives the Fisher matrix a geometric interpretation:
- Large eigenvalues identify directions in parameter space where the probability distribution changes rapidly.
- Small eigenvalues identify directions where the parameters can move substantially while changing the represented distribution only slightly.
Score Variance as Information
A wide score distribution means different observations generate strong positive or negative recommendations for how the parameter should move. The observed data is highly sensitive to that parameter, so nearby parameter values are easier to distinguish.
A narrow score distribution means most observations produce slopes close to zero. The log-likelihood is locally insensitive to that parameter, so nearby values are difficult to distinguish.
This is the key statistical interpretation of Fisher information:
Equivalent Hessian Form
Under standard regularity conditions, Fisher information can also be written as the negative expected Hessian of the log-likelihood:
These two forms describe the same local sensitivity:
The score form and the curvature form describe the same local sensitivity from two different viewpoints. The score-variance form asks how strongly different observations recommend moving the parameter. The curvature form asks how sharply the expected log-likelihood falls as the parameter moves away from its current value.
Strong negative expected curvature means the likelihood is sharply peaked, so the parameter is easy to locate and Fisher information is large. A relatively flat expected log-likelihood means many nearby parameter values explain the data similarly well, so Fisher information is small.
Connecting Fisher Information Back to KL Divergence
We originally introduced KL divergence because we wanted to measure how much a parameter update changes the represented probability distribution.
Let the current parameter be \(\theta\), and consider a nearby parameter
Define the KL divergence as a function of the proposed update:
We want to understand how much the distribution changes for a small parameter update, so we approximate \(f\) around the no-update point \(\Delta\theta=0\).
The constant term is zero because a distribution has zero KL divergence from itself:
The first-order term is also zero because KL divergence is minimized when the two distributions are identical:
Therefore, the first non-zero local term is second order:
The KL Hessian Is the Fisher Information Matrix
To identify the Hessian term, write the KL divergence as
Differentiating with respect to the second parameter \(\theta'\),
Differentiating again,
Evaluating at \(\theta'=\theta\), and using the equivalent curvature form of Fisher information,
gives
So the Fisher information matrix is exactly the local curvature of KL divergence in parameter space.
Local Geometry of Distribution Space
Substituting the Fisher matrix into the Taylor approximation gives
Compare this with the ordinary Euclidean squared distance
Euclidean distance treats every parameter direction equally. The Fisher quadratic form reweights parameter movement according to how much it changes the represented probability distribution.
Therefore,
acts as a local notion of squared distance in distribution space. The Fisher information matrix provides the local geometry of the statistical manifold formed by the family of distributions \(p_\theta\).
Why This Matters for Policy Optimization
A policy \(\pi_\theta(a\mid s)\) is itself a parameterized probability distribution. Its score is
The corresponding Fisher matrix measures how sensitively the policy distribution changes under parameter perturbations:
This is exactly the matrix that appears when KL divergence between nearby policies is expanded to second order:
Thus the Fisher information matrix provides the local geometry of policy space. A Euclidean step \(\Delta\theta\) is not judged by its raw length, but by how much it changes the represented distribution.
Summary
When observations make a parameter easy to locate, the score varies strongly across samples and the expected log-likelihood has strong negative curvature. Both views produce large Fisher information.
When observations are noisy or insensitive to a parameter, scores remain close to zero and the expected log-likelihood is comparatively flat. Fisher information is then small.
In higher dimensions, this becomes directional:
- large Fisher eigenvalues correspond to distributionally sensitive directions;
- small Fisher eigenvalues correspond to relatively insensitive directions;
- off-diagonal entries encode interactions between parameter coordinates.
This is the bridge from likelihood geometry to natural gradients, trust regions, and advanced policy-gradient methods.