← Notes

Fisher Information

Overview

Fisher information measures how much information observable data carries about the parameters of a probability distribution. Equivalently, it measures how sensitively the log-likelihood changes when the parameters move.

This gives us a principled way to reason about parameter uncertainty and, later, about how large a parameter update is in distribution space rather than ordinary Euclidean parameter space.

The Statistical Setup

Suppose observations \(x\) are generated according to a parameterized probability density

\[ p_\theta(x). \]

The density accepts an observation \(x\) and a parameter vector \(\theta\), and tells us how plausible that observation is under those parameters.

The goal is to understand how strongly the observed data identifies the true parameter value. If small changes in \(\theta\) produce large changes in likelihood, then the data is highly informative about \(\theta\). If the likelihood barely changes, then the data provides weak information about the parameter.

Intuition: Narrow vs. Broad Likelihoods

Consider a Gaussian model whose unknown parameter is its mean:

\[ X \sim \mathcal N(\mu,\sigma^2), \qquad \theta=\mu. \]

Compare a low-variance Gaussian with a high-variance Gaussian. When the variance is small, observations cluster tightly around the true mean, so the likelihood as a function of \(\mu\) is narrow and the parameter is easier to identify. When the variance is large, many values of \(\mu\) explain the same observation reasonably well, so the likelihood is broad.

Low-Variance Model

Different observations produce likelihood curves whose peaks cluster in a narrow region. The data localizes the mean precisely.

High-Variance Model

The likelihood curves are much broader. A single observation gives weak evidence about where the mean should be.

The Score Function

The derivative of the log-likelihood with respect to the parameters is called the score:

\[ s_\theta(x) = \nabla_\theta \log p_\theta(x). \]

It answers:

If I slightly change \(\theta\), in which direction and how strongly does the model assign more probability to the observed sample \(x\)?

Large score magnitude means the observation strongly favors moving the parameter in some direction. A score near zero means the observation provides little local evidence for moving the parameter.

The Expected Score Is Zero

Under standard regularity conditions, the score has zero expectation when samples are drawn from the same model:

\[ \mathbb E_{x\sim p_\theta} \left[ s_\theta(x) \right] = 0. \]

This follows from normalization:

\[ \begin{aligned} \mathbb E_{x\sim p_\theta} \left[ \nabla_\theta \log p_\theta(x) \right] &= \int p_\theta(x) \frac{\nabla_\theta p_\theta(x)}{p_\theta(x)} \,dx \\ &= \int \nabla_\theta p_\theta(x) \,dx \\ &= \nabla_\theta \int p_\theta(x)\,dx \\ &= \nabla_\theta 1 = 0. \end{aligned} \]

Moving \(\theta\) can redistribute probability mass among outcomes, but it cannot change the total probability mass.

For a finite sample, the empirical mean score is generally not exactly zero:

\[ \bar{s}_N(\theta) = \frac{1}{N} \sum_{i=1}^{N} s_\theta(x_i), \]

but its expectation is zero.

Fisher Information

Since the score is centered at zero, the natural measure of how strongly it fluctuates is its second moment. The Fisher information matrix is

\[ F(\theta) = \mathbb E_{x\sim p_\theta} \left[ s_\theta(x)s_\theta(x)^\top \right]. \]

Substituting the score definition,

\[ F(\theta) = \mathbb E_{x\sim p_\theta} \left[ \nabla_\theta \log p_\theta(x) \nabla_\theta \log p_\theta(x)^\top \right]. \]

In one dimension this reduces to

\[ I(\theta) = \mathbb E \left[ \left( \frac{\partial}{\partial\theta} \log p_\theta(X) \right)^2 \right]. \]

Because the expected score is zero, the scalar Fisher information is also the variance of the score.

Gaussian Example

Let

\[ X\sim\mathcal N(\mu,\sigma^2), \]

where the unknown parameter is the mean \(\mu\). The score of one sample is

\[ s_\mu(x) = \frac{\partial}{\partial\mu} \log p(x\mid\mu) = \frac{x-\mu}{\sigma^2}. \]

Therefore,

\[ \operatorname{Var}[s_\mu(X)] = \frac{1}{\sigma^2}. \]

So the Fisher information is

\[ I(\mu) = \frac{1}{\sigma^2}. \]

A low-variance Gaussian has large Fisher information. Even small deviations of a sample from the mean create a comparatively large score, so each observation gives a strong recommendation about how the mean should move.

A high-variance Gaussian has small Fisher information. Large deviations are normal, so the same observation gives much weaker evidence about how the mean should change.

Distribution of Scores

Narrow data distributions produce wide score distributions; noisy data produces scores concentrated closer to zero.

From Scalar Fisher Information to the Matrix

For a single scalar parameter \(\theta\), the score is a scalar,

\[ s_\theta(x) = \frac{\partial}{\partial\theta} \log p_\theta(x), \]

and, because the expected score is zero, the Fisher information is both its second moment and its variance:

\[ I(\theta) = \mathbb E \left[ s_\theta(X)^2 \right] = \operatorname{Var} \left[ s_\theta(X) \right]. \]

For a vector parameter \(\theta\in\mathbb R^d\), the score becomes a vector:

\[ s_\theta(x) = \nabla_\theta \log p_\theta(x) = \begin{bmatrix} \frac{\partial}{\partial\theta_1}\log p_\theta(x) \\ \vdots \\ \frac{\partial}{\partial\theta_d}\log p_\theta(x) \end{bmatrix}. \]

The Fisher information matrix is then the covariance matrix of that score vector:

\[ F(\theta) = \operatorname{Cov}_{x\sim p_\theta} \left[ s_\theta(x) \right] = \mathbb E_{x\sim p_\theta} \left[ s_\theta(x)s_\theta(x)^\top \right]. \]

The covariance form and the expected outer-product form are equal because \(\mathbb E[s_\theta(X)]=0\).

What Each Matrix Entry Means

The \((i,j)\) entry of the Fisher matrix is

\[ F_{ij}(\theta) = \mathbb E \left[ \frac{\partial}{\partial\theta_i} \log p_\theta(X) \; \frac{\partial}{\partial\theta_j} \log p_\theta(X) \right]. \]

The diagonal terms measure sensitivity to individual parameter directions:

\[ F_{ii}(\theta) = \mathbb E \left[ \left( \frac{\partial}{\partial\theta_i} \log p_\theta(X) \right)^2 \right]. \]

The off-diagonal terms measure how changes in two parameter directions interact in their effect on the distribution.

This gives the Fisher matrix a geometric interpretation:

Score Variance as Information

A wide score distribution means different observations generate strong positive or negative recommendations for how the parameter should move. The observed data is highly sensitive to that parameter, so nearby parameter values are easier to distinguish.

A narrow score distribution means most observations produce slopes close to zero. The log-likelihood is locally insensitive to that parameter, so nearby values are difficult to distinguish.

This is the key statistical interpretation of Fisher information:

\[ \text{more variable scores} \quad\Longleftrightarrow\quad \text{more information about the parameter}. \]

Equivalent Hessian Form

Under standard regularity conditions, Fisher information can also be written as the negative expected Hessian of the log-likelihood:

\[ F(\theta) = - \mathbb E_{x\sim p_\theta} \left[ \nabla_\theta^2 \log p_\theta(x) \right]. \]

These two forms describe the same local sensitivity:

\[ \boxed{ F(\theta) = \mathbb E \left[ \nabla_\theta \log p_\theta(x) \nabla_\theta \log p_\theta(x)^\top \right] = - \mathbb E \left[ \nabla_\theta^2\log p_\theta(x) \right] }. \]

The score form and the curvature form describe the same local sensitivity from two different viewpoints. The score-variance form asks how strongly different observations recommend moving the parameter. The curvature form asks how sharply the expected log-likelihood falls as the parameter moves away from its current value.

Strong negative expected curvature means the likelihood is sharply peaked, so the parameter is easy to locate and Fisher information is large. A relatively flat expected log-likelihood means many nearby parameter values explain the data similarly well, so Fisher information is small.

Connecting Fisher Information Back to KL Divergence

We originally introduced KL divergence because we wanted to measure how much a parameter update changes the represented probability distribution.

Let the current parameter be \(\theta\), and consider a nearby parameter

\[ \theta' = \theta + \Delta\theta. \]

Define the KL divergence as a function of the proposed update:

\[ f(\Delta\theta) = D_{\mathrm{KL}} \left( p_\theta \,\middle\|\, p_{\theta+\Delta\theta} \right). \]

We want to understand how much the distribution changes for a small parameter update, so we approximate \(f\) around the no-update point \(\Delta\theta=0\).

\[ \begin{aligned} f(\Delta\theta) \approx\;& f(0) + \nabla_{\Delta\theta}f(0)^\top\Delta\theta \\ &+ \frac{1}{2} \Delta\theta^\top \nabla_{\Delta\theta}^2 f(0) \Delta\theta. \end{aligned} \]

The constant term is zero because a distribution has zero KL divergence from itself:

\[ f(0) = D_{\mathrm{KL}}(p_\theta\|p_\theta) = 0. \]

The first-order term is also zero because KL divergence is minimized when the two distributions are identical:

\[ \nabla_{\Delta\theta}f(0) = 0. \]

Therefore, the first non-zero local term is second order:

\[ D_{\mathrm{KL}} \left( p_\theta \,\middle\|\, p_{\theta+\Delta\theta} \right) \approx \frac{1}{2} \Delta\theta^\top \nabla_{\Delta\theta}^2 D_{\mathrm{KL}} \left( p_\theta \,\middle\|\, p_{\theta+\Delta\theta} \right) \bigg|_{\Delta\theta=0} \Delta\theta. \]

The KL Hessian Is the Fisher Information Matrix

To identify the Hessian term, write the KL divergence as

\[ D_{\mathrm{KL}} \left( p_\theta \,\middle\|\, p_{\theta'} \right) = \mathbb E_{x\sim p_\theta} \left[ \log p_\theta(x) - \log p_{\theta'}(x) \right]. \]

Differentiating with respect to the second parameter \(\theta'\),

\[ \nabla_{\theta'} D_{\mathrm{KL}} \left( p_\theta \,\middle\|\, p_{\theta'} \right) = - \mathbb E_{x\sim p_\theta} \left[ \nabla_{\theta'} \log p_{\theta'}(x) \right]. \]

Differentiating again,

\[ \nabla_{\theta'}^2 D_{\mathrm{KL}} \left( p_\theta \,\middle\|\, p_{\theta'} \right) = - \mathbb E_{x\sim p_\theta} \left[ \nabla_{\theta'}^2 \log p_{\theta'}(x) \right]. \]

Evaluating at \(\theta'=\theta\), and using the equivalent curvature form of Fisher information,

\[ F(\theta) = - \mathbb E_{x\sim p_\theta} \left[ \nabla_\theta^2 \log p_\theta(x) \right], \]

gives

\[ \boxed{ \nabla_{\theta'}^2 D_{\mathrm{KL}} \left( p_\theta \,\middle\|\, p_{\theta'} \right) \bigg|_{\theta'=\theta} = F(\theta) }. \]

So the Fisher information matrix is exactly the local curvature of KL divergence in parameter space.

Local Geometry of Distribution Space

Substituting the Fisher matrix into the Taylor approximation gives

\[ \boxed{ D_{\mathrm{KL}} \left( p_\theta \,\middle\|\, p_{\theta+\Delta\theta} \right) \approx \frac{1}{2} \Delta\theta^\top F(\theta) \Delta\theta }. \]

Compare this with the ordinary Euclidean squared distance

\[ \|\Delta\theta\|_2^2 = \Delta\theta^\top I\,\Delta\theta. \]

Euclidean distance treats every parameter direction equally. The Fisher quadratic form reweights parameter movement according to how much it changes the represented probability distribution.

Therefore,

\[ \Delta\theta^\top F(\theta) \Delta\theta \]

acts as a local notion of squared distance in distribution space. The Fisher information matrix provides the local geometry of the statistical manifold formed by the family of distributions \(p_\theta\).

Why This Matters for Policy Optimization

A policy \(\pi_\theta(a\mid s)\) is itself a parameterized probability distribution. Its score is

\[ \nabla_\theta \log \pi_\theta(a\mid s). \]

The corresponding Fisher matrix measures how sensitively the policy distribution changes under parameter perturbations:

\[ F(\theta) = \mathbb E \left[ \nabla_\theta \log \pi_\theta(a\mid s) \nabla_\theta \log \pi_\theta(a\mid s)^\top \right]. \]

This is exactly the matrix that appears when KL divergence between nearby policies is expanded to second order:

\[ D_{\mathrm{KL}} \left( \pi_\theta \,\middle\|\, \pi_{\theta+\Delta\theta} \right) \approx \frac{1}{2} \Delta\theta^\top F(\theta) \Delta\theta. \]

Thus the Fisher information matrix provides the local geometry of policy space. A Euclidean step \(\Delta\theta\) is not judged by its raw length, but by how much it changes the represented distribution.

Summary

When observations make a parameter easy to locate, the score varies strongly across samples and the expected log-likelihood has strong negative curvature. Both views produce large Fisher information.

When observations are noisy or insensitive to a parameter, scores remain close to zero and the expected log-likelihood is comparatively flat. Fisher information is then small.

In higher dimensions, this becomes directional:

This is the bridge from likelihood geometry to natural gradients, trust regions, and advanced policy-gradient methods.