← Notes

The Hessian

Definition

The Hessian is the matrix of all second partial derivatives of a scalar-valued function. For

\[ f:\mathbb{R}^n\rightarrow\mathbb{R}, \]

the Hessian is

\[ H_f(x) = \nabla^2 f(x) = \begin{bmatrix} \frac{\partial^2 f}{\partial x_1^2} & \cdots & \frac{\partial^2 f}{\partial x_1\partial x_n} \\ \vdots & \ddots & \vdots \\ \frac{\partial^2 f}{\partial x_n\partial x_1} & \cdots & \frac{\partial^2 f}{\partial x_n^2} \end{bmatrix}. \]

The gradient tells us the local slope. The Hessian tells us how that slope changes as we move: it describes local curvature.

Second-Order Response

In a second-order Taylor expansion, curvature contributes through the quadratic form

\[ \frac{1}{2}h^\top H_f(x_0)h. \]

The direction \(h\) matters: the same Hessian can imply very different curvature along different directions.

Example

For

\[ f(x,y)=x^2+xy+2y^2, \]

the Hessian is

\[ H_f = \begin{bmatrix} 2&1\\ 1&4 \end{bmatrix}. \]

With displacement

\[ h = \begin{bmatrix} 0\\ 0.1 \end{bmatrix}, \]

first apply the Hessian:

\[ H_f h = \begin{bmatrix} 2&1\\ 1&4 \end{bmatrix} \begin{bmatrix} 0\\ 0.1 \end{bmatrix} = \begin{bmatrix} 0.1\\ 0.4 \end{bmatrix}. \]

Then

\[ h^\top H_f h = \begin{bmatrix} 0&0.1 \end{bmatrix} \begin{bmatrix} 0.1\\ 0.4 \end{bmatrix} = 0.04, \]

so the second-order correction is

\[ \frac{1}{2}h^\top H_f h = 0.02. \]

Why the Hessian Matters for Policy Optimization

When KL divergence is expanded locally around the current policy, its first-order term vanishes and its second-order term controls the local change in distribution:

\[ D_{\mathrm{KL}} \left( p_\theta \,\middle\|\, p_{\theta+\Delta\theta} \right) \approx \frac{1}{2} \Delta\theta^\top F(\theta) \Delta\theta. \]

Here the Hessian of the local KL is the Fisher information matrix. This is the bridge from ordinary curvature to the geometry used by natural-gradient and trust-region methods.