Natural Gradient
The ordinary gradient \(g\) tells us which direction improves the objective fastest per unit of Euclidean parameter movement. This is not generally the direction that gives the largest improvement per unit of policy change.
The natural-gradient direction is
where \(F\) is the Fisher information matrix.
Multiplying by \(F^{-1}\) reduces movement in parameter directions that strongly change the policy distribution and permits more movement in directions that change it weakly. It reshapes the raw gradient according to the local geometry of probability distributions.
Equivalently, the natural gradient answers: which direction improves the objective fastest while changing the policy distribution by only a small amount?
This is the direction used by TRPO.