Advanced Policy Gradient Methods
The Problem with Vanilla Policy Gradients
A standard policy-gradient update has the form
where, using the notation developed previously,
This is a first-order method: the gradient tells us a locally improving direction, but the learning rate \(\alpha\) is measured in parameter space. There is no principled connection between a step of a given size in \(\theta\) and how much the represented policy \(\pi_\theta(a\mid s)\) actually changes.
Why This Is More Consequential in Reinforcement Learning
In supervised learning, the objective is typically defined over a fixed data distribution:
Updating \(\theta\) changes the model's predictions, but it does not change which training examples exist or what their target labels are. If an update is somewhat too large, performance may worsen, but the same underlying data distribution remains available for the next update.
Reinforcement learning is different because the policy determines the data distribution. Under policy \(\pi_\theta\), trajectories are sampled according to
with
Changing \(\theta\) therefore changes the action probabilities, the states the agent visits, the trajectories it collects, the rewards and advantages it observes, and the distribution used to estimate the next gradient. A policy update does not merely change predictions on a fixed dataset; it changes future training data.
Destructive Updates
This makes overly large policy updates particularly dangerous. A small Euclidean step in parameter space can correspond to a large change in the induced action distribution. One bad update can push the policy into a very different region of behavior, invalidate advantage estimates collected under the old policy, and cause the next batch of data to come from a much worse policy.
The core issue is that Euclidean distance between parameter vectors and distance between the represented probability distributions are related by a state-dependent transformation. That relationship is not constant throughout parameter space.
This motivates the question:
If the parameters change by a small amount, how much does the represented policy distribution actually change?
How a Neural Network Parameterizes a Policy
A policy network does not usually output a probability distribution directly. It outputs the parameters of a chosen distribution.
Gaussian Policy for Continuous Actions
For continuous actions, a common choice is a Gaussian policy:
The network outputs the mean and, depending on the implementation, may also output or learn the standard deviation or log standard deviation. For a scalar action,
Categorical Policy for Discrete Actions
For \(K\) discrete actions, the network commonly outputs logits \(z_\theta(s)\in\mathbb{R}^K\). The policy is then
and therefore
In both cases, a change in neural-network parameters induces a change in a probability distribution, and the magnitude of that distributional change is what ultimately matters for policy optimization.
Exploration and Policy Variance
For a Gaussian policy, the standard deviation controls how broadly actions are sampled around the mean. Larger variance encourages broader exploration, while smaller variance produces more concentrated behavior. Some implementations learn this quantity; others keep it fixed or schedule it explicitly.
This is another example of why parameter-space distance alone is insufficient: changing a parameter that controls \(\sigma\) can alter the policy distribution in a qualitatively different way from changing the mean.
What We Want from a Distance Between Policies
To control how much a policy changes, we need a functional \(D(p,q)\) that quantifies the difference between two probability distributions. Useful properties include:
- \(D(p,q)\ge 0\), with \(D(p,q)=0\) when \(p=q\), so the quantity detects distributional difference.
- It should respond meaningfully to where probability mass is placed, not merely to raw parameter distance.
- It should connect to something operationally meaningful and estimable from samples, since sampled trajectories and actions are the data available in reinforcement learning.
These requirements point naturally toward divergence measures between probability distributions. In policy-gradient methods, the most important example is the Kullback-Leibler divergence, which gives a principled way to measure how much one policy distribution differs from another.
Returning to the Original Problem
With KL divergence, Taylor expansion, and the Fisher information matrix in place, we can now answer the question that motivated the detour:
Why can a seemingly small vanilla policy-gradient step produce a destructive change in the represented policy?
The key result is
Euclidean step size \(\|\Delta\theta\|_2\) does not determine how much the policy changes. The Fisher metric does. The same numerical parameter displacement can be benign in one region of parameter space and destructive in another.
Near-Deterministic Blowup
Consider a Gaussian policy with mean \(\mu_\theta(s)\) and standard deviation \(\sigma\):
Its score with respect to the policy parameters is
As \(\sigma\) becomes small, the factor \(1/\sigma^2\) becomes large. For the same action deviation \(a-\mu_\theta(s)\), the score can therefore grow dramatically.
Two effects compound:
- the raw policy-gradient estimate itself can become much larger in low-entropy regions; and
- the Fisher curvature also becomes larger, so the same parameter displacement produces a larger KL change.
This is exactly why a fixed learning rate in parameter space is a poor notion of a universally safe step size.
How a Single Update Can Collapse Exploration
Suppose a batch contains a large-magnitude advantage estimate \(\hat A_t\), perhaps because of a noisy return or an imperfect critic. In a region where the policy already has relatively small variance, the resulting gradient can be large enough to move the policy mean by many standard deviations in one update.
That is not a small behavioral change. It can relocate most of the action probability mass, sharply reduce the probability of actions that were previously explored, and effectively collapse entropy in that region of state space.
Once this happens, recovery is difficult. The updated policy may stop generating the trajectories needed to discover that the update was bad. Vanilla policy gradient has no mechanism that explicitly constrains how much the policy distribution may change in one step.
Walking Off the Data Manifold
The problem is not only where the new policy moves; it is also where our advantage estimates are trustworthy.
Whether \(\hat A_t\) is constructed using Monte Carlo returns, TD targets, or GAE, it relies on a value function that was fit using states sampled from the old policy's state-visitation distribution.
The critic is therefore most reliable near the region of state space represented in that training distribution. If the policy changes enough that new trajectories visit very different states, the critic is forced to extrapolate.
Regression extrapolation is least reliable precisely in the directions for which there was little or no data. A destructive update can therefore move the policy into a region where the advantage estimates that justified the move are no longer trustworthy.
The Surrogate Objective Is Local
There is a more structural way to state the same problem. Policy optimization commonly uses a surrogate objective built from data collected under the current policy. One standard form is
The true improvement identity depends on the new policy's state-visitation distribution. The surrogate replaces that unknown distribution with the old one because that is where the data came from.
This replacement is exact at \(\theta=\theta_{\mathrm{old}}\) and is useful locally, but its error grows as the new policy moves away. Vanilla policy gradient effectively follows the local ascent direction without explicitly checking whether the resulting policy remains in the neighborhood where the surrogate is trustworthy.
Once the policy has moved far enough, its trajectories visit a different state distribution. That distribution shift compounds through time: a slightly different early action can produce a different next state, which changes subsequent actions, states, rewards, and critic errors.
Sample Inefficiency from Strict On-Policy Learning
The policy-gradient theorem ties the gradient to the same policy that generated the data:
If a batch was collected using \(\theta_{\mathrm{old}}\), then the ordinary Monte Carlo estimator
is an estimate of the policy gradient at \(\theta_{\mathrm{old}}\). It is a local uphill direction at that specific point in parameter space.
After taking a step to a new parameter value, simply reusing the same samples and plugging the new \(\theta\) into the log-probability does not produce the new on-policy gradient. The integrand changed, but the samples are still distributed according to the old policy.
Why Naively Taking a Second Step on the Same Batch Breaks
Suppose the batch was collected from \(\pi_{\theta_{\mathrm{old}}}\), but after one update we evaluate
The samples \((s_i,a_i)\) are still drawn from the old state-action distribution. Therefore, in expectation, we are computing
not
The integrand may look familiar, but the measure is wrong. There is a distribution mismatch between the old-policy data being averaged over and the new-policy expectation we actually want.
This is why the most literal on-policy prescription is expensive: take one update, discard the batch, interact with the environment again, and collect fresh data from the updated policy.
Turning the Policy Gradient into an Objective
We now have the machinery needed to improve on vanilla policy gradients. Start from the sampled policy-gradient estimator collected under the old policy:
We can package this gradient into the simple policy-gradient objective
where the sampled advantages are treated as constants during policy optimization. Its gradient is
At the data-collection policy \(\theta=\theta_{\mathrm{old}}\), this is exactly the sampled policy-gradient direction.
Despite often being implemented as a loss, this quantity is conceptually maximized. In code, optimizers usually minimize \(-L^{\mathrm{PG}}\).
Why We Cannot Optimize the Plain Objective Indefinitely
The batch \((s_t,a_t,\hat A_t)\) was generated by \(\pi_{\theta_{\mathrm{old}}}\). As soon as \(\theta\) moves, the same samples are no longer distributed according to the policy being optimized.
Repeatedly maximizing \(L^{\mathrm{PG}}\) on the same fixed batch therefore keeps changing the integrand while leaving the sampling distribution frozen. The objective becomes increasingly stale as \(\theta\) moves away from \(\theta_{\mathrm{old}}\).
In the most literal on-policy implementation, the remedy is to take one update, discard the batch, and collect fresh data. That is statistically clean but sample inefficient.
Correcting the Distribution Mismatch with Importance Sampling
Importance sampling gives us a way to reuse actions sampled under the old policy while evaluating expectations associated with a new policy. Define the per-step likelihood ratio
The ratio directly measures the relative probability of the sampled action:
- \(r_t(\theta)=1\): no probability change;
- \(r_t(\theta)>1\): the sampled action is more likely under the new policy;
- \(r_t(\theta)<1\): the sampled action is less likely under the new policy.
Using this ratio gives the importance-sampled surrogate objective
This is not the same function as \(L^{\mathrm{PG}}\). Importance sampling does not transform \(\log\pi_\theta\) into a probability ratio. The two objectives are different away from the old policy.
Why the Ratio Objective Has the Same Initial Gradient
Differentiate the ratio:
At the starting point,
so
Consequently,
Both objectives therefore produce the same initial policy-gradient update.
But Their Gradients Diverge Away from the Old Policy
Away from \(\theta_{\mathrm{old}}\), the ratio objective has gradient
The importance ratio reweights each old-policy sample according to how likely its action is under the current policy. The plain log-probability objective has no such correction.
This makes repeated optimization on the same batch more meaningful, but it does not make it safe to optimize the surrogate to convergence.
Why Importance Sampling Alone Is Not Enough
After several gradient updates on the same batch, several problems remain:
- The current policy may become very different from the policy that generated the data, so the fixed batch and advantage estimates become stale.
- Advantage estimates are noisy, so aggressively optimizing the finite batch can overfit estimation error.
- Pushing the surrogate too far drives actions with positive sampled advantages toward higher probability and actions with negative advantages toward lower probability, regardless of whether those estimates remain trustworthy far from the data-collection policy.
Importance sampling corrects one mismatch—the action-probability measure—but it does not guarantee that the new policy stays in the local region where the critic, advantage estimates, and surrogate approximation remain reliable.
This Motivates a Trust Region
We now want two things simultaneously:
- reuse old-policy data through the importance ratio \(r_t(\theta)\); and
- prevent the updated policy from moving too far from \(\pi_{\theta_{\mathrm{old}}}\).
The natural measure of that second requirement is the KL divergence. A trust-region method therefore seeks to improve the surrogate objective while constraining the distributional change:
This is the central idea behind trust-region policy optimization: preserve the useful local policy-gradient direction, reuse the same batch more intelligently, and explicitly limit how far the represented policy distribution is allowed to move.
What We Need Next
We now have two related problems:
- stability: constrain policy updates so the new distribution does not move too far from the one on which our local objective and value estimates are reliable; and
- sample efficiency: reuse data collected under an older policy without pretending those samples came from the new policy.
The second problem is exactly what importance sampling solves. It lets us rewrite an expectation under one distribution using samples from another and correct for the mismatch with a likelihood ratio.
These ideas are the machinery behind trust-region methods and PPO: reuse old-policy data, correct the distribution mismatch, and explicitly prevent the updated policy from moving too far away.
Supporting Concepts
The next step is to quantify how a movement in parameter space changes the represented policy distribution. The supporting derivations are separated into the following pages: