← Advanced Policy Gradient Methods

Proximal Policy Optimization (PPO)

Overview

Proximal Policy Optimization (PPO) keeps the core trust-region intuition of TRPO, but avoids solving a constrained second-order optimization problem at every update.

Instead of explicitly computing a natural-gradient step and enforcing a hard KL constraint, PPO uses a first-order objective designed to discourage the new policy from moving too far away from the old policy.

Starting Point: The Importance-Sampled Surrogate

Recall the surrogate objective built from data collected under the old policy \(\pi_{\theta_{\mathrm{old}}}\):

\[ L^{\mathrm{IS}}(\theta) = \mathbb{E}_t \left[ r_t(\theta)\,\hat A_t \right], \qquad r_t(\theta) = \frac{ \pi_\theta(a_t\mid s_t) }{ \pi_{\theta_{\mathrm{old}}}(a_t\mid s_t) }. \]

The ratio \(r_t(\theta)\) tells us how the new policy changes the probability of the sampled action:

This objective is the natural starting point because it reweights old-policy samples so they can be used to estimate a quantity associated with the new policy.

Why TRPO Was Introduced

If we optimize \(L^{\mathrm{IS}}(\theta)\) too aggressively, the current policy can move far from the policy that generated the batch. Then the fixed data and fixed advantage estimates become stale, and the optimization target becomes unreliable.

TRPO addresses this by solving

\[ \max_\theta \quad L^{\mathrm{IS}}(\theta) \qquad \text{subject to} \qquad \bar D_{\mathrm{KL}} \left( \pi_{\theta_{\mathrm{old}}} \,\middle\|\, \pi_\theta \right) \le \delta. \]

PPO keeps the same motivation, but uses simpler first-order machinery.

PPO with a KL Penalty

One way to soften the TRPO constraint is to replace it with a penalty term:

\[ \max_\theta \quad \mathbb{E}_t \left[ r_t(\theta)\,\hat A_t \right] - \beta\, \bar D_{\mathrm{KL}} \left( \pi_{\theta_{\mathrm{old}}} \,\middle\|\, \pi_\theta \right). \]

Here \(\beta>0\) controls how strongly policy movement is penalized.

Compared with TRPO:

PPO does not explicitly calculate the natural-gradient direction \(F^{-1}g\). Instead, \(\beta\) can be adapted from the KL divergence actually produced by the update.

Adaptive KL Penalty

Suppose the desired average KL change is

\[ D_{\mathrm{target}} = 0.01. \]

After optimizing on the current rollout batch, measure

\[ D_{\mathrm{actual}} = \bar D_{\mathrm{KL}} \left( \pi_{\theta_{\mathrm{old}}} \,\middle\|\, \pi_{\theta_{\mathrm{new}}} \right). \]

The penalty coefficient can then be adjusted according to how far the measured KL is from the desired value. One simple rule is

\[ \beta \leftarrow \begin{cases} 2\beta, & D_{\mathrm{actual}} > 1.5 D_{\mathrm{target}},\\[6pt] \beta/2, & D_{\mathrm{actual}} < D_{\mathrm{target}}/1.5,\\[6pt] \beta, & \text{otherwise}. \end{cases} \]

This creates a tolerance band around the target KL. For \(D_{\mathrm{target}}=0.01\):

The goal is not to force the KL to equal the target exactly. The adaptive penalty instead keeps policy updates in a useful neighborhood around the desired scale while retaining ordinary first-order minibatch optimization.

The Clipped Objective

The more widely used PPO variant replaces the explicit KL penalty with a clipped importance ratio. Define

\[ L^{\mathrm{clip}}(\theta) = \mathbb{E}_t \left[ \min\!\Big( r_t(\theta)\,\hat A_t, \operatorname{clip}\!\big(r_t(\theta),1-\epsilon,1+\epsilon\big)\,\hat A_t \Big) \right]. \]

This objective can be understood as a pessimistic lower bound on the surrogate. For each sample,

\[ \min(x,y)\le x. \]

Taking \(x=r_t(\theta)\hat A_t\) gives

\[ L^{\mathrm{clip}}(\theta) \le L^{\mathrm{IS}}(\theta). \]

The clipping rule says: once the ratio moves beyond an acceptable range, PPO stops giving additional credit for pushing it farther in the same direction.

Positive-Advantage Case

If \(\hat A_t>0\), then the sampled action was better than expected, so we want to increase its probability.

\[ \min\!\Big( r_t\hat A_t,\, \operatorname{clip}(r_t,1-\epsilon,1+\epsilon)\hat A_t \Big) = \begin{cases} r_t\hat A_t, & r_t \le 1+\epsilon,\\[6pt] (1+\epsilon)\hat A_t, & r_t > 1+\epsilon. \end{cases} \]

So once the action probability has already increased by more than about \(1+\epsilon\), PPO gives no additional reward for increasing it further.

“I will only credit this improvement up to the clipping threshold.”

Negative-Advantage Case

If \(\hat A_t<0\), then the sampled action was worse than expected, so we want to decrease its probability.

\[ \min\!\Big( r_t\hat A_t,\, \operatorname{clip}(r_t,1-\epsilon,1+\epsilon)\hat A_t \Big) = \begin{cases} (1-\epsilon)\hat A_t, & r_t < 1-\epsilon,\\[6pt] r_t\hat A_t, & r_t \ge 1-\epsilon. \end{cases} \]

Here too, once the action probability has been reduced enough, PPO stops rewarding a more extreme change.

Why Clipping Helps

The clipped objective does not enforce a hard trust region, but it removes the incentive to keep pushing a sample's ratio arbitrarily far once that sample has already been changed “enough.”

This helps in several ways:

Practical PPO Loop

In practice, PPO typically works as follows:

  1. collect a batch of trajectories with \(\pi_{\theta_{\mathrm{old}}}\);
  2. estimate returns and advantages \(\hat A_t\), often with GAE;
  3. freeze the old log-probabilities;
  4. optimize the clipped objective for several epochs over shuffled minibatches;
  5. optionally monitor the KL divergence and stop early if it becomes too large;
  6. set \(\theta_{\mathrm{old}}\leftarrow\theta\), collect a fresh batch, and repeat.

PPO vs. TRPO

The relation between the two methods is straightforward:

PPO is usually easier to implement, easier to scale, and more computationally practical, which is why it became the default policy-gradient baseline in many settings.

Summary

PPO starts from the importance-sampled surrogate objective and modifies it so that repeated optimization on the same batch does not keep rewarding increasingly extreme policy-ratio changes.

The KL-penalty version does this with an explicit penalty term. The clipped version does it by directly capping the effective credit assigned to overly large ratio changes.

In both cases, the goal is the same: preserve the useful trust-region intuition of TRPO while using simple, scalable first-order optimization.