Proximal Policy Optimization (PPO)
Overview
Proximal Policy Optimization (PPO) keeps the core trust-region intuition of TRPO, but avoids solving a constrained second-order optimization problem at every update.
Instead of explicitly computing a natural-gradient step and enforcing a hard KL constraint, PPO uses a first-order objective designed to discourage the new policy from moving too far away from the old policy.
Starting Point: The Importance-Sampled Surrogate
Recall the surrogate objective built from data collected under the old policy \(\pi_{\theta_{\mathrm{old}}}\):
The ratio \(r_t(\theta)\) tells us how the new policy changes the probability of the sampled action:
- \(r_t(\theta)=1\): no change;
- \(r_t(\theta)>1\): the sampled action becomes more likely;
- \(r_t(\theta)<1\): the sampled action becomes less likely.
This objective is the natural starting point because it reweights old-policy samples so they can be used to estimate a quantity associated with the new policy.
Why TRPO Was Introduced
If we optimize \(L^{\mathrm{IS}}(\theta)\) too aggressively, the current policy can move far from the policy that generated the batch. Then the fixed data and fixed advantage estimates become stale, and the optimization target becomes unreliable.
TRPO addresses this by solving
PPO keeps the same motivation, but uses simpler first-order machinery.
PPO with a KL Penalty
One way to soften the TRPO constraint is to replace it with a penalty term:
Here \(\beta>0\) controls how strongly policy movement is penalized.
Compared with TRPO:
- TRPO locally approximates the objective and KL constraint, computes a natural-gradient direction, scales the step, and performs a line search.
- PPO-KL directly optimizes the penalized objective with ordinary minibatch first-order optimization over multiple epochs.
PPO does not explicitly calculate the natural-gradient direction \(F^{-1}g\). Instead, \(\beta\) can be adapted from the KL divergence actually produced by the update.
Adaptive KL Penalty
Suppose the desired average KL change is
After optimizing on the current rollout batch, measure
The penalty coefficient can then be adjusted according to how far the measured KL is from the desired value. One simple rule is
This creates a tolerance band around the target KL. For \(D_{\mathrm{target}}=0.01\):
- if \(D_{\mathrm{actual}}>0.015\), increase \(\beta\), making future updates more strongly penalized;
- if \(D_{\mathrm{actual}}<0.0067\), decrease \(\beta\), allowing larger policy changes;
- otherwise, leave \(\beta\) unchanged.
The goal is not to force the KL to equal the target exactly. The adaptive penalty instead keeps policy updates in a useful neighborhood around the desired scale while retaining ordinary first-order minibatch optimization.
The Clipped Objective
The more widely used PPO variant replaces the explicit KL penalty with a clipped importance ratio. Define
This objective can be understood as a pessimistic lower bound on the surrogate. For each sample,
Taking \(x=r_t(\theta)\hat A_t\) gives
The clipping rule says: once the ratio moves beyond an acceptable range, PPO stops giving additional credit for pushing it farther in the same direction.
Positive-Advantage Case
If \(\hat A_t>0\), then the sampled action was better than expected, so we want to increase its probability.
So once the action probability has already increased by more than about \(1+\epsilon\), PPO gives no additional reward for increasing it further.
“I will only credit this improvement up to the clipping threshold.”
Negative-Advantage Case
If \(\hat A_t<0\), then the sampled action was worse than expected, so we want to decrease its probability.
Here too, once the action probability has been reduced enough, PPO stops rewarding a more extreme change.
Why Clipping Helps
The clipped objective does not enforce a hard trust region, but it removes the incentive to keep pushing a sample's ratio arbitrarily far once that sample has already been changed “enough.”
This helps in several ways:
- it reduces the damage from noisy or high-variance advantage estimates;
- it discourages excessively large policy updates on a fixed batch;
- it makes multiple epochs of minibatch optimization much more stable.
Practical PPO Loop
In practice, PPO typically works as follows:
- collect a batch of trajectories with \(\pi_{\theta_{\mathrm{old}}}\);
- estimate returns and advantages \(\hat A_t\), often with GAE;
- freeze the old log-probabilities;
- optimize the clipped objective for several epochs over shuffled minibatches;
- optionally monitor the KL divergence and stop early if it becomes too large;
- set \(\theta_{\mathrm{old}}\leftarrow\theta\), collect a fresh batch, and repeat.
PPO vs. TRPO
The relation between the two methods is straightforward:
- TRPO solves a locally constrained optimization problem using second-order information.
- PPO uses a simpler first-order objective that approximately enforces the same “don’t move too far” principle.
PPO is usually easier to implement, easier to scale, and more computationally practical, which is why it became the default policy-gradient baseline in many settings.
Summary
PPO starts from the importance-sampled surrogate objective and modifies it so that repeated optimization on the same batch does not keep rewarding increasingly extreme policy-ratio changes.
The KL-penalty version does this with an explicit penalty term. The clipped version does it by directly capping the effective credit assigned to overly large ratio changes.
In both cases, the goal is the same: preserve the useful trust-region intuition of TRPO while using simple, scalable first-order optimization.