Policy Gradient Methods
The Objective
The main objective is to optimize the policy parameters to maximize the expected cumulative reward:
Now how the hell do we optimize this objective?
Well, let's obtain the gradient w.r.t the policy parameters \(\theta\).
Vanilla Policy Gradient / REINFORCE Derivation
Thus, plugging back into (2):
The policy gradient is written as an expectation over all possible trajectories. In practice, however, we cannot enumerate every possible trajectory exactly. Instead, we can execute the policy in the environment and collect a finite number of trajectories sampled from the policy.
We can then approximate the expectation using the sample mean. By the law of large numbers, as the number of independent samples increases, the sample mean approaches the true expected value:
In our case, each sampled trajectory gives us one sampled trajectory reward. Therefore, if we collect a set of trajectories \(D = \{\tau_i\}_{i=1}^{N}\) by executing the current policy, we can estimate the policy gradient by averaging over those trajectories:
This gives us an unbiased gradient estimator of the true policy gradient.
Interpreting the Update
The policy parameters \(\theta\) are then updated to increase the expected return under the trajectory distribution induced by the policy.
The term \(\nabla_\theta \log \pi_\theta(a_t \mid s_t)\) tells us:
Change \(\theta\) in the direction that would make the sampled action \(a_t\) more likely under the policy at state \(s_t\).
Then \(\Phi_t\) determines the sign and strength of that instruction.
In the simplest case, \(\Phi_t\) may be the trajectory return \(R(\tau)\). Therefore:
Gradient Ascent on Expected Return
So, at a high level, the policy gradient attempts to:
- increase the probability of trajectories associated with higher return,
- decrease the probability of trajectories associated with lower return.
When the full trajectory return \(R(\tau)\) is used, every action in the trajectory receives the same scalar weighting: actions taken early and late in the trajectory are all reinforced or discouraged according to the same final return. We can do better by choosing a more informative \(\Phi_t\).