← Notes

Policy Gradient Methods

The Objective

The main objective is to optimize the policy parameters to maximize the expected cumulative reward:

\[ J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[R(\tau)\right] \tag{1} \label{eq:policy-objective} \]

Now how the hell do we optimize this objective?

Well, let's obtain the gradient w.r.t the policy parameters \(\theta\).

Vanilla Policy Gradient / REINFORCE Derivation

\[ \begin{aligned} \nabla_\theta J(\theta) &= \nabla_\theta \mathbb{E}_{\tau \sim \pi_\theta} \left[R(\tau)\right] \\[6pt] &= \nabla_\theta \int_{\tau} p(\tau \mid \theta) R(\tau)\,d\tau && \text{expand expectation} \\[6pt] &= \int_{\tau} \nabla_\theta p(\tau \mid \theta) R(\tau)\,d\tau && \text{bring gradient into integral} \\[6pt] &= \int_{\tau} p(\tau \mid \theta) \nabla_\theta \log p(\tau \mid \theta) R(\tau)\,d\tau && \text{log derivative trick} \end{aligned} \]
\[ \boxed{ \nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \nabla_\theta \log p(\tau \mid \theta) R(\tau) \right] } \tag{2} \]

\[ p(\tau \mid \theta) = \underbrace{p(s_0)}_{\text{initial state}} \; \prod_{t=0}^{T} \underbrace{\pi_\theta(a_t \mid s_t)}_{\text{policy}} \; \underbrace{P(s_{t+1} \mid s_t, a_t)}_{\text{environment dynamics}} \]
\[ \log p(\tau \mid \theta) = \log p(s_0) + \sum_{t=0}^{T} \log \pi_\theta(a_t \mid s_t) + \sum_{t=0}^{T} \log P(s_{t+1} \mid s_t, a_t) \]
\[ \begin{aligned} \nabla_\theta \log p(\tau \mid \theta) &= \nabla_\theta \left[ \log p(s_0) + \sum_{t=0}^{T} \log \pi_\theta(a_t \mid s_t) + \sum_{t=0}^{T} \log P(s_{t+1} \mid s_t, a_t) \right] \\[8pt] &= \underbrace{\nabla_\theta \log p(s_0)}_{0} + \sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t \mid s_t) + \underbrace{ \nabla_\theta \sum_{t=0}^{T} \log P(s_{t+1} \mid s_t, a_t) }_{0} \\[8pt] &= \sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t \mid s_t) \end{aligned} \]

Thus, plugging back into (2):

\[ \boxed{ \nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t \mid s_t) R(\tau) \right] } \]

The policy gradient is written as an expectation over all possible trajectories. In practice, however, we cannot enumerate every possible trajectory exactly. Instead, we can execute the policy in the environment and collect a finite number of trajectories sampled from the policy.

We can then approximate the expectation using the sample mean. By the law of large numbers, as the number of independent samples increases, the sample mean approaches the true expected value:

\[ \hat{\mu} = \frac{1}{N} \sum_{i=1}^{N} X_i \longrightarrow \mathbb{E}[X]. \]

In our case, each sampled trajectory gives us one sampled trajectory reward. Therefore, if we collect a set of trajectories \(D = \{\tau_i\}_{i=1}^{N}\) by executing the current policy, we can estimate the policy gradient by averaging over those trajectories:

\[ \nabla_\theta J(\theta) \approx \hat{g} = \frac{1}{|D|} \sum_{\tau \in D} \sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t \mid s_t) R(\tau). \]

This gives us an unbiased gradient estimator of the true policy gradient.

Interpreting the Update

The policy parameters \(\theta\) are then updated to increase the expected return under the trajectory distribution induced by the policy.

\[ \nabla_\theta \log \pi_\theta(a_t \mid s_t)\,\Phi_t \]

The term \(\nabla_\theta \log \pi_\theta(a_t \mid s_t)\) tells us:

Change \(\theta\) in the direction that would make the sampled action \(a_t\) more likely under the policy at state \(s_t\).

Then \(\Phi_t\) determines the sign and strength of that instruction.

In the simplest case, \(\Phi_t\) may be the trajectory return \(R(\tau)\). Therefore:

\[ \nabla_\theta \log \pi_\theta(a_t \mid s_t)\,R(\tau) \]

Gradient Ascent on Expected Return

\[ \theta \leftarrow \theta + \alpha \nabla_\theta J(\theta) \]

So, at a high level, the policy gradient attempts to:

When the full trajectory return \(R(\tau)\) is used, every action in the trajectory receives the same scalar weighting: actions taken early and late in the trajectory are all reinforced or discouraged according to the same final return. We can do better by choosing a more informative \(\Phi_t\).