← Notes

Advantage Estimation

The advantage function measures how much better or worse it is to take a particular action than to follow the policy's average behavior from the same state:

\[ A^\pi(s_t,a_t) = Q^\pi(s_t,a_t) - V^\pi(s_t). \]

In policy-gradient methods, this is the signal we want to multiply by \(\nabla_\theta \log \pi_\theta(a_t\mid s_t)\). Positive advantage increases the probability of the sampled action; negative advantage decreases it.

Why Estimate the Advantage?

We could estimate \(Q^\pi(s_t,a_t)\) from the full Monte Carlo return and subtract \(V^\pi(s_t)\), but the full return can have high variance. Instead, actor-critic methods use the critic to bootstrap and construct lower-variance estimates of the advantage.

Different rollout lengths give different estimates. Short estimates use more of the critic; long estimates use more real rewards. Generalized Advantage Estimation combines these choices rather than committing to a single horizon.

One-Step Temporal-Difference Error

Define the one-step TD error:

\[ \delta_t = r_t + \gamma V(s_{t+1}) - V(s_t). \]

This is a one-step estimate of the advantage:

\[ \delta_t \approx A^\pi(s_t,a_t). \]

It asks: was the observed reward plus the estimated value of the next state better or worse than what the critic predicted for the current state?

Generalized Advantage Estimation

Generalized Advantage Estimation (GAE) forms a \(\lambda\)-weighted combination of multi-step temporal-difference estimates. Equivalently, it can be written as an exponentially weighted sum of future TD errors.

\[ \hat A_t^{\mathrm{GAE}(\gamma,\lambda)} = \sum_{l=0}^{T-t-1} (\gamma\lambda)^l\,\delta_{t+l}. \]

For a finite trajectory, this expands to

\[ \hat A_t = \delta_t + \gamma\lambda\,\delta_{t+1} + (\gamma\lambda)^2\delta_{t+2} + \cdots. \]

Each future TD error contributes additional evidence about whether the action at time \(t\) was better or worse than expected, with its influence decaying geometrically by \(\gamma\lambda\).

Recursive Form

The same GAE quantity can be computed efficiently backward through the rollout:

\[ \hat A_t = \delta_t + \gamma\lambda\,\hat A_{t+1}, \]

with the terminal condition

\[ \hat A_T = 0 \]

at a true terminal state.

The Role of \(\lambda\)

The parameter \(\lambda\) determines how strongly future TD errors contribute to the current advantage estimate.

\[ \lambda = 0 \quad\Longrightarrow\quad \hat A_t = \delta_t, \]

which reduces to a one-step TD estimate: lower variance, but greater dependence on the critic.

\[ \lambda \rightarrow 1 \quad\Longrightarrow\quad \text{longer-horizon information receives more weight}, \]

moving the estimate toward a Monte Carlo-style return. Thus, \(\lambda\) controls the bias-variance tradeoff between short-horizon bootstrapping and long-horizon sampled returns.