Advantage Estimation
The advantage function measures how much better or worse it is to take a particular action than to follow the policy's average behavior from the same state:
In policy-gradient methods, this is the signal we want to multiply by \(\nabla_\theta \log \pi_\theta(a_t\mid s_t)\). Positive advantage increases the probability of the sampled action; negative advantage decreases it.
Why Estimate the Advantage?
We could estimate \(Q^\pi(s_t,a_t)\) from the full Monte Carlo return and subtract \(V^\pi(s_t)\), but the full return can have high variance. Instead, actor-critic methods use the critic to bootstrap and construct lower-variance estimates of the advantage.
Different rollout lengths give different estimates. Short estimates use more of the critic; long estimates use more real rewards. Generalized Advantage Estimation combines these choices rather than committing to a single horizon.
One-Step Temporal-Difference Error
Define the one-step TD error:
This is a one-step estimate of the advantage:
It asks: was the observed reward plus the estimated value of the next state better or worse than what the critic predicted for the current state?
Generalized Advantage Estimation
Generalized Advantage Estimation (GAE) forms a \(\lambda\)-weighted combination of multi-step temporal-difference estimates. Equivalently, it can be written as an exponentially weighted sum of future TD errors.
For a finite trajectory, this expands to
Each future TD error contributes additional evidence about whether the action at time \(t\) was better or worse than expected, with its influence decaying geometrically by \(\gamma\lambda\).
Recursive Form
The same GAE quantity can be computed efficiently backward through the rollout:
with the terminal condition
at a true terminal state.
The Role of \(\lambda\)
The parameter \(\lambda\) determines how strongly future TD errors contribute to the current advantage estimate.
which reduces to a one-step TD estimate: lower variance, but greater dependence on the critic.
moving the estimate toward a Monte Carlo-style return. Thus, \(\lambda\) controls the bias-variance tradeoff between short-horizon bootstrapping and long-horizon sampled returns.