← Notes

Actor-Critic Methods

Recall the likelihood-ratio policy-gradient estimator with a value-function baseline:

\[ \hat g = \frac{1}{m} \sum_{i=1}^{m} \sum_{t=0}^{H_i-1} \nabla_\theta \log \pi_\theta\!\left(a_t^{(i)} \mid s_t^{(i)}\right) \left( \sum_{k=t}^{H_i-1} R\!\left(s_k^{(i)},a_k^{(i)}\right) - V^\pi\!\left(s_t^{(i)}\right) \right). \]

The return-to-go inside the parentheses is a single-rollout estimate of \(Q^\pi(s_t,a_t)\), while \(V^\pi(s_t)\) is the expected return from the current state.

\[ Q^\pi(s,a) = \mathbb{E}\!\left[ r_0 + r_1 + r_2 + \cdots \mid s_0=s,\,a_0=a \right]. \]

If we estimate \(Q^\pi\) using the full Monte Carlo return from a single rollout, each sample can have high variance because every future transition introduces additional randomness.

Reducing Variance

One way to reduce the influence of distant, uncertain rewards is discounting:

\[ Q^{\pi,\gamma}(s,a) = \mathbb{E}\!\left[ r_0 + \gamma r_1 + \gamma^2 r_2 + \cdots \mid s_0=s,\,a_0=a \right]. \]

A more important variance-reduction step is function approximation. Rather than waiting for the entire future trajectory, we replace part of the future return with a learned estimate. That learned estimator is the critic.

\[ \begin{aligned} Q^{\pi,\gamma}(s,a) &= \mathbb{E}\!\left[ r_0 + \gamma r_1 + \gamma^2 r_2 + \cdots \mid s_0=s,\,a_0=a \right] \\ &= \mathbb{E}\!\left[ r_0 + \gamma V^\pi(s_1) \mid s_0=s,\,a_0=a \right] \\ &= \mathbb{E}\!\left[ r_0 + \gamma r_1 + \gamma^2 V^\pi(s_2) \mid s_0=s,\,a_0=a \right] \\ &= \cdots \end{aligned} \]

We can therefore use any number of real future rewards before bootstrapping from the critic. A generic \(k\)-step target is

\[ \hat Q_t^{(k)} = \sum_{l=0}^{k-1} \gamma^l r_{t+l} + \gamma^k V_\phi(s_{t+k}). \]

Smaller \(k\) relies more heavily on the critic and generally reduces variance at the cost of more bootstrapping bias. Larger \(k\) uses more observed rewards and approaches the Monte Carlo return.

The Actor and the Critic

The actor is the policy \(\pi_\theta(a\mid s)\). The critic is a learned approximation such as \(V_\phi(s)\) or \(Q_\phi(s,a)\). The critic does not choose actions directly; it evaluates the actor's behavior and supplies the signal used to improve the policy.

In the common state-value formulation, the critic predicts \(V_\phi(s_t)\), and the actor is updated using an estimate of how much better or worse the sampled action performed relative to that prediction. This quantity is the advantage.

This leads directly to advantage estimation.