Baselines
Recall the policy gradient estimator:
This estimator is unbiased, but it can have very high variance, especially when the return comes directly from the environment. The baseline's job is variance reduction.
Given a baseline value \(b\), we can form the estimator
Intuitively, we want to increase the probability of trajectories whose return is better than average and decrease the probability of trajectories whose return is worse than average.
This estimator remains unbiased as long as the baseline does not depend on the sampled action.
Why the Baseline Does Not Change the Expected Gradient
If \(b\) is constant with respect to the sampled action, then the extra term has zero expectation:
So subtracting the baseline changes the variance, not the mean. To pull \(b\) outside the sum, it must be constant with respect to the sampled action.
Reward-to-Go and State-Dependent Baselines
The current objective makes each action interact with rewards at all times. What we really want is for an action to interact with the future rewards that come after it.
A natural state-dependent baseline is the expected return from that state:
This is exactly the quantity we want: \(V^\pi(s_t)\) tells us how much reward the current policy is expected to obtain on average from state \(s_t\).
Estimating the Value Function
The key practical question is how to estimate \(V^\pi(s_t)\). Here are two approaches using a neural network value function.
1) Monte Carlo Value Estimate
- Initialize a value network \(V_{\phi_0}^{\pi}\).
- Collect trajectories \(\tau_1, \ldots, \tau_m\).
- Regress the value network onto the empirical return.
The network learns to predict the total reward observed after being in a given state.
2) Temporal-Difference Value Estimate
- Initialize a value network \(V_{\phi_0}^{\pi}\).
- Collect transition data \(\{s, s', r\}\).
- Fit the network to a bootstrapped target.
Fitted value iteration:
This makes the value at state \(s\) match the immediate reward plus the value of the next state according to the current estimate.
Monte Carlo vs. TD for Estimating \(V^\pi\)
The Monte Carlo estimator is unbiased, but it is often not very sample efficient. The TD estimator is usually more sample efficient because it exploits Bellman structure.
By computing target values from this Bellman relationship and regressing onto them, TD value estimates can extract a learning target from every transition.
Why TD Can Be More Sample Efficient
- TD reuses previously learned information. Suppose the agent has already learned \(V(s_{t+1}) \approx 100\). After seeing a single transition from \(s_t\) to \(s_{t+1}\) with reward \(2\), TD can infer \(V(s_t) \approx 2 + \gamma 100\). It reuses knowledge already stored in the value function.
- TD targets usually have lower variance. A Monte Carlo return contains randomness from every future transition: \[ G_t = r_t + \gamma r_{t+1} + \gamma^2 r_{t+2} + \cdots. \] The TD target \[ r_t + \gamma V(s_{t+1}) \] still contains randomness from the immediate transition, but replaces the rest of the random trajectory with an average prediction.
Example. Suppose entering \(s_{t+1}\) produces either \(+100\) or \(-100\) with equal probability. Then \(V(s_{t+1}) = 0\). Monte Carlo targets from \(s_t\) may swing between approximately \(+100\) and \(-100\), so many episodes are needed for those returns to average toward zero. Once TD has learned that \(V(s_{t+1}) \approx 0\), every transition into \(s_{t+1}\) gets a target near \(r_t + \gamma \cdot 0\), which is much less noisy.
The tradeoff is that the TD estimate is biased whenever the bootstrap value \(V(s_{t+1})\) is wrong. This is the classic bias-variance tradeoff: less variance and faster reuse of data in exchange for bootstrapping bias. This does not necessarily mean TD converges to the wrong answer, because the source of bias changes over time as the value estimate improves.
Good Choices for the Baseline
- Constant baseline: \[ b = \mathbb{E}[R(\tau)] \approx \frac{1}{m} \sum_{i=1}^{m} R(\tau^{(i)}). \]
- Optimal constant baseline: \[ b = \frac{ \sum_i \left\|\nabla_\theta \log p\bigl(\tau^{(i)};\theta\bigr)\right\|^2 R(\tau^{(i)}) }{ \sum_i \left\|\nabla_\theta \log p\bigl(\tau^{(i)};\theta\bigr)\right\|^2 }. \] This minimizes variance among constant baselines, although in practice it is usually only a minor improvement.
- State-dependent expected return: \[ b(s_t) = V^{\pi}(s_t). \] This compares the realized return after taking an action to the return the current policy would have expected from that state.