Gradient Estimators
We rely on gradient estimators whenever an objective is defined as an expectation over a stochastic process and/or the exact gradient is too expensive or intractable to compute.
The ideal parameter update is typically written as
but in practice we often cannot compute the exact gradient \(\nabla_\theta J(\theta)\). We can instead replace it with an estimator:
Example: Minibatch Gradients
Suppose the true loss is the average over an entire dataset of \(N\) examples:
The exact gradient is
Computing this over millions of examples at every optimization step can be expensive. So instead, we sample a minibatch \(B\) and estimate the full-batch gradient with
If the minibatch is sampled properly, stochastic gradient descent gives an unbiased estimate of the full-batch gradient:
Each individual update is noisy, but its average direction is correct.
Although minibatching is motivated primarily by computational efficiency, the resulting gradient noise can also affect optimization behavior by helping the optimizer move through sharp or poor local regions and can sometimes improve generalization.
Bias
The estimator must match the correct gradient identity. The law of large numbers illustrates that a sample average will converge to its expectation, but it cannot fix an estimator that is biased or incorrectly derived.
An estimator is biased if, on average, it does not equal the true quantity being estimated. If \(\hat{g}\) estimates a true value \(g\), its bias is
Therefore,
Bias, Variance, and Mean Squared Error
For an estimator \(\hat{g}\) of a true value \(g\), define
Ultimately, we care about how far our estimator is from the true value. One measure of this estimation error is the mean squared error:
This error can arise for two different reasons: the estimator may fluctuate around its mean, or its mean may itself be shifted away from the truth. Accordingly, the MSE decomposes as
Thus, total estimation error consists of random variability and systematic bias.
What Averaging Can and Cannot Fix
Bias and variance behave differently under averaging. Suppose we draw \(N\) i.i.d. estimators \(\hat{g}_1, \ldots, \hat{g}_N\), each with
and average them:
Expectation of the Average
Every term has the same expectation \(\mu\), so averaging does not move the expectation. If \(\mu \neq g\), then
This is identical to the bias of a single sample. Averaging a million biased estimators does not remove the systematic offset.
Variance of the Average
Thus the variance shrinks proportionally to \(1/N\). More samples produce a tighter spread around the same mean, but they do not correct bias.
Connection to Baselines
Policy-gradient estimators can be unbiased while still having high variance. The purpose of a baseline is variance reduction without changing the expected policy gradient.