KL Divergence
Definition
For two probability distributions \(p\) and \(q\) defined over the same space, the Kullback-Leibler divergence from \(q\) to \(p\) is
For a continuous random variable, this is
For a discrete random variable, the equivalent form is
Interpretation
The KL divergence asks how differently \(q\) assigns probability to outcomes that are actually likely under \(p\). One way to read the definition is:
- Sample \(x\) according to \(p\).
- Compare the probability that \(p\) assigns to \(x\) with the probability that \(q\) assigns to \(x\).
- Take the logarithm of that probability ratio.
- Average this quantity over outcomes weighted by how likely they are under \(p\).
If \(q\) assigns similar probabilities to the outcomes that \(p\) commonly produces, then \(D_{\mathrm{KL}}(p\|q)\) is small. If \(q\) assigns very low probability to outcomes that are common under \(p\), then the KL divergence becomes large.
Why the Direction Matters
KL divergence is asymmetric:
The first argument determines which distribution supplies the weighting. In \(D_{\mathrm{KL}}(p\|q)\), outcomes are weighted according to \(p\), so the divergence is most sensitive to how \(q\) behaves on outcomes that matter under \(p\).
Connection to Entropy and Cross-Entropy
KL divergence can be rewritten in terms of entropy and cross-entropy:
where the entropy of \(p\) is
and the cross-entropy between \(p\) and \(q\) is
Therefore, KL divergence measures the extra expected surprise incurred when \(q\) is used to represent data generated according to \(p\), relative to using the true distribution \(p\) itself.
Connection to Policy Optimization
In policy-gradient methods, the distributions of interest are often two policies evaluated at the same state. If \(\pi_{\theta_{\text{old}}}\) is the policy before an update and \(\pi_\theta\) is the updated policy, then a common quantity is
This measures how much the new action distribution differs from the old one at state \(s\). Unlike Euclidean distance between parameter vectors, it measures change directly in policy space, which is the quantity that matters for controlling destructive policy updates.