Entropy, KL Divergence, and Cross-Entropy

Reinforcement learning is full of distributions: a policy over actions, a target over labels, an old policy and a new one. Three quantities measure them. Entropy scores how spread out one distribution is, KL divergence scores how far one sits from another, and cross-entropy is the sum of the two. Every policy regularizer and trust region in modern RL is built from these.

Entropy: how uncertain is one distribution

For a discrete distribution $p$, the Shannon entropy is

$$H(p) = -\sum_x p(x)\,\log p(x) = \mathbb{E}_{x\sim p}\!\left[-\log p(x)\right] \ge 0 .$$

It is the average surprise, in nats when $\log$ is natural. A point mass has $H = 0$; the uniform distribution over $k$ outcomes maximizes it at $\log k$. For a continuous density the sum becomes an integral, the differential entropy $h(p) = -\int p(x)\log p(x)\,dx$, which for a Gaussian is¹

$$h\big(\mathcal{N}(\mu,\sigma^2)\big) = \tfrac{1}{2}\log\!\big(2\pi e\,\sigma^2\big) .$$

Only the variance enters: a wider Gaussian is more uncertain.

KL divergence: how far apart are two

The Kullback-Leibler divergence of $p$ from $q$ is

$$D(q\,\|\,p) = \sum_x q(x)\,\log\frac{q(x)}{p(x)} = \mathbb{E}_{x\sim q}\!\left[\log\frac{q(x)}{p(x)}\right] .$$

By Jensen's inequality $D(q\,\|\,p) \ge 0$, with equality iff $q = p$ almost everywhere.² It is not a metric: it is asymmetric, $D(q\,\|\,p) \ne D(p\,\|\,q)$ in general, and obeys no triangle inequality. The expectation is taken under $q$, so $D(q\,\|\,p)$ penalizes placing $q$-mass where $p$ is small, and ignores regions $q$ never visits.

Cross-entropy ties them together

The cross-entropy of $q$ relative to $p$ is $H(q,p) = -\sum_x q(x)\log p(x)$. Splitting the log of the ratio gives the identity that connects all three:

$$H(q,p) = -\sum_x q(x)\log p(x) = \underbrace{-\sum_x q(x)\log q(x)}_{H(q)} + \underbrace{\sum_x q(x)\log\frac{q(x)}{p(x)}}_{D(q\,\|\,p)} .$$ $$\boxed{\,H(q,p) = H(q) + D(q\,\|\,p)\,}$$

Since $H(q)$ does not depend on $p$, minimizing cross-entropy over $p$ is exactly minimizing $D(q\,\|\,p)$. That is why the classification and behavior-cloning loss is written as a cross-entropy: with $q$ the fixed data (or expert) distribution, driving $H(q,p)$ down pulls the model $p$ toward $q$.

Why RL cares

Three uses, one toolkit.

• Entropy bonus. Maximum-entropy RL augments the reward with the policy entropy, optimizing $\mathbb{E}\big[\sum_t r_t + \alpha\,H(\pi(\cdot\mid s_t))\big]$ so the agent solves the task while staying as random as possible. This keeps exploration alive and yields more robust policies; Soft Actor-Critic makes the entropy term the center of the objective.³
• KL trust region. TRPO limits how far each update moves the policy by constraining $\mathbb{E}_s\big[D(\pi_{\text{old}}\,\|\,\pi_\theta)\big] \le \delta$, a trust region in distribution space rather than in parameters.
• Cross-entropy loss. Supervised heads, including behavior cloning of an expert, minimize cross-entropy, which by the identity above is fitting $p$ to the data distribution $q$ by KL.

Worked example: KL between two Gaussians

The KL of one 1D Gaussian from another has a closed form,¹

$$D\!\big(\mathcal{N}(\mu_1,\sigma_1^2)\,\|\,\mathcal{N}(\mu_2,\sigma_2^2)\big) = \log\frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (\mu_1-\mu_2)^2}{2\sigma_2^2} - \frac{1}{2} .$$

Take $q = \mathcal{N}(0,1)$ and $p = \mathcal{N}(1,1)$, so only the means differ by one. Then $\log(\sigma_2/\sigma_1) = \log 1 = 0$, and

$$D(q\,\|\,p) = 0 + \frac{1 + (0-1)^2}{2\cdot 1} - \frac{1}{2} = \frac{2}{2} - \frac{1}{2} = 0.5 .$$

With equal variances the reverse is identical, $D(p\,\|\,q) = 0.5$: the mean gap enters symmetrically. The asymmetry shows up once the widths differ. Compare a broad $q = \mathcal{N}(0,4)$ against a narrow reference $p = \mathcal{N}(0,1)$:

$$D(q\,\|\,p) = \log\tfrac{1}{2} + \frac{4 + 0}{2\cdot 1} - \tfrac{1}{2} = -0.693 + 2 - 0.5 = 0.807 ,$$ $$D(p\,\|\,q) = \log 2 + \frac{1 + 0}{2\cdot 4} - \tfrac{1}{2} = 0.693 + 0.125 - 0.5 = 0.318 .$$

Same pair of Gaussians, two different numbers. The forward direction $D(q\,\|\,p)$ is larger because $q$ has mass out in the tails where the narrow $p$ is nearly zero, and $\log(q/p)$ blows up there; the reverse averages over the narrow $p$, which never sees those tails.

Entropy and KL of two Gaussiansinteractive

μ = 1.00σ = 1.00

H(q) nats

1.419

D(q‖p)

0.500

D(p‖q)

0.500

H(q,p) = H(q)+D(q‖p)

1.919

|D(q‖p) − D(p‖q)|

0.000

Solid accent is q = N(μ, σ²); dashed is the reference p = N(0,1). The two divergences move independently: only μ = 0, σ = 1 sends both to zero.

The single case that zeroes both divergences is $\mu = 0,\ \sigma = 1$, where $q = p$. Everywhere else the two numbers differ, which is why the direction of the KL is a modeling choice, not a detail: a trust region on $D(\pi_{\text{old}}\,\|\,\pi_\theta)$ constrains something genuinely different from its reverse.

Footnotes

1. Cover & Thomas, Elements of Information Theory (2006). ↩ ↩²

2. MacKay, Information Theory, Inference, and Learning Algorithms (2003). ↩

3. Haarnoja et al., Soft Actor-Critic (2018). ↩