Actor-Critic and Advantage Estimation

The policy gradient needs one number per action: how much better was $a_t$ than the policy's average at $s_t$. The raw Monte Carlo return answers this but with huge variance. Actor-critic learns a value function to serve as a baseline, and Generalized Advantage Estimation turns that baseline into a tunable dial between a low-variance biased estimate and a high-variance unbiased one.

The advantage function and baselines

The policy gradient can be written with any per-step weight $\Psi_t$,

$$\nabla_\theta J(\theta) = \mathbb{E}_{\tau\sim\pi_\theta}\!\left[\sum_t \nabla_\theta \log\pi_\theta(a_t\mid s_t)\,\Psi_t\right].$$

Subtracting a state-only baseline $b(s_t)$ from $\Psi_t$ leaves the expectation unchanged, because at each state

$$\mathbb{E}_{a\sim\pi}\!\big[\nabla_\theta\log\pi_\theta(a\mid s)\,b(s)\big] = b(s)\sum_a \nabla_\theta\pi_\theta(a\mid s) = b(s)\,\nabla_\theta 1 = 0 .$$

The variance-reducing choice is $b(s)=V^\pi(s)$, which turns the weight into the advantage¹

$$A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s), \qquad \mathbb{E}_{a\sim\pi}\!\big[A^\pi(s,a)\big] = 0 .$$

It measures how much action $a$ beats the state's average, and it is centered at zero, so good actions get pushed up and bad ones down.

Actor-critic: the critic is the baseline

The actor is the policy $\pi_\theta$; the critic is a learned $V_\phi \approx V^\pi$, fit by regressing on returns or TD targets. The actor update uses the critic to form an advantage estimate $\hat A_t$,

$$\nabla_\theta J(\theta) \approx \mathbb{E}\!\left[\sum_t \nabla_\theta \log\pi_\theta(a_t\mid s_t)\,\hat A_t\right].$$

The critic slashes variance relative to the raw return. The price is bias: when $V_\phi \ne V^\pi$, that error leaks into $\hat A_t$.

The TD error estimates the advantage

The one-step temporal-difference error is

$$\delta_t = r_t + \gamma\,V(s_{t+1}) - V(s_t).$$

If the critic is exact, $V = V^\pi$, then conditioning on $(s_t,a_t)$,

$$\mathbb{E}\big[\delta_t \mid s_t, a_t\big] = \underbrace{\mathbb{E}\big[r_t + \gamma V^\pi(s_{t+1}) \mid s_t,a_t\big]}_{Q^\pi(s_t,a_t)} - V^\pi(s_t) = A^\pi(s_t,a_t).$$

So $\delta_t$ is an unbiased one-sample estimate of the advantage, using only the current reward and one bootstrapped value. If the critic is wrong, $\delta_t$ inherits its bias.

Generalized Advantage Estimation

Summing TD errors gives the $k$-step advantage estimator

$$\hat A_t^{(k)} = \sum_{l=0}^{k-1} \gamma^l \delta_{t+l} = -V(s_t) + \sum_{l=0}^{k-1}\gamma^l r_{t+l} + \gamma^k V(s_{t+k}).$$

Small $k$ leans on the bootstrap $V(s_{t+k})$: low variance, high bias. Large $k$ leans on sampled rewards: high variance, low bias. GAE is their exponentially weighted average with decay $\lambda$,¹

$$\hat A_t^{\text{GAE}(\gamma,\lambda)} = (1-\lambda)\sum_{k=1}^{\infty} \lambda^{k-1}\,\hat A_t^{(k)} = \sum_{l=0}^{\infty} (\gamma\lambda)^l\,\delta_{t+l},$$

which collapses to a one-line backward recursion, $\hat A_t = \delta_t + \gamma\lambda\,\hat A_{t+1}$. The two ends are the familiar estimators:

$$\lambda = 0:\ \hat A_t = \delta_t \quad(\text{TD(0), low variance / biased}), \qquad \lambda = 1:\ \hat A_t = \sum_{l\ge 0}\gamma^l r_{t+l} - V(s_t)\quad(\text{Monte Carlo, unbiased / high variance}).$$

The weight $(\gamma\lambda)^l$ decays with an effective horizon $1/(1-\gamma\lambda)$ steps: that is how far downstream reward is trusted before the critic takes over.

Worked example

Take a six-step episode with $\gamma = 0.9$, a single reward $r = (0,0,0,0,0,1)$ at the end, and an imperfect critic $V = (0.4,0.5,0.6,0.7,0.8,0.9)$ with $V(\text{terminal}) = 0$. The TD errors $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$ are

$$\delta = \big(0.05,\ 0.04,\ 0.03,\ 0.02,\ 0.01,\ 0.10\big),$$

for instance $\delta_0 = 0.9(0.5) - 0.4 = 0.05$ and $\delta_5 = 1 + 0.9(0) - 0.9 = 0.10$. Now read the advantage at $t=0$ through the dial $\hat A_0 = \sum_{l=0}^{5}(\gamma\lambda)^l \delta_l$:

$$\lambda = 0:\quad \hat A_0 = \delta_0 = 0.050 ,$$ $$\lambda = 0.5\ (\gamma\lambda = 0.45):\quad \hat A_0 = 0.05 + 0.45(0.04) + 0.45^2(0.03) + \cdots + 0.45^5(0.10) = 0.078 ,$$ $$\lambda = 1\ (\gamma\lambda = 0.9):\quad \hat A_0 = \sum_{l=0}^{5}0.9^l\,\delta_l = 0.9^5 - 0.4 = 0.590 - 0.4 = 0.190 .$$

At $\lambda = 1$ the sum telescopes to the Monte Carlo return minus the baseline, confirming the identity. The effective horizon grows from $1/(1-0)=1$ step through $1/(1-0.45)=1.82$ to $1/(1-0.9)=10$: as $\lambda$ rises, the eventual reward at step $5$ flows further back into $\hat A_0$, climbing from $0.05$ (the critic's local bootstrap) to $0.190$ (the actual discounted return).

TD error, GAE advantage, and the bias-variance dialinteractive

δ ₜ (TD error)Aₜ (GAE)±1σ over noisy returns

γ = 0.90λ = 0.50 reward noise σ = 0.10

A₀(λ)

0.078

eff. horizon 1/(1−γλ)

1.82

std(A₀) · variance

0.100

|A₀(λ)−A₀(1)| · bias

0.112

λ = 0 is TD(0): short horizon, low variance, biased by the critic. λ = 1 is Monte Carlo: unbiased, but the whiskers blow up. Slide λ to trade one for the other.

Reading the two readouts together is the whole story: bias shrinks and variance grows along one knob. This is the dial inside PPO, which computes its clipped-objective advantages with GAE at roughly $\gamma \approx 0.99$ and $\lambda \approx 0.95$, buying most of Monte Carlo's low bias while the critic keeps the variance in check.¹ The same value baseline that made the plain policy gradient usable² becomes, through $\lambda$, a continuous choice about how far to trust the world versus the critic.

Footnotes

1. Schulman et al., High-Dimensional Continuous Control Using Generalized Advantage Estimation (2016). ↩ ↩² ↩³

2. Sutton & Barto, Reinforcement Learning: An Introduction (2018). ↩