Natural Gradient, TRPO, and PPO

Policy gradients tell you which way to nudge the parameters $\theta$, but not how that nudge is measured. Plain gradient ascent moves in the direction of steepest increase under Euclidean distance in $\theta$, a choice that has nothing to do with the policy the parameters encode. The natural gradient replaces that arbitrary ruler with one built from the policy itself, and TRPO and PPO are two ways of taking a controlled step under it.

Steepest ascent depends on the metric

The gradient $\nabla_\theta J$ is not "the" steepest direction: it is the steepest direction only if you agree to measure step size by $\|d\theta\|_2$. Formally,

$$\nabla_\theta J = \arg\max_{\|d\theta\|_2 = 1} \nabla_\theta J^\top d\theta ,$$

so the answer is welded to the Euclidean norm on parameters. That norm is not intrinsic. Rescale one logit, or reparameterize the network, and $\|d\theta\|_2$ changes meaning while the policy does not. Two parameter vectors a fixed Euclidean distance apart can encode nearly identical policies in one region and wildly different ones in another. Steepest ascent under such a ruler zigzags and crawls, exactly the pathology of an ill-conditioned problem.

The Fisher metric and the natural gradient

The fix is to measure distance between $\pi_\theta$ and $\pi_{\theta+d\theta}$ by how different the distributions are, using the KL divergence. To second order,

$$\mathrm{KL}\!\big(\pi_\theta \,\|\, \pi_{\theta+d\theta}\big) = \tfrac{1}{2}\, d\theta^\top F(\theta)\, d\theta + O(\|d\theta\|^3), \qquad F(\theta) = \mathbb{E}_{a\sim\pi_\theta}\!\big[\nabla_\theta \log\pi_\theta\,\nabla_\theta \log\pi_\theta^\top\big],$$

where $F$ is the Fisher information matrix, the expected outer product of the score.¹ It is the local metric that KL induces on parameter space. Asking for steepest ascent under this metric,

$$\arg\max_{\frac{1}{2} d\theta^\top F\, d\theta \,\le\, \varepsilon} \nabla_\theta J^\top d\theta \;\propto\; F^{-1}\nabla_\theta J ,$$

gives the natural gradient $\widetilde{\nabla} J = F^{-1}\nabla_\theta J$. Because KL between distributions does not depend on how those distributions are coordinatized, $F$ transforms so that $F^{-1}\nabla J$ is invariant to reparameterization: it points at the same policy change no matter how you labelled $\theta$. That is the property vanilla gradient ascent lacks. A KL trust region of radius $\varepsilon$ is the ellipse $\{d\theta : \tfrac{1}{2} d\theta^\top F\, d\theta \le \varepsilon\}$, and the natural gradient is the direction that climbs $J$ fastest inside it.

Worked example: a 3-action softmax

Take a softmax policy over three actions with logits $(\theta_1,\theta_2,0)$; the third logit is pinned at $0$, so $\theta=(\theta_1,\theta_2)$ has two free components. Then $\pi_a = e^{\theta_a}/\sum_b e^{\theta_b}$ and the objective is the expected immediate reward $J(\theta)=\sum_a \pi_a r_a$. Two derivatives are all we need. The softmax score is $\partial_{\theta_j}\log\pi_a = \delta_{aj}-\pi_j$, which gives

$$\frac{\partial J}{\partial \theta_j} = \pi_j\big(r_j - \bar r\big), \qquad \bar r = \sum_a \pi_a r_a ,$$

and the Fisher on the free coordinates is the restricted covariance of the score,

$$F = \operatorname{diag}(p) - p\,p^\top \;\Big|_{1,2} = \begin{bmatrix} p_1(1-p_1) & -p_1 p_2 \\[2pt] -p_1 p_2 & p_2(1-p_2) \end{bmatrix}.$$

Evaluate at $\theta=(0,0)$, so $p=(\tfrac13,\tfrac13,\tfrac13)$, with rewards $r=(2,0,0)$ and $\bar r = \tfrac23$. The vanilla gradient is

$$\nabla_\theta J = \Big(\tfrac13(2-\tfrac23),\; \tfrac13(0-\tfrac23)\Big) = \Big(\tfrac49,\, -\tfrac29\Big) \;\propto\; (2,\,-1).$$

The Fisher and its inverse are

$$F = \begin{bmatrix} \tfrac29 & -\tfrac19 \\[2pt] -\tfrac19 & \tfrac29 \end{bmatrix}, \qquad F^{-1} = \begin{bmatrix} 6 & 3 \\ 3 & 6 \end{bmatrix},$$

so the natural gradient is

$$\widetilde{\nabla} J = F^{-1}\nabla_\theta J = \begin{bmatrix} 6 & 3 \\ 3 & 6 \end{bmatrix}\!\begin{bmatrix} \tfrac49 \\[2pt] -\tfrac29 \end{bmatrix} = (2,\,0).$$

The two point about $26.6^\circ$ apart. The reward favors action 1 and is indifferent between actions 2 and 3, so the honest move raises $\theta_1$ alone and keeps $\pi_2=\pi_3$: that is exactly $\widetilde{\nabla}J=(2,0)$. The vanilla gradient instead pushes $\theta_2$ down as well, distorting the policy in a direction the rewards never asked for. Rescaling by $F^{-1}$ removes that distortion.

Vanilla vs natural gradient on a softmax policyinteractive

J(θ) over the free logits (drag θ)

r₁ = 2.0r₂ = 0.0r₃ = 0.0

KL radius δ = 0.20

π

(0.33, 0.33, 0.33)

J(θ)

0.667

∇J

(0.44, -0.22)

F⁻¹∇J

(2.00, 0.00)

angle ∠(∇J, F⁻¹∇J)

26.6°

TRPO dθ*

(1.34, 0.00)

Red is ∇J, green is F⁻¹∇J (drawn to a shared scale). The amber ellipse is the KL ball; the green dot is the TRPO step on its boundary.

TRPO: the largest step in the KL ball

Trust Region Policy Optimization makes this a full update rule. It maximizes the surrogate advantage $L(\theta)=\mathbb{E}\big[\tfrac{\pi_\theta(a\mid s)}{\pi_{\text{old}}(a\mid s)}\,A(s,a)\big]$ subject to a hard KL constraint,

$$\max_\theta\; L(\theta) \quad\text{s.t.}\quad \mathbb{E}_s\big[\mathrm{KL}(\pi_{\text{old}}\,\|\,\pi_\theta)\big] \le \delta .$$

Linearizing $L$ and using the quadratic KL model turns this into $\max\, g^\top d\theta$ s.t. $\tfrac12 d\theta^\top F\,d\theta \le \delta$, whose solution is the natural-gradient step scaled to sit on the trust-region boundary,²

$$\boxed{\; d\theta^\star = \sqrt{\dfrac{2\delta}{g^\top F^{-1} g}}\; F^{-1} g \;}$$

Continuing the example with $\delta = 0.01$: here $g^\top F^{-1} g = g^\top \widetilde{\nabla}J = \tfrac49\cdot 2 = \tfrac89$, so the step length is $\sqrt{2(0.01)/(8/9)} = 0.15$ and

$$d\theta^\star = 0.15\,(2,0) = (0.3,\,0), \qquad \tfrac12\, d\theta^{\star\top} F\, d\theta^\star = 0.01 = \delta ,$$

landing exactly on the KL ellipse, as the demo's green dot shows. TRPO backtracks along this direction until the true KL constraint and a surrogate-improvement check both hold, which is what earns its monotonic-improvement guarantee. The price is $F^{-1} g$: forming or even applying the Fisher inverse (via conjugate gradients on Fisher-vector products) is the expensive part.

PPO: clipping instead of a trust region

Proximal Policy Optimization keeps the trust-region intent but drops the machinery.³ Let $\rho(\theta)=\pi_\theta(a\mid s)/\pi_{\text{old}}(a\mid s)$ be the likelihood ratio. PPO maximizes a clipped surrogate,

$$L^{\text{CLIP}}(\theta) = \mathbb{E}\Big[\min\!\big(\rho\, A,\; \operatorname{clip}(\rho,\,1-\epsilon,\,1+\epsilon)\, A\big)\Big].$$

When an update would move $\rho$ outside $[1-\epsilon, 1+\epsilon]$, the clip flattens the objective there so its gradient vanishes and the incentive to keep moving disappears. This caps the per-sample policy change directly on the ratio, a crude stand-in for the KL ball, using only first-order gradients and no $F$ at all. PPO gives up TRPO's formal guarantee and its exact geometry, and in exchange runs with ordinary minibatch SGD, which is why it became the default.

Footnotes

1. Kakade, A Natural Policy Gradient (2001). ↩

2. Schulman et al., Trust Region Policy Optimization (2015). ↩

3. Schulman et al., Proximal Policy Optimization Algorithms (2017). ↩