The Score-Function Gradient

In reinforcement learning the objective is an expected return $J(\theta)=\mathbb{E}_{x\sim p_\theta}[f(x)]$, where the policy $p_\theta$ both generates the samples and carries the parameter you want to optimize. You cannot naively push $\nabla_\theta$ inside the average, because the distribution you are averaging against is itself moving with $\theta$.

The log-derivative trick

Write the expectation as an integral and differentiate:

$$\nabla_\theta \mathbb{E}_{x\sim p_\theta}[f(x)] = \nabla_\theta \int f(x)\,p_\theta(x)\,dx = \int f(x)\,\nabla_\theta p_\theta(x)\,dx.$$

The one identity that rescues this is $\nabla_\theta p_\theta = p_\theta\,\nabla_\theta\log p_\theta$. Substituting it turns the integral back into an expectation:

$$\nabla_\theta \mathbb{E}_{x\sim p_\theta}[f(x)] = \int f(x)\,p_\theta(x)\,\nabla_\theta\log p_\theta(x)\,dx = \mathbb{E}_{x\sim p_\theta}\!\big[f(x)\,\nabla_\theta\log p_\theta(x)\big].$$

The vector $\nabla_\theta\log p_\theta(x)$ is the score. This is the likelihood-ratio or score-function estimator, known in RL as REINFORCE.¹²

It never differentiates $f$

Read the right-hand side carefully: $f(x)$ enters only as a plain multiplicative weight, never inside a derivative. So $f$ can be a black box: a discontinuous reward, a raw game score, an entire non-differentiable simulator. You need only sample $x$, evaluate $f(x)$, and know the score of your own sampling distribution. Contrast the pathwise (reparameterization) gradient, which pushes $\nabla$ through $f$ and therefore requires $\mathrm{d}f/\mathrm{d}x$ to exist, the way a differentiable simulator does. The score-function estimator trades that requirement for sampling, so it works exactly where the pathwise gradient cannot go.

Baselines cut variance for free

Any constant $b$ may be subtracted from $f$ without biasing the estimator, because the score has mean zero:

$$\mathbb{E}_{x\sim p_\theta}[\nabla_\theta\log p_\theta(x)] = \int \nabla_\theta p_\theta(x)\,dx = \nabla_\theta\int p_\theta(x)\,dx = \nabla_\theta 1 = 0.$$

Hence

$$\mathbb{E}\big[(f(x)-b)\,\nabla_\theta\log p_\theta(x)\big] = \mathbb{E}\big[f(x)\,\nabla_\theta\log p_\theta(x)\big]$$

for every $b$. The mean is untouched, but the variance is not: choosing $b$ near the typical reward shrinks the fluctuating factor $(f(x)-b)$, and with it the spread of the estimate. A running mean reward is the standard cheap choice.

A worked example

Take a scalar Gaussian policy $p_\theta=\mathcal{N}(\theta,1)$ with reward $f(x)=x$. Its log-density and score are

$$\log p_\theta(x) = -\tfrac12 (x-\theta)^2 - \tfrac12\log 2\pi, \qquad \nabla_\theta\log p_\theta(x) = x-\theta.$$

So the estimator is $g = f(x)\,(x-\theta) = x\,(x-\theta)$, and its mean recovers the exact gradient:

$$\mathbb{E}[x(x-\theta)] = \mathbb{E}[x^2]-\theta\,\mathbb{E}[x] = (\theta^2+1) - \theta^2 = 1,$$

which matches $\nabla_\theta\mathbb{E}[x]=\nabla_\theta\theta=1$ directly. A single sample is noisy: at $\theta=2$, drawing $x=2.5$ gives $g = 2.5(0.5)=1.25$, and drawing $x=1.3$ gives $g=1.3(-0.7)=-0.91$. Each is far from $1$; averaged, they converge to it.

Now the variance. Let $u=x-\theta\sim\mathcal N(0,1)$ and $c=\theta-b$, so with baseline $b$ the estimator is $(u+c)\,u=u^2+cu$. Using $\mathbb{E}[u]=0$, $\mathrm{Var}(u^2)=2$, and $\mathbb{E}[u^3]=0$,

$$\mathrm{Var}\big[(f(x)-b)(x-\theta)\big] = \mathrm{Var}(u^2+cu) = 2 + c^2 = 2 + (\theta-b)^2.$$

Without a baseline, $b=0$ and $c=\theta$, so the variance is $2+\theta^2$; at $\theta=2$ that is $6$. The variance-minimizing baseline is $b=\theta$, which for $f(x)=x$ equals the mean reward $\mathbb{E}[x]=\theta$, driving the variance down to its floor of $2$. Same estimate of $1$, a third of the variance.

The demo below draws $N$ samples of $x\sim\mathcal N(\theta,1)$, forms both estimators over the identical draws, and plots the per-sample values on a shared gradient axis. Slide $\theta$ up and the no-baseline spread (amber) widens as $2+\theta^2$; the baselined spread (green) stays pinned near $2$. Both clouds center on the same true gradient, $1$.

Score-function gradient of N(θ, 1)interactive

θ = 2.00N = 60

baseline: onResample

estimate

1.187 vs 1.000

var, no baseline

6.85 (2+θ²=6.00)

var, + baseline

2.75 (≈2)

variance cut

2.5×

A score-function estimator has exactly two knobs. What distribution you sample from fixes the score you multiply by; what you subtract from the reward fixes the variance you pay. Neither touches the bias, and neither needs a derivative of the reward.

Footnotes

1. Williams, Simple statistical gradient-following algorithms for connectionist reinforcement learning (1992). ↩

2. Sutton, McAllester, Singh & Mansour, Policy Gradient Methods for Reinforcement Learning with Function Approximation (2000). ↩