Backpropagation, Step by Step

Backpropagation is not a special algorithm for neural nets. It is reverse-mode automatic differentiation applied to one particular composite function: a scalar loss $L$ stacked on top of a layered network. A forward pass evaluates the network and caches each activation; a backward pass then pushes the loss gradient through those cached values, layer by layer, reusing the same weights the forward pass used. One backward sweep costs about one forward pass and returns $\partial L/\partial\theta$ for every parameter at once, which is exactly why training deep nets, and the policy and value networks of deep reinforcement learning, is tractable.¹²

The two passes

Write the network with pre-activations $\mathbf z^{l}$ and activations $\mathbf a^{l}$ . The forward pass is

\mathbf a^{0} = \mathbf x, \qquad \mathbf z^{l} = W^{l}\mathbf a^{l-1} + \mathbf b^{l}, \qquad \mathbf a^{l} = \sigma(\mathbf z^{l}),

ending in a scalar loss $L(\mathbf a^{L}, \mathbf y)$ . Define the error at layer $l$ as $\boldsymbol\delta^{l} = \partial L / \partial \mathbf z^{l}$ . The chain rule gives it a recurrence that runs from the output back to the input,

\boldsymbol\delta^{L} = \nabla_{\mathbf a}L \odot \sigma'(\mathbf z^{L}), \qquad \boldsymbol\delta^{l} = \big((W^{l+1})^{\top}\boldsymbol\delta^{l+1}\big)\odot\sigma'(\mathbf z^{l}),

and the parameter gradients fall out of each $\boldsymbol\delta^{l}$ directly,

\nabla_{W^{l}}L = \boldsymbol\delta^{l}\,(\mathbf a^{l-1})^{\top}, \qquad \nabla_{\mathbf b^{l}}L = \boldsymbol\delta^{l}.

The transpose $(W^{l+1})^{\top}$ sends the error backward along the same edges the forward pass used, $\sigma'(\mathbf z^{l})$ is the local derivative of the nonlinearity, and the outer product with the cached $\mathbf a^{l-1}$ turns a layer's error into its weight gradient. Nothing here is a heuristic; it is the chain rule, evaluated in the order that shares work.

A 2-2-1 network, by hand

Take two inputs, two $\tanh$ hidden units, one linear output, and the single-sample squared-error loss $L = \tfrac12(a^{2}-y)^{2}$ (the $\tfrac12$ makes $\partial L/\partial a^{2} = a^{2}-y$ ). Fix the weights, input, and target:

W^{1}=\begin{bmatrix}0.10 & 0.20\\ 0.30 & 0.40\end{bmatrix},\; \mathbf b^{1}=\begin{bmatrix}0.10\\ 0.20\end{bmatrix},\; W^{2}=\begin{bmatrix}0.50 & 0.60\end{bmatrix},\; b^{2}=0.20,\; \mathbf x=\begin{bmatrix}0.5\\ 0.8\end{bmatrix},\; y=1.

Forward. The hidden pre-activations and activations are

\mathbf z^{1}=\begin{bmatrix}0.10 & 0.20\\ 0.30 & 0.40\end{bmatrix}\begin{bmatrix}0.5\\ 0.8\end{bmatrix}+\begin{bmatrix}0.10\\ 0.20\end{bmatrix}=\begin{bmatrix}0.31\\ 0.67\end{bmatrix}, \qquad \mathbf a^{1}=\tanh\mathbf z^{1}=\begin{bmatrix}0.3004\\ 0.5850\end{bmatrix}.

The output and loss are

a^{2}=z^{2}=\begin{bmatrix}0.50 & 0.60\end{bmatrix}\begin{bmatrix}0.3004\\ 0.5850\end{bmatrix}+0.20=0.7012, \qquad L=\tfrac12(0.7012-1)^{2}=0.0446.

Backward, output layer. The output is linear, so $\sigma'(z^{2})=1$ and

\delta^{2}=(a^{2}-y)\cdot 1 = -0.2988, \qquad \nabla_{W^{2}}L=\delta^{2}(\mathbf a^{1})^{\top}=-0.2988\begin{bmatrix}0.3004 & 0.5850\end{bmatrix}=\begin{bmatrix}-0.0898 & -0.1748\end{bmatrix}.

Backward, hidden layer. For $\tanh$ , $\sigma'(z)=1-\tanh^{2}z=1-(a^{1})^{2}$ , so

(W^{2})^{\top}\delta^{2}=\begin{bmatrix}0.50\\ 0.60\end{bmatrix}(-0.2988)=\begin{bmatrix}-0.1494\\ -0.1793\end{bmatrix}, \qquad \sigma'(\mathbf z^{1})=\begin{bmatrix}0.9097\\ 0.6578\end{bmatrix},

\boldsymbol\delta^{1}=\begin{bmatrix}-0.1494\\ -0.1793\end{bmatrix}\odot\begin{bmatrix}0.9097\\ 0.6578\end{bmatrix}=\begin{bmatrix}-0.1359\\ -0.1179\end{bmatrix}, \qquad \nabla_{W^{1}}L=\boldsymbol\delta^{1}(\mathbf a^{0})^{\top}=\begin{bmatrix}-0.0680 & -0.1087\\ -0.0590 & -0.0943\end{bmatrix}.

One SGD step $W\leftarrow W-\eta\,\nabla_{W}L$ with $\eta=0.5$ drives this sample's loss from $0.0446$ down to $0.0003$ on the next forward pass.

Backpropagation through a 2-2-1 netinteractive

stateloss = n/aepoch = 0

1 forward · layer 1 ← next

2 forward · layer 2 + loss

3 backward · output

4 backward · hidden

5 apply · SGD update

press Step

Step through the forward and backward passes to see the numbers.

The same 2-2-1 network, computed live. Step forward (blue) to fill z and a at each layer, then backward (red) to compute the output error, the hidden error, and the weight gradients, with the substituted arithmetic in the panel. Apply runs one SGD update at eta = 0.5, and the loss-per-epoch trace shows the loss dropping. Auto loops through forward, backward, and update.

Why it scales

The backward recurrence touches each weight exactly once and reuses activations already cached in the forward pass, so computing the full gradient costs the same order as a single evaluation of the network, independent of how many parameters there are. That is the reverse-mode payoff: differentiate a scalar output with respect to millions of inputs in one sweep. Stack it with SGD and you can fit deep networks, and because a policy or value network is just another composite function ending in a scalar objective, the identical machinery is what makes deep reinforcement learning trainable.

Rumelhart, Hinton & Williams, Learning representations by back-propagating errors (Nature 1986). ↩
Baydin, Pearlmutter, Radul & Siskind, Automatic Differentiation in Machine Learning: a Survey (2018). ↩

The two passes

A 2-2-1 network, by hand

Why it scales

Footnotes