Contents · Matrix Calculus & Autodiff

Part 0 · Matrix Calculus

Matrix Calculus: Gradients and Jacobians

Part 1 · Automatic Differentiation

Automatic Differentiation: Forward vs Reverse

Part 2 · Backpropagation

Backpropagation, Step by Step

Part 3 · Differentiable Simulation

Differentiating Through a Simulator

Part 1 · Automatic Differentiation

Automatic Differentiation: Forward vs Reverse

Automatic differentiation evaluates exact derivatives of a program by applying the chain rule to its computational graph, one primitive operation at a time. It is not symbolic differentiation, which manipulates formulas and can blow up in size, and it is not numerical differentiation, which perturbs inputs and pays a truncation and rounding cost. AD works at machine precision and comes in two sweeps of the same graph, forward and reverse.¹

The computational graph

Take $y = \sin(x_1) + x_1 x_2$ evaluated at $(x_1, x_2) = (1, 2)$ . Break it into primitives, giving intermediate variables $v_1,\dots,v_5$ :

\begin{aligned} v_1 &= x_1 = 1, & v_2 &= x_2 = 2, \\ v_3 &= \sin(v_1) = \sin 1 \approx 0.8415, & v_4 &= v_1 v_2 = 2, \\ v_5 &= v_3 + v_4 \approx 2.8415 &&= y . \end{aligned}

Every edge $v_i \to v_j$ carries a known local partial $\partial v_j / \partial v_i$ . Both modes reuse these same locals; they differ only in the direction they are chained.

Forward mode: propagate tangents

Pick a direction in input space and carry a tangent $\dot v_i = \partial v_i / \partial x_j$ alongside each value. Seed the input you want, $\dot x_1 = 1$ and $\dot x_2 = 0$ , then push tangents input-to-output by the chain rule $\dot v_j = \sum_{i \to j} (\partial v_j / \partial v_i)\, \dot v_i$ :

\begin{aligned} \dot v_1 &= 1, & \dot v_2 &= 0, \\ \dot v_3 &= \cos(v_1)\,\dot v_1 = \cos 1 \approx 0.5403, \\ \dot v_4 &= \dot v_1\, v_2 + v_1\, \dot v_2 = 1\cdot 2 + 1\cdot 0 = 2, \\ \dot v_5 &= \dot v_3 + \dot v_4 = \cos 1 + 2 \approx 2.5403 = \frac{\partial y}{\partial x_1} . \end{aligned}

One pass gives the derivative with respect to the seeded input only. To get $\partial y / \partial x_2$ you rerun with $\dot x_2 = 1$ . Forward mode is cheap when there are few inputs.

Reverse mode: propagate adjoints

Now run the values forward first, then sweep back carrying an adjoint $\bar v_i = \partial y / \partial v_i$ , the sensitivity of the output to each node. Seed the output, $\bar v_5 = 1$ , and push adjoints output-to-input by $\bar v_i = \sum_{i \to j} \bar v_j\, (\partial v_j / \partial v_i)$ :

\begin{aligned} \bar v_5 &= 1, \\ \bar v_3 &= \bar v_5 \tfrac{\partial v_5}{\partial v_3} = 1, & \bar v_4 &= \bar v_5 \tfrac{\partial v_5}{\partial v_4} = 1, \\ \bar v_2 &= \bar v_4\, v_1 = 1 &&= \frac{\partial y}{\partial x_2}, \\ \bar v_1 &= \bar v_3 \cos(v_1) + \bar v_4\, v_2 = \cos 1 + 2 \approx 2.5403 &&= \frac{\partial y}{\partial x_1} . \end{aligned}

Note $\bar v_1$ accumulates two contributions, one from each edge leaving $v_1$ . A single reverse pass returns the derivative with respect to every input at once, $\partial y / \partial x_1$ and $\partial y / \partial x_2$ together. Reverse mode is cheap when there is one scalar output and many inputs. This backward sweep is exactly backpropagation, the subject of the next article.

Forward vs reverse ADinteractive

seed

nodeopvaluev̇

v₁x₁11

v₂x₂··

v₃sin··

v₄×··

v₅+··

step 1/5 · forward · seed

v₁ = x₁ = 1

v̇₁ = 1 (seed)

Forward: one pass per input.

seed v̇₁=1 → ∂y/∂x₁ = 2.5403

the other input needs a second pass: n inputs ⇒ n passes.

The graph of y = sin(x1) + x1*x2 at (1, 2). Forward mode carries each node's value and tangent (v-dot) input-to-output; switch the seed to see it returns one derivative per pass. Reverse mode runs the forward pass for values, seeds v-bar = 1 at the output, then accumulates adjoints backward, returning both derivatives in one pass. The panel shows the arithmetic of the highlighted step.

Which mode, and why

For a function $f : \mathbb{R}^n \to \mathbb{R}^m$ with Jacobian $J \in \mathbb{R}^{m \times n}$ , forward mode seeded with $\dot x = e_j$ returns $J e_j$ , the $j$ -th column, so building all of $J$ costs $n$ passes. Reverse mode seeded with $\bar y = e_i$ returns $e_i^{\top} J$ , the $i$ -th row, so all of $J$ costs $m$ passes:

\text{forward} \sim n \text{ passes}, \qquad \text{reverse} \sim m \text{ passes} .

A machine-learning loss is the extreme case $m = 1$ with $n$ in the millions: one reverse pass yields the full gradient, while forward mode would need one pass per parameter. When instead the map has few inputs and many outputs, forward wins. Same locals, same graph; the only choice is which direction to chain.¹

Baydin et al., Automatic Differentiation in Machine Learning: a Survey (2018). ↩ ↩²

The computational graph

Forward mode: propagate tangents

Reverse mode: propagate adjoints

Which mode, and why

Footnotes