Pontryagin and Hamilton-Jacobi-Bellman

Dynamic programming in discrete time gives the Bellman equation. Push the time step to zero and the same optimality splits into two continuous-time views: a pointwise condition on the control (Pontryagin), and a partial differential equation for the value function (Hamilton-Jacobi-Bellman). Both start from one object, the Hamiltonian, and both feed the sampled, model-free machinery of reinforcement learning.

The Hamiltonian and Pontryagin's principle

Take a system $\dot{x} = f(x, u)$ with running cost $c(x, u)$, and minimize $J = \int_0^T c(x, u)\, dt$ over controls $u(t) \in U$. Introduce a costate (or adjoint) $p(t) \in \mathbb{R}^n$, the continuous analogue of a Lagrange multiplier on the dynamics, and form the Hamiltonian

$$H(x, u, p) = c(x, u) + p^{\top} f(x, u) .$$

Pontryagin's Maximum Principle (here a minimum principle, since we minimize cost) states that along an optimal trajectory the control minimizes $H$ pointwise, and the costate runs backward under Hamilton's equations:¹

$$u^{\star}(t) = \operatorname*{arg\,min}_{u \in U} H(x, u, p), \qquad \dot{x} = \frac{\partial H}{\partial p} = f, \qquad \dot{p} = -\frac{\partial H}{\partial x} .$$

When the dynamics are control-affine, $f(x, u) = a(x) + B(x)\, u$, and $c$ does not depend on $u$, the Hamiltonian is linear in $u$:

$$H = c(x) + p^{\top} a(x) + \bigl(B(x)^{\top} p\bigr)^{\top} u .$$

Minimizing a linear function over a box $|u_i| \le 1$ pins each control to a vertex, $u_i^{\star} = -\operatorname{sign}\!\big((B^{\top} p)_i\big)$. The control is bang-bang: it sits at its bounds and jumps when the switching function $B^{\top} p$ changes sign.

The Hamilton-Jacobi-Bellman equation

The value function $V(x)$ is the optimal cost-to-go from $x$. The discrete Bellman recursion over a step $\Delta t$ reads

$$V(x) = \min_{u}\, \big[\, c(x, u)\,\Delta t + V\!\big(x + f(x,u)\,\Delta t\big) \,\big] .$$

Expand $V(x + f\,\Delta t) \approx V(x) + \nabla V(x)^{\top} f\, \Delta t$, cancel $V(x)$, and let $\Delta t \to 0$:

$$\min_{u}\, \Big[\, c(x, u) + \nabla V(x)^{\top} f(x, u) \,\Big] = 0 .$$

That is the (stationary) HJB equation, the continuous limit of the Bellman equation.² The bracket is exactly the Hamiltonian with $p = \nabla V$: the Pontryagin costate is the gradient of the value function along the optimal path, $p(t) = \nabla V\!\big(x^{\star}(t)\big)$. HJB is the field (every state), PMP is one characteristic curve through it.

Worked example: the minimum-time double integrator

Drive $\dot{x} = v,\ \dot{v} = u$ with $|u| \le 1$ to rest at the origin in minimum time. Here $c \equiv 1$ (each second costs one), so

$$H = 1 + p_x v + p_v u .$$

The costate equations are $\dot{p}_x = -\partial H/\partial x = 0$ and $\dot{p}_v = -\partial H/\partial v = -p_x$, so $p_x$ is constant and $p_v(t) = p_v(0) - p_x t$ is affine in time. Minimizing $H$ over $u$ gives $u^{\star} = -\operatorname{sign}(p_v)$, and an affine $p_v$ changes sign at most once: bang-bang with a single switch.

Under constant $u = \pm 1$, eliminating $t$ from $\dot{x}=v,\ \dot{v}=u$ gives parabolas $x = \tfrac{1}{2u} v^2 + \text{const}$. The two arcs that actually reach the origin form the switching curve

$$x = -\tfrac{1}{2}\, v\, |v| \quad\Longleftrightarrow\quad \begin{cases} x = -\tfrac{1}{2} v^2, & v > 0 \ \ (\text{terminal } u = -1), \\[2pt] x = +\tfrac{1}{2} v^2, & v < 0 \ \ (\text{terminal } u = +1). \end{cases}$$

With the switching function $\sigma(x, v) = x + \tfrac{1}{2} v |v|$, the optimal feedback law is $u^{\star} = -\operatorname{sign}(\sigma)$: above the curve ($\sigma > 0$) use $u = -1$, below it ($\sigma < 0$) use $u = +1$, and ride the curve into the origin.

Take the start $(x_0, v_0) = (1, 0)$. Then $\sigma = 1 > 0$, so $u = -1$ and the state follows $x = 1 - \tfrac{1}{2} v^2$. It meets the lower terminal arc $x = \tfrac{1}{2} v^2$ where

$$1 - \tfrac{1}{2} v^2 = \tfrac{1}{2} v^2 \;\Rightarrow\; v^2 = 1 \;\Rightarrow\; (x_{\text{sw}}, v_{\text{sw}}) = (0.5,\, -1) .$$

Switch to $u = +1$ and coast up the curve $x = \tfrac{1}{2}v^2$ to $(0,0)$. Each phase changes $v$ by one unit at rate $|u| = 1$, so

$$t_1 = |0 - (-1)| = 1, \qquad t_2 = |-1 - 0| = 1, \qquad T^{\star} = 2 .$$

Minimum-time bang-bang on the switching curveinteractive

phase plane (drag the start state)

start (x, v)

(1.00, 0.00)

region

above the curve, u = -1

control

u = -1, then u = +1

switch at

(0.50, -1.00)

phase 1 time

1.00

phase 2 time

1.00

total time T*

2.00

u = -1 region (above)

u = +1 region (below)

switching curve x = -½ v|v|

Push at full authority against the curve, ride it in: one switch, minimum time. This is the Pontryagin bang-bang solution.

From optimal control to reinforcement learning

HJB is model-based dynamic programming in continuous time: it needs $f$ and $c$ and returns the value field $V$. Reinforcement learning is what remains when the model is unknown. Sampling transitions in place of the expansion above turns HJB into a temporal-difference update, and the learned $\nabla V$ (or $Q$) is the critic that a policy improves against.² The Pontryagin side survives too: the costate $p = \nabla V$ is the pathwise gradient that adjoint methods and differentiable controllers backpropagate, and the bang-bang structure warns that optimal continuous-control policies can be genuinely discontinuous, not the smooth outputs a naive function approximator prefers.

Footnotes

1. Liberzon, Calculus of Variations and Optimal Control Theory (2012). ↩

2. Bertsekas, Dynamic Programming and Optimal Control. ↩ ↩²