The LQR Problem

Most optimal-control problems have no closed form: you discretize, you iterate, you hope. The linear-quadratic regulator is the exception. Linear dynamics plus a quadratic cost collapse the whole infinite-horizon problem into one matrix equation, and the optimal policy comes out as a single constant gain. That closed form is why LQR is both a workhorse controller and the local model that iterative methods and reinforcement learning lean on.

Linear dynamics, quadratic cost

Take a discrete linear system with state $x_k \in \mathbb{R}^n$ and input $u_k \in \mathbb{R}^m$,

$$x_{k+1} = A x_k + B u_k ,$$

and charge a quadratic cost for both wandering off the origin and spending control effort,

$$J = \sum_{k=0}^{\infty} \left( x_k^\top Q\, x_k + u_k^\top R\, u_k \right), \qquad Q = Q^\top \succeq 0, \quad R = R^\top \succ 0 .$$

$Q$ says how much you dislike state error, $R$ how much you dislike effort. Their ratio, not their absolute size, sets the behavior: a large $Q/R$ buys an aggressive controller, a large $R/Q$ a lazy one.

The optimal policy is linear feedback

The optimal cost-to-go is quadratic, $V(x) = x^\top P x$, and minimizing the Bellman right-hand side over $u$ gives a policy that is linear in the state,

$$u_k = -K x_k ,$$

where $K$ and the matrix $P$ solve the discrete algebraic Riccati equation. The clean way to reach $P$ is to run the Riccati recursion backward from $P = 0$ until it stops moving,¹

$$P \;\leftarrow\; Q + A^\top P A - A^\top P B \left(R + B^\top P B\right)^{-1} B^\top P A ,$$ $$K = \left(R + B^\top P B\right)^{-1} B^\top P A .$$

This is exact value iteration, but on a quadratic $V$ it never leaves the class of quadratics, so a handful of matrix operations per step drive $P$ to its fixed point. No sampling, no function approximation. The controller is the gain $K$, and nothing else.

Worked example: balancing a pole

Take a pendulum of mass $m$ and length $L$, torque $u$ at the pivot, and let $\theta$ be the angle measured from upright. The nonlinear plant is $\ddot\theta = (g/L)\sin\theta + u/(mL^2)$. Linearizing about the upright equilibrium ($\sin\theta \approx \theta$) with state $x = [\theta, \dot\theta]^\top$ gives

$$A_c = \begin{bmatrix} 0 & 1 \\ g/L & 0 \end{bmatrix}, \qquad B_c = \begin{bmatrix} 0 \\ 1/(mL^2) \end{bmatrix}.$$

The upright point is unstable: $A_c$ has eigenvalues $\pm\sqrt{g/L}$, one of them positive, so any tilt grows. Euler-discretize with step $\Delta t$,

$$A = I + A_c\,\Delta t, \qquad B = B_c\,\Delta t .$$

Plug in $g = 9.8$, $L = 1$, $m = 1$, $\Delta t = 0.02$:

$$A = \begin{bmatrix} 1 & 0.02 \\ 0.196 & 1 \end{bmatrix}, \qquad B = \begin{bmatrix} 0 \\ 0.02 \end{bmatrix}.$$

Its eigenvalues are $1.063$ and $0.937$: one sits outside the unit circle, the discrete signature of the same instability. Now pick weights $Q = \operatorname{diag}(10, 1)$ (care mostly about the angle) and $R = 1$, and iterate the recursion to convergence. The gain comes out to

$$K \approx \begin{bmatrix} 19.48 & 6.22 \end{bmatrix}, \qquad u = -19.48\,\theta - 6.22\,\dot\theta .$$

Feeding this back moves the closed-loop eigenvalues of $A - BK$ to about $0.94$, both inside the unit circle: the tilt now decays instead of growing. The demo runs exactly this pipeline, then applies $u = -Kx$ (torque-limited) to the full nonlinear pendulum.

LQR balance · inverted penduluminteractive

angle weight Q₁₁ = 10effort weight R = 1.0

KickPauseReset

θ (from upright)

14.3°

u (torque)

+0.00

K

[19.48, 6.22]

correcting

Why LQR matters beyond linear systems

Real robots are not linear, but LQR is the atom that nonlinear methods are built from. iLQR / iterative LQG takes a nonlinear system, linearizes the dynamics and quadratizes the cost around the current trajectory, solves the resulting time-varying LQR problem for a local feedback correction, rolls it out, and repeats.² Each inner solve is exactly the Riccati recursion above, just run over a finite horizon with $A_k, B_k$ that change along the trajectory.

For reinforcement learning, LQR is the one continuous-control problem where the optimal policy, value function, and Bellman fixed point are all known in closed form. That makes it the standard sanity check: a policy-gradient or actor-critic method on linear-quadratic dynamics should recover $-Kx$ and the quadratic $V = x^\top P x$, and if it cannot, the bug is in the algorithm, not the task.

Footnotes

1. Bertsekas, Dynamic Programming and Optimal Control. ↩

2. Todorov & Li, A generalized iterative LQG method for locally-optimal feedback control of constrained nonlinear stochastic systems (ACC 2005). ↩