Symmetric Matrices, the Spectral Theorem, and Positive Definiteness

Among all square matrices, the symmetric ones, $A = A^\top$, have the cleanest structure of all. Their eigenvalues are real, their eigenvectors can be chosen orthonormal, and the whole matrix factors into a rotation times a diagonal stretch times the inverse rotation. This is not a curiosity: mass matrices, covariance matrices, graph Laplacians, and every product $A^\top A$ are symmetric, so the theory below is the theory of most matrices that arise from real problems.

The spectral theorem

If $A = A^\top$ is real and $n \times n$, then it has an orthonormal basis of eigenvectors with real eigenvalues. Equivalently, there is an orthogonal $Q$ (so $Q^\top Q = I$) and a diagonal $\Lambda$ with

$$A = Q\,\Lambda\,Q^\top = \sum_{i=1}^{n} \lambda_i\, q_i q_i^\top , \qquad \Lambda = \operatorname{diag}(\lambda_1,\dots,\lambda_n), \quad \lambda_i \in \mathbb{R}.$$

The eigenvalues are real because for an eigenpair $(\lambda, x)$ with $x \ne 0$,

$$\lambda\,\bar x^\top x = \bar x^\top A x = \overline{(A\bar x)}^{\top} x = \bar\lambda\,\bar x^\top x \;\Longrightarrow\; \lambda = \bar\lambda .$$

Eigenvectors of distinct eigenvalues are orthogonal, since

$$\lambda_i\, q_i^\top q_j = (A q_i)^\top q_j = q_i^\top A q_j = \lambda_j\, q_i^\top q_j \;\Longrightarrow\; (\lambda_i - \lambda_j)\, q_i^\top q_j = 0 .$$

So $A$ acts by resolving any $x$ into the orthonormal directions $q_i$ and scaling each by $\lambda_i$. Nothing rotates into anything else.

Quadratic forms measure curvature

Attach to $A$ the scalar function

$$f(x) = x^\top A x = \sum_{i,j} a_{ij}\, x_i x_j ,$$

its quadratic form. Substituting the spectral factorization and setting $y = Q^\top x$ (an orthonormal change of coordinates, so $\lVert y\rVert = \lVert x\rVert$) diagonalizes it completely:

$$f(x) = x^\top Q\Lambda Q^\top x = y^\top \Lambda y = \sum_{i=1}^{n} \lambda_i\, y_i^2 .$$

In the eigenbasis the form is just a weighted sum of squares. Its sign is therefore dictated entirely by the signs of the eigenvalues: $f$ curves upward along a $q_i$ with $\lambda_i > 0$ and downward where $\lambda_i < 0$.

Positive definiteness, four ways

$A$ is positive definite when $f(x) = x^\top A x > 0$ for every $x \ne 0$. For a symmetric $A$ the following are equivalent, and each is a usable test.¹

$$x^\top A x > 0 \ \ \forall x \ne 0 \iff \text{all } \lambda_i > 0 \iff \text{all pivots} > 0 \iff \text{all leading principal minors } d_k > 0 .$$

The eigenvalue test is immediate from $f = \sum \lambda_i y_i^2$. The pivot test reads the diagonal of elimination ($A = LDL^\top$ has $D_{ii} > 0$). The minor test is Sylvester's criterion: the determinants $d_k = \det A_{1:k,\,1:k}$ of the top-left $k \times k$ blocks are all positive. Geometrically, positive definite means the level sets $f(x) = \text{const}$ are ellipsoids around a single minimum: the graph is a bowl. If the eigenvalues have mixed signs the matrix is indefinite and the graph is a saddle; if some vanish it is only semidefinite, with flat directions.

A clean consequence: $A$ is positive definite exactly when it has a Cholesky factorization $A = L L^\top$ with $L$ lower triangular and positive diagonal, which is elimination specialized to the symmetric positive definite case and the workhorse for solving $A x = b$ in practice.²

Quadratic form: bowl vs saddleinteractive

a = 2.0b = 1.0c = 2.0

A

[2.0 1.0 ; 1.0 2.0]

λ₁ (larger)

3.00

λ₂ (smaller)

1.00

minor d₁ = a

2.00

minor d₂ = det

4.00 − 1.00 = 3.00

classification: positive definite

A worked example

Take

$$A = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}, \qquad A = A^\top .$$

The eigenvalues of a symmetric $\begin{bmatrix} a & b \\ b & c \end{bmatrix}$ are

$$\lambda = \frac{(a + c) \pm \sqrt{(a - c)^2 + 4 b^2}}{2} = \frac{4 \pm \sqrt{0 + 4}}{2} = \{\, 3,\ 1 \,\},$$

both positive, so $A$ is positive definite. The other three tests agree. The leading principal minors are

$$d_1 = 2 > 0, \qquad d_2 = \det A = (2)(2) - (1)(1) = 3 > 0 ,$$

and elimination gives pivots $2$ and $\tfrac{3}{2}$, both positive. The eigenvectors are $q_1 = \tfrac{1}{\sqrt2}(1,1)$ for $\lambda = 3$ and $q_2 = \tfrac{1}{\sqrt2}(1,-1)$ for $\lambda = 1$, orthogonal as promised, so

$$A = 3\, q_1 q_1^\top + 1\, q_2 q_2^\top , \qquad f(x) = 3 y_1^2 + y_2^2 ,$$

a bowl steeper along $(1,1)$ than along $(1,-1)$. Contrast this with

$$B = \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix}, \qquad \lambda = \frac{0 \pm \sqrt{0 + 4}}{2} = \{\, +1,\ -1 \,\} ,$$

whose eigenvalues have opposite signs. Here $d_2 = \det B = -1 < 0$ already fails Sylvester's test, and $f(x) = 2 x_1 x_2$ is a saddle: positive on the diagonal $x_1 = x_2$, negative on the anti-diagonal.

Why this matters

The forms that show up in simulation and estimation are symmetric by construction, so definiteness is the property that decides whether they behave. The joint-space mass matrix $M(q)$ is symmetric positive definite because kinetic energy $T = \tfrac12\,\dot q^\top M(q)\,\dot q$ is positive for any nonzero motion, which is exactly what lets forward dynamics solve $M \ddot q = \tau$ by Cholesky. Any Gram matrix $A^\top A$ is symmetric positive semidefinite, and positive definite precisely when $A$ has full column rank, which is what makes the normal equations of least squares solvable. And a critical point of a smooth function is a strict minimum exactly when its Hessian is positive definite: the same curvature test, one dimension at a time.

Footnotes

1. Strang, Introduction to Linear Algebra (5th ed., 2016). ↩

2. Golub & Van Loan, Matrix Computations (4th ed., 2013). ↩