The Least-Squares Solution of a Linear System (With Collinearity)

In the previous lesson, we computed the least-squares solution

$$\boldsymbol{\hat\beta} = (X^TX)^{-1}X^T\mathbf{y}$$

under the assumption that the columns of the design matrix $X$ were linearly independent.

When the columns of $X$ are linearly dependent, we say that $X$ exhibits collinearity. In that case, $X^TX$ is singular ($\det(X^TX) = 0$), so $(X^TX)^{-1}$ does not exist and the formula above cannot be applied.

The normal equations

$$X^TX\boldsymbol{\beta} = X^T\mathbf{y}$$

are still valid—any least-squares solution must satisfy them—but the system is now consistent with infinitely many solutions. There is an entire family of vectors $\boldsymbol\beta$ that minimize the residual sum of squares $\lVert\mathbf{y}-X\boldsymbol\beta\rVert^2$.

To detect collinearity, we form $X^TX$ and check whether $\det(X^TX) = 0$. For instance, with

$$X = \begin{bmatrix} 1 & 3 \\ 0 & 0 \\ 1 & 3 \end{bmatrix},$$

column 2 is $3$ times column 1, so the columns are linearly dependent. Computing

$$X^TX = \begin{bmatrix} 2 & 6 \\ 6 & 18 \end{bmatrix}, \qquad \det(X^TX) = 36 - 36 = 0,$$

confirms that $X^TX$ is singular.