Linear Regression With Matrices

Linear regression seeks to model a linear relationship between an independent variable $x$ and a dependent variable $y$:

$$y = \beta_0 + \beta_1 x,$$

where $\beta_0$ is the intercept and $\beta_1$ is the slope.

Given $n$ data points $(x_1,y_1),(x_2,y_2),\ldots,(x_n,y_n),$ we want to find the coefficients $\beta_0$ and $\beta_1.$ Substituting each data point into the model gives $n$ equations:

$$\begin{align*} \beta_0 + \beta_1 x_1 &= y_1 \\ \beta_0 + \beta_1 x_2 &= y_2 \\ &\,\,\vdots \\ \beta_0 + \beta_1 x_n &= y_n. \end{align*}$$

We can write this system compactly as $A\boldsymbol\beta = \mathbf{y},$ where

$$A = \begin{bmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n \end{bmatrix},\qquad \boldsymbol\beta = \begin{bmatrix} \beta_0 \\ \beta_1 \end{bmatrix},\qquad \mathbf{y} = \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix}.$$

The matrix $A$ is called the design matrix. Its first column is all $1$s (these multiply the intercept $\beta_0$), and its second column holds the $x$-values of the data.

For example, the data points $(2,5),(4,6),(7,10)$ give the matrix equation

$$\begin{bmatrix} 1 & 2 \\ 1 & 4 \\ 1 & 7 \end{bmatrix}\begin{bmatrix} \beta_0 \\ \beta_1 \end{bmatrix} = \begin{bmatrix} 5 \\ 6 \\ 10 \end{bmatrix}.$$