The Sample Covariance Matrix

The sample variance measures how a single variable spreads around its own mean. When we record two variables on the same subjects, the analogous quantity measuring how they vary together is the sample covariance.

Given paired observations $(x_1,y_1),(x_2,y_2),\dots,(x_n,y_n)$ with sample means $\bar{x}$ and $\bar{y}$, the sample covariance between $x$ and $y$ is

$$s_{xy} = \dfrac{1}{n-1}\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y}).$$

This formula mirrors the sample variance — the sample variance is just the covariance of a variable with itself:

$$s_x^2 = \dfrac{1}{n-1}\sum_{i=1}^n (x_i-\bar{x})^2 = s_{xx}.$$

The sign of $s_{xy}$ indicates the direction of linear association:

• $s_{xy}>0$: when $x_i$ is above $\bar{x}$, $y_i$ tends to be above $\bar{y}$ as well.
• $s_{xy}<0$: when one variable is above its mean, the other tends to be below.
• $s_{xy}=0$: no linear association is detected in the sample.

To illustrate, consider the three paired observations $(1,2),(3,6),(5,4)$. We have $\bar{x}=3$ and $\bar{y}=4$, so

$$s_{xy}=\dfrac{1}{2}\Big[(-2)(-2)+(0)(2)+(2)(0)\Big]=\dfrac{4}{2}=2.$$