The Covariance of Two Random Variables

The covariance of two random variables $X$ and $Y$ measures how they vary together. It is defined as

$$\operatorname{Cov}(X,Y) = E\!\left[\,(X-\mu_X)(Y-\mu_Y)\,\right],$$

where $\mu_X = E[X]$ and $\mu_Y = E[Y]$ denote the means of $X$ and $Y.$

Interpretation: When $X$ tends to be above its mean exactly when $Y$ is above its mean, the product $(X-\mu_X)(Y-\mu_Y)$ is usually positive and the covariance is positive. When one variable tends to be above its mean while the other is below, the product is usually negative and the covariance is negative. A covariance of zero means there is no such linear tendency.

For discrete random variables with joint pmf $p(x,y),$ the covariance becomes a double sum:

$$\operatorname{Cov}(X,Y) = \sum_x \sum_y (x-\mu_X)(y-\mu_Y)\,p(x,y).$$

For example, suppose $X\in\{0,1\}$ and $Y\in\{1,2\}$ have joint pmf

$$p(0,1)=0.2,\quad p(0,2)=0.3,\quad p(1,1)=0.1,\quad p(1,2)=0.4.$$

The marginals give $\mu_X = 0(0.5)+1(0.5) = 0.5$ and $\mu_Y = 1(0.3)+2(0.7) = 1.7.$ Hence

$$\begin{align*} \operatorname{Cov}(X,Y) &= (-0.5)(-0.7)(0.2) + (-0.5)(0.3)(0.3) \\ &\quad + (0.5)(-0.7)(0.1) + (0.5)(0.3)(0.4) \\ &= 0.07 - 0.045 - 0.035 + 0.06 \\ &= 0.05. \end{align*}$$