Covariance

The covariance between two random variables $X$ and $Y$ is defined as $\operatorname {cov} (X,Y)=\mathrm {E} ((X-\mathrm {E} (X))(Y-\mathrm {E} (Y)))$.

Questions/things to explain

• A lot of explanations of covariance say things like "if the covariance is high, then the two variables vary together, so that when one is higher than average the other is as well". That sounds more like conditional expectation though, specifically $\mathrm {E} (Y-\mathrm {E} (Y)\mid X-\mathrm {E} (X))$. Can we express covariance in terms of this conditional language?

Here is one attempt, though I don't think it helps much:

${\begin{aligned}E[(X-EX)(Y-EY)]&=\sum _{x}\sum _{y}P(X=x,Y=x)(x-EX)(y-EY)\\&=\sum _{x}(x-EX)\sum _{y}P(x,y)(y-EY)\\&=\sum _{x}P(X=x)(x-EX)\sum _{y}{\frac {P(x,y)}{P(X=x)}}(y-EY)\\&=\sum _{x}P(X=x)(x-EX)\sum _{y}P(Y=y|X=x)(y-EY)\\&=E_{X}[(X-EX)E[Y-EY|X]]\end{aligned}}$

• Explain difference (in units, range of values) with correlation. Can we get positive/negative/large/small covariance and negative/positive/small/large correlation?
  • This seems like a good example of where correlation vs covariance matters: "Correlation is covariance divided by variance, so if A is highly predictive of B, there can be a strong “correlation” between them even if A is ranging from 0 to 9 and B is only ranging from 50.0001 and 50.0009. Price’s Equation runs on covariance of characteristics with reproduction—not correlation! If you can compress variance in characteristics into a tiny band, the covariance goes way down, and so does the cumulative change in the characteristic." [1]
• Visualization as signed area of rectangles: between all points [2] [3] and between the axes drawn by the means [4] [5]
• comparison with independence of random variables
• clarify how the expectation is taken in each of the three appearances