MathLabs
TheoremProved

Sum-of-squares decomposition and $R^2$ bounds

Statement

In simple linear regression, SST=SSR+SSESST=SSR+SSE, so 0≤R2≤10\le R^2\le 1; moreover R2R^2 equals the square of the sample correlation coefficient rr between xx and yy: R2=r2R^2=r^2.

Why is it true?

The residuals from a least-squares fit are always uncorrelated with the fitted values, because the normal equations that define β^0,β^1\hat\beta_0,\hat\beta_1 are precisely the conditions that force this. That orthogonality is exactly what makes the total variation split cleanly into an "explained" piece and a "leftover" piece with no cross-term.

Proof sketch

The two normal equations from the least-squares theorem above say ∑i=1nei=0\sum_{i=1}^n e_i=0 and ∑i=1nxiei=0\sum_{i=1}^n x_ie_i=0, where ei=yi−y^ie_i=y_i-\hat y_i. Since y^i=β^0+β^1xi\hat y_i=\hat\beta_0+\hat\beta_1x_i is a linear combination of 11 and xix_i, both normal equations combine to give ∑i=1neiy^i=β^0∑i=1nei+β^1∑i=1nxiei=0\sum_{i=1}^n e_i\hat y_i=\hat\beta_0\sum_{i=1}^n e_i+\hat\beta_1\sum_{i=1}^n x_ie_i=0. Combined with ∑ei=0\sum e_i=0, this gives ∑i=1nei(y^i−yˉ)=∑eiy^i−yˉ∑ei=0\sum_{i=1}^n e_i(\hat y_i-\bar y)=\sum e_i\hat y_i-\bar y\sum e_i=0.

Now expand SST=∑i=1n(yi−yˉ)2=∑i=1n((yi−y^i)+(y^i−yˉ))2=∑ei2+2∑ei(y^i−yˉ)+∑(y^i−yˉ)2SST=\sum_{i=1}^n(y_i-\bar y)^2=\sum_{i=1}^n\big((y_i-\hat y_i)+(\hat y_i-\bar y)\big)^2=\sum e_i^2+2\sum e_i(\hat y_i-\bar y)+\sum(\hat y_i-\bar y)^2. The cross term vanishes by the orthogonality just shown, leaving SST=SSE+SSRSST=SSE+SSR.

Since SSE=∑ei2≥0SSE=\sum e_i^2\ge0 and SSR=∑(y^i−yˉ)2≥0SSR=\sum(\hat y_i-\bar y)^2\ge0, dividing SST=SSR+SSESST=SSR+SSE by SST>0SST>0 gives R2=SSR/SST∈[0,1]R^2=SSR/SST\in[0,1].

Finally, y^i−yˉ=β^1(xi−xˉ)\hat y_i-\bar y=\hat\beta_1(x_i-\bar x) (since y^i=β^0+β^1xi\hat y_i=\hat\beta_0+\hat\beta_1x_i and yˉ=β^0+β^1xˉ\bar y=\hat\beta_0+\hat\beta_1\bar x), so SSR=β^12SxxSSR=\hat\beta_1^2S_{xx}. Substituting β^1=Sxy/Sxx\hat\beta_1=S_{xy}/S_{xx} gives SSR=Sxy2/SxxSSR=S_{xy}^2/S_{xx}, and since SST=Syy=∑(yi−yˉ)2SST=S_{yy}=\sum(y_i-\bar y)^2, R2=Sxy2SxxSyy=(SxySxxSyy)2=r2R^2=\dfrac{S_{xy}^2}{S_{xx}S_{yy}}=\left(\dfrac{S_{xy}}{\sqrt{S_{xx}S_{yy}}}\right)^2=r^2, the square of the sample correlation coefficient.

Topics that use this theorem

Step-by-step proofs

No step-by-step proof yet for this theorem.