MathLabs
TheoremProved

Ordinary least squares estimators

Statement

Among all lines y=b0+b1xy=b_0+b_1x, the choice that minimizes the sum of squared residuals ∑i=1n(yi−b0−b1xi)2\sum_{i=1}^n (y_i-b_0-b_1x_i)^2 is b1=β^1=SxySxxb_1=\hat\beta_1=\dfrac{S_{xy}}{S_{xx}} and b0=β^0=yˉ−β^1xˉb_0=\hat\beta_0=\bar y-\hat\beta_1\bar x, where Sxy=∑i=1n(xi−xˉ)(yi−yˉ)S_{xy}=\sum_{i=1}^n(x_i-\bar x)(y_i-\bar y) and Sxx=∑i=1n(xi−xˉ)2S_{xx}=\sum_{i=1}^n(x_i-\bar x)^2.

Why is it true?

The sum of squared residuals is a smooth (quadratic, convex) function of b0b_0 and b1b_1, so its minimum is found exactly where both partial derivatives vanish — the same idea as finding the bottom of a bowl by setting its slope to zero in every direction.

Proof sketch

Write Q(b0,b1)=∑i=1n(yi−b0−b1xi)2Q(b_0,b_1)=\sum_{i=1}^n (y_i-b_0-b_1x_i)^2. Taking the partial derivative with respect to b0b_0 and setting it to zero: ∂Q∂b0=−2∑i=1n(yi−b0−b1xi)=0\dfrac{\partial Q}{\partial b_0}=-2\sum_{i=1}^n(y_i-b_0-b_1x_i)=0, which simplifies to the first normal equation ∑i=1nyi=nb0+b1∑i=1nxi\sum_{i=1}^n y_i = nb_0+b_1\sum_{i=1}^n x_i, i.e. b0=yˉ−b1xˉb_0=\bar y-b_1\bar x.

Taking the partial derivative with respect to b1b_1 and setting it to zero: ∂Q∂b1=−2∑i=1nxi(yi−b0−b1xi)=0\dfrac{\partial Q}{\partial b_1}=-2\sum_{i=1}^n x_i(y_i-b_0-b_1x_i)=0, the second normal equation ∑i=1nxiyi=b0∑i=1nxi+b1∑i=1nxi2\sum_{i=1}^n x_iy_i = b_0\sum_{i=1}^n x_i+b_1\sum_{i=1}^n x_i^2.

Substitute b0=yˉ−b1xˉb_0=\bar y-b_1\bar x from the first equation into the second: ∑i=1nxiyi=(yˉ−b1xˉ)nxˉ+b1∑i=1nxi2=nxˉyˉ+b1(∑i=1nxi2−nxˉ2)\sum_{i=1}^n x_iy_i = (\bar y-b_1\bar x)n\bar x+b_1\sum_{i=1}^n x_i^2 = n\bar x\bar y+b_1\left(\sum_{i=1}^n x_i^2-n\bar x^2\right). Rearranging isolates b1b_1: b1(∑i=1nxi2−nxˉ2)=∑i=1nxiyi−nxˉyˉb_1\left(\sum_{i=1}^n x_i^2-n\bar x^2\right)=\sum_{i=1}^n x_iy_i-n\bar x\bar y.

A direct algebraic expansion shows ∑i=1nxi2−nxˉ2=∑i=1n(xi−xˉ)2=Sxx\sum_{i=1}^n x_i^2-n\bar x^2=\sum_{i=1}^n(x_i-\bar x)^2=S_{xx} and ∑i=1nxiyi−nxˉyˉ=∑i=1n(xi−xˉ)(yi−yˉ)=Sxy\sum_{i=1}^n x_iy_i-n\bar x\bar y=\sum_{i=1}^n(x_i-\bar x)(y_i-\bar y)=S_{xy}, so b1=Sxy/Sxxb_1=S_{xy}/S_{xx}. Substituting back gives b0=yˉ−b1xˉb_0=\bar y-b_1\bar x. Because QQ is a sum of squares in a quadratic form that grows without bound as ∣b0∣,∣b1∣→∞|b_0|,|b_1|\to\infty, this unique stationary point must be the global minimum.

Topics that use this theorem

Step-by-step proofs

No step-by-step proof yet for this theorem.