Probability and statistics

Regression

Modeling the relationship between variables to predict one from the others.

IntuitionPredicting one variable from another

Suppose you plot the number of hours a group of students studied against the scores they got on an exam. The points do not fall on a perfect line, but they clearly drift upward: more hours tend to go with higher scores. Regression is the tool that turns this drifting cloud of points into a single best-fitting line (or curve), so you can summarize the trend and predict a score for a new number of hours.

Ordinary least-squares linear regression: the amber line minimizes the sum of squared vertical residuals (dashed segments) to the observed data points.

UndergraduateThe simple linear regression model

Definition: Simple linear regression

Given paired data $(x_1,y_1),\dots,(x_n,y_n)$ , the simple linear regression model assumes that $y$ depends on $x$ through a straight line plus random noise: $y=\beta_0+\beta_1x+\epsilon$ , where $\beta_0$ is the intercept, $\beta_1$ is the slope, and $\epsilon$ is a random error term with mean 0 that absorbs everything the line does not capture.

y=\beta_0+\beta_1x+\epsilon

The intercept $\beta_0$ and slope $\beta_1$ are unknown population parameters: fixed numbers we never observe directly. What we do have is a sample of $n$ points, from which we compute estimates $\hat\beta_0$ and $\hat\beta_1$ . The fitted line $\hat y=\hat\beta_0+\hat\beta_1x$ is our best guess at the true line, and for each data point the residual $e_i=y_i-\hat y_i$ measures how far the fitted line misses that particular observation.

\hat\beta_1=\dfrac{\sum_{i=1}^n (x_i-\bar x)(y_i-\bar y)}{\sum_{i=1}^n (x_i-\bar x)^2},\qquad \hat\beta_0=\bar y-\hat\beta_1\bar x

Theorem: Ordinary least squares estimators

Among all lines $y=b_0+b_1x$ , the choice that minimizes the sum of squared residuals $\sum_{i=1}^n (y_i-b_0-b_1x_i)^2$ is $b_1=\hat\beta_1=\dfrac{S_{xy}}{S_{xx}}$ and $b_0=\hat\beta_0=\bar y-\hat\beta_1\bar x$ , where $S_{xy}=\sum_{i=1}^n(x_i-\bar x)(y_i-\bar y)$ and $S_{xx}=\sum_{i=1}^n(x_i-\bar x)^2$ .

Why is it true?

The sum of squared residuals is a smooth (quadratic, convex) function of $b_0$ and $b_1$ , so its minimum is found exactly where both partial derivatives vanish — the same idea as finding the bottom of a bowl by setting its slope to zero in every direction.

Proof

Write $Q(b_0,b_1)=\sum_{i=1}^n (y_i-b_0-b_1x_i)^2$ . Taking the partial derivative with respect to $b_0$ and setting it to zero: $\dfrac{\partial Q}{\partial b_0}=-2\sum_{i=1}^n(y_i-b_0-b_1x_i)=0$ , which simplifies to the first normal equation $\sum_{i=1}^n y_i = nb_0+b_1\sum_{i=1}^n x_i$ , i.e. $b_0=\bar y-b_1\bar x$ .

Taking the partial derivative with respect to $b_1$ and setting it to zero: $\dfrac{\partial Q}{\partial b_1}=-2\sum_{i=1}^n x_i(y_i-b_0-b_1x_i)=0$ , the second normal equation $\sum_{i=1}^n x_iy_i = b_0\sum_{i=1}^n x_i+b_1\sum_{i=1}^n x_i^2$ .

Substitute $b_0=\bar y-b_1\bar x$ from the first equation into the second: $\sum_{i=1}^n x_iy_i = (\bar y-b_1\bar x)n\bar x+b_1\sum_{i=1}^n x_i^2 = n\bar x\bar y+b_1\left(\sum_{i=1}^n x_i^2-n\bar x^2\right)$ . Rearranging isolates $b_1$ : $b_1\left(\sum_{i=1}^n x_i^2-n\bar x^2\right)=\sum_{i=1}^n x_iy_i-n\bar x\bar y$ .

A direct algebraic expansion shows $\sum_{i=1}^n x_i^2-n\bar x^2=\sum_{i=1}^n(x_i-\bar x)^2=S_{xx}$ and $\sum_{i=1}^n x_iy_i-n\bar x\bar y=\sum_{i=1}^n(x_i-\bar x)(y_i-\bar y)=S_{xy}$ , so $b_1=S_{xy}/S_{xx}$ . Substituting back gives $b_0=\bar y-b_1\bar x$ . Because $Q$ is a sum of squares in a quadratic form that grows without bound as $|b_0|,|b_1|\to\infty$ , this unique stationary point must be the global minimum.

UndergraduateGoodness of fit: the coefficient of determination

R^2=1-\dfrac{SSE}{SST}

Definition: Sums of squares and $R^2$

Decompose the total variation in $y$ into three sums of squares: $SST=\sum_{i=1}^n(y_i-\bar y)^2$ (total), $SSR=\sum_{i=1}^n(\hat y_i-\bar y)^2$ (explained by the regression), and $SSE=\sum_{i=1}^n(y_i-\hat y_i)^2$ (left over as residuals). The coefficient of determination $R^2=SSR/SST=1-SSE/SST$ is the fraction of the variation in $y$ that the line explains.

Theorem: Sum-of-squares decomposition and

R^2

bounds

In simple linear regression, $SST=SSR+SSE$ , so $0\le R^2\le 1$ ; moreover $R^2$ equals the square of the sample correlation coefficient $r$ between $x$ and $y$ : $R^2=r^2$ .

Why is it true?

The residuals from a least-squares fit are always uncorrelated with the fitted values, because the normal equations that define $\hat\beta_0,\hat\beta_1$ are precisely the conditions that force this. That orthogonality is exactly what makes the total variation split cleanly into an "explained" piece and a "leftover" piece with no cross-term.

Proof

The two normal equations from the least-squares theorem above say $\sum_{i=1}^n e_i=0$ and $\sum_{i=1}^n x_ie_i=0$ , where $e_i=y_i-\hat y_i$ . Since $\hat y_i=\hat\beta_0+\hat\beta_1x_i$ is a linear combination of $1$ and $x_i$ , both normal equations combine to give $\sum_{i=1}^n e_i\hat y_i=\hat\beta_0\sum_{i=1}^n e_i+\hat\beta_1\sum_{i=1}^n x_ie_i=0$ . Combined with $\sum e_i=0$ , this gives $\sum_{i=1}^n e_i(\hat y_i-\bar y)=\sum e_i\hat y_i-\bar y\sum e_i=0$ .

Now expand $SST=\sum_{i=1}^n(y_i-\bar y)^2=\sum_{i=1}^n\big((y_i-\hat y_i)+(\hat y_i-\bar y)\big)^2=\sum e_i^2+2\sum e_i(\hat y_i-\bar y)+\sum(\hat y_i-\bar y)^2$ . The cross term vanishes by the orthogonality just shown, leaving $SST=SSE+SSR$ .

Since $SSE=\sum e_i^2\ge0$ and $SSR=\sum(\hat y_i-\bar y)^2\ge0$ , dividing $SST=SSR+SSE$ by $SST>0$ gives $R^2=SSR/SST\in[0,1]$ .

Finally, $\hat y_i-\bar y=\hat\beta_1(x_i-\bar x)$ (since $\hat y_i=\hat\beta_0+\hat\beta_1x_i$ and $\bar y=\hat\beta_0+\hat\beta_1\bar x$ ), so $SSR=\hat\beta_1^2S_{xx}$ . Substituting $\hat\beta_1=S_{xy}/S_{xx}$ gives $SSR=S_{xy}^2/S_{xx}$ , and since $SST=S_{yy}=\sum(y_i-\bar y)^2$ , $R^2=\dfrac{S_{xy}^2}{S_{xx}S_{yy}}=\left(\dfrac{S_{xy}}{\sqrt{S_{xx}S_{yy}}}\right)^2=r^2$ , the square of the sample correlation coefficient.

Example: Fitting a line to five data points

A small dataset has $x=(1,2,3,4,5)$ and $y=(2,4,5,4,5)$ . Find the least-squares line and compute $R^2$ .

Solution

The means are $\bar x=3$ and $\bar y=4$ . The deviations $(x_i-\bar x, y_i-\bar y)$ are $(-2,-2),(-1,0),(0,1),(1,0),(2,1)$ , so $S_{xy}=4+0+0+0+2=6$ and $S_{xx}=4+1+0+1+4=10$ .

This gives $\hat\beta_1=S_{xy}/S_{xx}=6/10=0.6$ and $\hat\beta_0=\bar y-\hat\beta_1\bar x=4-0.6\times3=2.2$ , so the fitted line is $\hat y=0.6x+2.2$ .

For $R^2$ : $SST=\sum(y_i-\bar y)^2=4+0+1+0+1=6$ , and $SSR=\hat\beta_1^2S_{xx}=0.36\times10=3.6$ , so $R^2=SSR/SST=3.6/6=0.6$ — the line explains 60% of the variation in $y$ .

UndergraduateChecking the model: residual analysis

Fitting a line is easy; trusting it requires checking whether the model's assumptions actually hold. Plotting the residuals $e_i=y_i-\hat y_i$ against $x_i$ (or against $\hat y_i$ ) is the standard diagnostic: if the linear model is appropriate, this residual plot should look like a random, structureless scatter around zero, with roughly constant spread.

Reading a residual plot
Pattern in the residual plot	What it suggests
Random scatter around zero, constant spread	The linear model and constant-variance assumption look reasonable
Funnel/fan shape (spread grows with $x$ )	Heteroscedasticity: the constant-variance assumption is violated
Curved (U-shaped or arc) pattern	The true relationship is nonlinear; a straight line is the wrong shape

UndergraduateReal-world applications and worked examples

Regression is one of the most widely used tools in applied science: economists use it to estimate how strongly a stock moves with the overall market, biologists use it to relate an animal's body size to its metabolic rate, engineers use it to calibrate instruments against a known reference, and epidemiologists use it to relate a drug's dose to a measured response. In every case, the same two questions arise: how good is the fit ( $R^2$ ), and does the residual plot reveal a hidden problem?

Example: Forecasting ice-cream sales from temperature

A shop records daily temperature $x$ (°C) and ice-cream sales $y$ (units) on five days: $x=(20,25,30,35,40)$ , $y=(30,45,55,65,80)$ . Fit a regression line and use it to forecast sales at $x=32$ °C.

Solution

The means are $\bar x=30$ , $\bar y=55$ . Deviations give $S_{xy}=(-10)(-25)+(-5)(-10)+0+5\times10+10\times25=250+50+0+50+250=600$ and $S_{xx}=100+25+0+25+100=250$ , so $\hat\beta_1=600/250=2.4$ and $\hat\beta_0=55-2.4\times30=-17$ : the fitted line is $\hat y=2.4x-17$ .

At $x=32$ : $\hat y=2.4\times32-17=76.8-17=59.8$ , so the shop should expect roughly 60 units sold.

Checking fit: $SST=625+100+0+100+625=1450$ and $SSR=\hat\beta_1^2S_{xx}=5.76\times250=1440$ , so $R^2=1440/1450\approx0.993$ — temperature explains about 99.3% of the variation in sales here, an unusually tight real-world fit.

Example: A high $R^2$ that still hides a problem

An engineer calibrates a new pressure sensor by fitting $\text{reading}=\beta_0+\beta_1\cdot\text{true pressure}$ against a reference instrument across many pressures, and gets $R^2=0.95$ — seemingly excellent. But plotting the residuals against true pressure reveals a clear U-shaped curve: residuals are positive at low and high pressures and negative in the middle. What does this mean, and what should the engineer do?

Solution

A high $R^2$ only says the line tracks the overall trend well; it does not certify that the relationship is linear. The systematic U-shaped residual pattern is exactly the "curved pattern" signature from the diagnostics table above: it means the true reading-versus-pressure relationship has curvature that the straight line is not capturing — the sensor likely has a genuine quadratic (or otherwise nonlinear) response.

Because the pattern is systematic rather than random, more data at the same pressures will not fix it — the model itself is misspecified, not just noisy. The residuals being simultaneously informative (clearly patterned) and numerically small (giving a high $R^2$ ) shows why relying on $R^2$ alone is a mistake for a calibration application where the whole point is precision.

The fix is to change the model, not just refit the line: add a quadratic term (fit $\text{reading}=\beta_0+\beta_1\cdot\text{pressure}+\beta_2\cdot\text{pressure}^2$ ) or apply a known physical transformation of the sensor's response, then re-check that the new residual plot looks like unstructured noise before trusting the calibration.

Given $S_{xy}=50$ and $S_{xx}=25$ for a dataset, what is $\hat\beta_1$ ?

If $R^2=0.81$ for a regression, what fraction of the variation in $y$ is left unexplained by the linear model?

A residual plot shows a clear funnel shape, widening as $x$ increases. What does this indicate?

A real-estate model fits $\text{price}=\hat\beta_0+\hat\beta_1\cdot\text{sqft}$ (price in thousands of dollars) with $\hat\beta_0=20$ and $\hat\beta_1=0.15$ . What price does it predict for a 1500-sqft house?