Probability and statistics

Estimation

Using sample data to infer unknown values of a population, with point and interval estimates.

IntuitionFrom a Handful to the Whole

Imagine tasting one spoonful from a large pot of soup to judge how salty the whole pot is. You don't drink the entire pot -- a single representative spoonful (a sample) is enough to guess the salt level of the whole thing (the population). Estimation is the mathematics of turning that spoonful into a trustworthy guess, together with a sense of how far off the guess might be.

Standard normal sampling distribution

\mathcal{N}(0,1)

with the central confidence interval

[-z, +z]

shaded in green; slide

p

to widen or narrow the confidence level.

SchoolPoint Estimates: A Single Best Guess

Definition: Point estimator

A point estimator is a single number, computed from sample data, used as a best guess for an unknown population parameter. For estimating the population mean $\mu$ , the natural point estimator is the sample mean $\bar X$ : the average of the observed values $X_1,\dots,X_n$ .

\bar X = \frac{1}{n}\sum_{i=1}^{n} X_i

Here $n$ is the sample size, $X_1,\dots,X_n$ are the individual sample observations, and $\bar X$ is their average. Because different samples give different values of $\bar X$ , it is itself a random variable with its own distribution -- called the sampling distribution of the mean -- whose spread shrinks as $n$ grows.

\bar X - z_{\alpha/2}\frac{\sigma}{\sqrt{n}} \le \mu \le \bar X + z_{\alpha/2}\frac{\sigma}{\sqrt{n}}

This is the $100(1-\alpha)\%$ confidence interval for $\mu$ when the population standard deviation $\sigma$ is known: $z_{\alpha/2}$ is the standard-normal critical value that leaves probability $\alpha/2$ in each tail, and $\frac{\sigma}{\sqrt n}$ is the standard error of $\bar X$ , the typical distance between $\bar X$ and $\mu$ .

Common confidence levels and their $z$ -critical values
Confidence level	$z_{\alpha/2}$
90%	$z_{\alpha/2}=1.645$
95%	$z_{\alpha/2}=1.960$
99%	$z_{\alpha/2}=2.576$

UndergraduateWhy the Confidence Interval Works

Theorem: The sample mean is unbiased

If $X_1,\dots,X_n$ are independent, identically distributed with $E[X_i] = \mu$ for every $i$ , then the sample mean $\bar X = \frac{1}{n}\sum_{i=1}^{n} X_i$ satisfies $E[\bar X] = \mu$ .

Why is it true?

Averaging removes systematic bias: on average, across many hypothetical samples, $\bar X$ lands exactly on the true population mean $\mu$ , neither systematically too high nor too low.

Proof

By definition, $\bar X = \frac{1}{n}\sum_{i=1}^{n} X_i$ . Expectation is a linear operator, so it distributes over the sum and the constant factor $\frac{1}{n}$ : $E[\bar X] = \frac{1}{n}\sum_{i=1}^{n} E[X_i]$ .

Since every $X_i$ comes from the same population, $E[X_i] = \mu$ for each of the $n$ terms, so the sum collapses to $\frac{1}{n}\sum_{i=1}^{n} \mu = \frac{1}{n}(n\mu) = \mu$ .

Therefore $E[\bar X] = \mu$ for every sample size $n$ : the sample mean is unbiased no matter how small or large the sample is. This is a statement about the center of its sampling distribution only -- the spread of that distribution, $\operatorname{Var}(\bar X) = \sigma^2/n$ , still shrinks as $n$ grows, which is what the confidence interval below uses.

Theorem: A confidence interval for the mean, via the Central Limit Theorem

If $X_1,\dots,X_n$ are independent, identically distributed with mean $\mu$ and finite variance $\sigma^2$ , then for large $n$ the confidence interval $\bar X - z_{\alpha/2}\frac{\sigma}{\sqrt n} \le \mu \le \bar X + z_{\alpha/2}\frac{\sigma}{\sqrt n}$ contains $\mu$ with probability approximately $1-\alpha$ .

Why is it true?

The Central Limit Theorem says the standardized sample mean behaves like a standard normal variable once $n$ is large, so we can use normal quantiles $z_{\alpha/2}$ to bracket $\mu$ with a known long-run success rate, regardless of the shape of the original population.

Proof

By the Central Limit Theorem, $\frac{\bar X - \mu}{\sigma/\sqrt n}$ converges in distribution to $N(0,1)$ as $n$ $\to\infty$ . So for large $n$ , $-z_{\alpha/2} \le \frac{\bar X - \mu}{\sigma/\sqrt n} \le z_{\alpha/2}$ holds with probability approximately $1-\alpha$ , where $z_{\alpha/2}$ is the value cutting probability $\alpha/2$ from each tail of $N(0,1)$ .

Multiply all three sides of the inequality by $\frac{\sigma}{\sqrt n}$ $>0$ (this preserves the inequality direction): $-z_{\alpha/2}\frac{\sigma}{\sqrt n} \le \bar X - \mu \le z_{\alpha/2}\frac{\sigma}{\sqrt n}$ .

$\bar X$ is subtracted from all sides, then all sides are multiplied by $-1$ (flipping the inequalities): $\bar X - z_{\alpha/2}\frac{\sigma}{\sqrt n} \le \mu \le \bar X + z_{\alpha/2}\frac{\sigma}{\sqrt n}$ . This is exactly the confidence-interval formula: since the underlying probability statement held with approximate probability $1-\alpha$ , so does this rearranged interval containing $\mu$ .

The margin of error $E = z_{\alpha/2}\frac{\sigma}{\sqrt n}$ shrinks only like $1/\sqrt n$ , not like $1/n$ : to cut the margin of error in half you must quadruple the sample size $n$ , and to cut it to a third you need nine times as many observations. Precision is expensive: each extra digit of accuracy costs far more data than the one before it.

UndergraduateReal-World Applications and Worked Examples

Confidence intervals appear wherever a decision must be made from an incomplete sample: quality control on a factory line, opinion polling before an election, dosage trials in medicine, or calibrating a telescope from repeated measurements. In every case, the same formula turns a sample average into a range that is honest about its own uncertainty.

Example: Confidence interval for a bolt's mean diameter

A quality-control engineer samples $n=64$ bolts and finds a sample mean diameter $\bar x = 12.02$ mm, with known population standard deviation $\sigma = 0.16$ mm. Construct a 95% confidence interval for the true mean diameter $\mu$ , using $z_{0.025}=1.96$ .

Solution

The standard error is $\frac{0.16}{\sqrt{64}} = 0.02$ mm.

The margin of error is $1.96 \times 0.02 = 0.0392$ mm.

The 95% confidence interval is $(12.02 - 0.0392,\ 12.02 + 0.0392) = (11.981,\ 12.059)$ mm: we are 95% confident that the true mean bolt diameter lies in this range.

Example: Estimating the share of voters supporting a policy

A pollster surveys $n=400$ randomly chosen voters and finds a sample proportion $\hat p = 0.53$ in favor of a policy. Treating the sample proportion as a sample mean of 0/1 responses, construct a 95% confidence interval for the true population proportion $\mu$ in favor, using $z_{0.025}=1.96$ .

Solution

The estimated standard deviation of a single 0/1 response is $\sqrt{\hat p(1-\hat p)} = \sqrt{0.53\times 0.47} \approx 0.499$ , so the standard error of the sample proportion is $\frac{0.499}{\sqrt{400}} \approx 0.025$ .

The margin of error is $1.96 \times 0.025 \approx 0.049$ .

The 95% confidence interval is $(0.53 - 0.049,\ 0.53 + 0.049) = (0.481,\ 0.579)$ : the poll is 95% confident the true support lies between about 48.1% and 57.9%, which is too wide to call the policy majority-supported with certainty -- a larger sample would be needed for a tighter call.

Using the margin-of-error formula $E = z_{\alpha/2}\frac{\sigma}{\sqrt n}$ , what is the margin of error when $z_{\alpha/2}=1.96$ , $\sigma=8$ , and $n=64$ ?

A 95% confidence interval for the mean daily temperature is computed from one sample. What is the correct interpretation of "95% confidence"?

The property $E[\bar X] = \mu$ for every sample size $n$ describes which characteristic of the estimator $\bar X$ ?

If the sample size $n$ increases from 100 to 400, with $\sigma$ and $z_{\alpha/2}$ unchanged, the margin of error is multiplied by ...

References

NIST/SEMATECH (2013). Confidence Limits for the Mean
Diez, D.; Cetinkaya-Rundel, M.; Barr, C. (2019). OpenIntro Statistics