The law of large numbers and the central limit theorem

Markov's inequality applied to the non-negative random variable $(X-\mu)^2$ gives $P((X-\mu)^2 \ge k^2) \le E[(X-\mu)^2]/k^2 = \sigma^2/k^2$; the event $(X-\mu)^2 \ge k^2$ is exactly $|X-\mu| \ge k$.

We first prove Markov's inequality: for a non-negative random variable $Y\ge0$ and any $a>0$, the indicator bound $Y\ge a\cdot\mathbb{1}_{\{Y\ge a\}}$ holds pointwise, because on the event $\{Y\ge a\}$ the right side equals $a\le Y$, and elsewhere the right side is $0\le Y$. Taking expectations of both sides preserves the inequality: $E[Y]\ge aP(Y\ge a)$, hence $P(Y\ge a)\le E[Y]/a$.

Now apply Markov's inequality to the non-negative random variable $Y=(X-\mu)^2$ with threshold $a=k^2$: $P((X-\mu)^2\ge k^2)\le E[(X-\mu)^2]/k^2$. By definition of variance, $E[(X-\mu)^2]=\sigma^2$, so the right side is exactly $\sigma^2/k^2$.

Finally, the events on either side of Markov's inequality coincide: $(X-\mu)^2\ge k^2 \iff |X-\mu|\ge k$ (both sides are non-negative, so squaring preserves the order). Substituting gives $P(|X-\mu|\ge k)\le\sigma^2/k^2$, which is Chebyshev's inequality.

$\mathrm{Var}(\bar X_n) = \sigma^2/n \to 0$ (when $\sigma^2$ is finite), so Chebyshev's inequality gives $P(|\bar X_n - \mu| \ge \varepsilon) \le \sigma^2/(n\varepsilon^2) \to 0$. (The full theorem needs only a finite mean, via a more delicate truncation argument, but the Chebyshev proof is the illuminating special case.)

Let $\bar X_n=\frac1n\sum_{i=1}^nX_i$ be the sample mean of $n$ i.i.d. copies of $X$ with mean $\mu$ and, for this Chebyshev-based proof, finite variance $\sigma^2$. Linearity of expectation gives $E[\bar X_n]=\mu$.

Independence makes variances add: $\mathrm{Var}(\bar X_n)=\frac1{n^2}\sum_{i=1}^n\mathrm{Var}(X_i)=\frac{\sigma^2}{n}$, since each $X_i$ contributes $\mathrm{Var}(X_i)=\sigma^2$ and the cross terms $\mathrm{Cov}(X_i,X_j)$ for $i\ne j$ vanish by independence.

Fix any $\varepsilon>0$ and apply Chebyshev's inequality, proved above, to $\bar X_n$: $P(|\bar X_n-\mu|\ge\varepsilon)\le\frac{\mathrm{Var}(\bar X_n)}{\varepsilon^2}=\frac{\sigma^2}{n\varepsilon^2}$.

As $n\to\infty$, the right side $\sigma^2/(n\varepsilon^2)\to0$ for every fixed $\varepsilon>0$, which is exactly the definition of convergence in probability. Hence $\bar X_n\to\mu$ in probability.

The classical proof (Kolmogorov, 1933) uses Kolmogorov's inequality — a strengthening of Chebyshev's that controls the whole trajectory of partial sums at once — together with a subsequence argument; it is considerably more delicate than the weak law's one-line Chebyshev proof.

We sketch the proof under the extra (common textbook) assumption that $X$ has a finite fourth moment $E[X^4]<\infty$; the general statement needs only a finite mean but requires a more delicate truncation argument (Kolmogorov, 1933). After centering, assume $E[X]=0$ (replace $X_i$ by $X_i-\mu$ throughout), and let $S_n=\sum_{i=1}^nX_i$.

Expand $E[S_n^4]=\sum_{i,j,k,l}E[X_iX_jX_kX_l]$ over all $n^4$ index quadruples. Independence and $E[X]=0$ kill every term that contains an index appearing exactly once (its factor averages to $0$), leaving only the $n$ terms with all four indices equal and the $3n(n-1)$ terms that pair up into two distinct equal indices. This gives $E[S_n^4]=nE[X^4]+3n(n-1)\sigma^4\le Cn^2$ for a constant $C$ depending only on $E[X^4]$ and $\sigma^2$.

Dividing by $n^4$: $E\left[\left(\frac{S_n}{n}\right)^4\right]\le\frac{C}{n^2}$. Summing over $n$, $\sum_{n=1}^\infty E\left[\left(\frac{S_n}{n}\right)^4\right]\le\sum_{n=1}^\infty\frac{C}{n^2}<\infty$ since $\sum 1/n^2$ converges.

A sum of non-negative random variables with finite expected total sum must itself be finite almost surely (monotone convergence), so $\sum_{n=1}^\infty\left(\frac{S_n}{n}\right)^4<\infty$ almost surely, which forces its individual terms to vanish: $\left(\frac{S_n}{n}\right)^4\to0$, i.e. $\frac{S_n}{n}\to0$ almost surely. Undoing the centering gives $\bar X_n\to\mu$ almost surely, the strong law.

Intuitively, the standardized sum's moment generating (or characteristic) function converges to that of $N(0,1)$ term by term via a Taylor expansion, because only the mean and variance survive the standardization — all higher moments get washed out as $n \to \infty$. This is why the same bell shape appears no matter what $X_i$'s original distribution looked like.

For a random variable $Z$, its characteristic function is $\varphi_Z(t)=E[e^{itZ}]$. Standardize each $X_i$ by setting $Z_i=\frac{X_i-\mu}{\sigma}$, so that $E[Z_i]=0,\ E[Z_i^2]=1$, and the standardized sample mean becomes a sum of these: $T_n=\frac{1}{\sqrt n}\sum_{i=1}^nZ_i=\frac{\bar X_n-\mu}{\sigma/\sqrt n}$.

Because the $Z_i$ are i.i.d., the characteristic function of a sum multiplies: $\varphi_{T_n}(t)=\left[\varphi_Z\!\left(\frac{t}{\sqrt n}\right)\right]^n$.

Since $E[Z]=0$ and $E[Z^2]=1$, a Taylor expansion of $\varphi_Z$ near $0$ gives $\varphi_Z(s)=1-\frac{s^2}{2}+o(s^2)$. Substituting $s=t/\sqrt n$: $\varphi_Z\!\left(\frac{t}{\sqrt n}\right)=1-\frac{t^2}{2n}+o\!\left(\frac1n\right)$.

Raising to the $n$-th power and using the standard limit $\left(1+\frac{c}{n}+o(1/n)\right)^n\to e^{c}$: $\varphi_{T_n}(t)=\left[1-\frac{t^2}{2n}+o\!\left(\frac1n\right)\right]^n\to e^{-t^2/2}$. Since $e^{-t^2/2}$ is precisely the characteristic function of the standard normal $N(0,1)$, Lévy's continuity theorem converts this pointwise convergence of characteristic functions into convergence in distribution: $T_n\xrightarrow{d}N(0,1)$.