Probability and statistics

Bayesian statistics

An approach to statistics that updates a prior belief into a posterior belief as new data arrives.

IntuitionUpdating beliefs as evidence arrives

Suppose you are handed a coin and asked: is it fair? Before flipping it even once, you might believe it is probably close to fair, but you are not sure — call this your prior belief. Now you flip it 10 times and see 8 heads. That data should shift your belief toward "probably biased toward heads," without throwing away what you believed before. Bayesian statistics is the precise machinery for combining a prior belief with new data to get an updated, posterior belief.

Bayesian Beta–Binomial updating: after observing

k \approx np

successes in

n

trials, the diffuse grey prior

\mathrm{Beta}(2,2)

sharpens into the violet posterior density around

p

UndergraduateBayesian updating: from prior to posterior

Definition: Prior, likelihood, posterior

Given an unknown parameter $\theta$ , a prior distribution $\pi(\theta)$ encodes belief about $\theta$ before seeing data $x$ . A likelihood $L(x\mid\theta)$ says how probable the observed data is for each value of $\theta$ . Bayes' theorem combines them into the posterior distribution: $\pi(\theta\mid x)\propto L(x\mid\theta)\,\pi(\theta)$ — posterior is proportional to likelihood times prior.

\pi(\theta\mid x)\propto L(x\mid\theta)\,\pi(\theta)

The proportionality symbol hides a normalizing constant $m(x)=\int L(x\mid\theta')\pi(\theta')\,d\theta'$ (called the marginal likelihood or evidence), which does not depend on $\theta$ and simply rescales the product $L(x\mid\theta)\pi(\theta)$ so that $\pi(\theta\mid x)$ integrates to $1$ over all $\theta$ , exactly like any other probability density.

\pi(\theta\mid x)=\dfrac{L(x\mid\theta)\pi(\theta)}{\int L(x\mid\theta')\pi(\theta')\,d\theta'}

Theorem: Bayes' theorem for a posterior distribution

If $\theta$ has prior density $\pi(\theta)$ and data $x$ has likelihood $L(x\mid\theta)$ , then the posterior density of $\theta$ given $x$ is $\pi(\theta\mid x)=\dfrac{L(x\mid\theta)\pi(\theta)}{m(x)}$ , where $m(x)=\int L(x\mid\theta')\pi(\theta')\,d\theta'$ ; equivalently $\pi(\theta\mid x)\propto L(x\mid\theta)\pi(\theta)$ .

Why is it true?

This is nothing more than the definition of a conditional density applied to $\theta$ and $x$ jointly: the posterior is the joint density of $(\theta,x)$ divided by the marginal density of $x$ alone, and the joint density factors as prior times likelihood.

Proof

The joint density of $(\theta,x)$ factors two ways: as $\pi(\theta)L(x\mid\theta)$ (prior times likelihood, by the definition of the likelihood as the conditional density of $x$ given $\theta$ ) and as $\pi(\theta\mid x)m(x)$ (posterior times the marginal density of $x$ , by the definition of the posterior as the conditional density of $\theta$ given $x$ ). Since both expressions equal the same joint density, $\pi(\theta)L(x\mid\theta)=\pi(\theta\mid x)m(x)$ .

Solving for $\pi(\theta\mid x)$ gives $\pi(\theta\mid x)=\dfrac{L(x\mid\theta)\pi(\theta)}{m(x)}$ , provided $m(x)>0$ .

It remains to check $m(x)=\int L(x\mid\theta')\pi(\theta')\,d\theta'$ is exactly the right normalizing constant: integrating both sides of the factorization $\pi(\theta)L(x\mid\theta)=\pi(\theta\mid x)m(x)$ over $\theta$ , the left side becomes $\int \pi(\theta)L(x\mid\theta)\,d\theta=m(x)$ by definition, and the right side becomes $m(x)\int\pi(\theta\mid x)\,d\theta=m(x)\times1$ since $\pi(\cdot\mid x)$ is a probability density. Both sides agree, confirming $m(x)$ is consistent and $\pi(\theta\mid x)$ integrates to $1$ as required of a density.

UndergraduateConjugate priors: the Beta-Binomial model

Definition: Conjugate prior

A prior family is conjugate to a likelihood if the posterior stays in the same family as the prior, just with updated parameters. This turns Bayesian updating from an integral (computing $m(x)$ ) into simple arithmetic on the family's parameters — the reason conjugate priors were the workhorse of Bayesian statistics before modern computational methods existed.

\pi(\theta)=\dfrac{\theta^{\alpha-1}(1-\theta)^{\beta-1}}{B(\alpha,\beta)},\qquad 0<\theta<1

Theorem: Beta-Binomial conjugacy

If $\theta\sim\mathrm{Beta}(\alpha,\beta)$ is the prior and, given $\theta$ , $k$ successes are observed in $n$ independent trials (so $k\mid\theta\sim\mathrm{Binomial}(n,\theta)$ ), then the posterior is $\theta\mid k\sim\mathrm{Beta}(\alpha+k,\ \beta+n-k)$ .

Why is it true?

The Binomial likelihood contributes a factor $\theta^k(1-\theta)^{n-k}$ , and the Beta prior contributes $\theta^{\alpha-1}(1-\theta)^{\beta-1}$ ; multiplying these just adds the exponents, landing exactly on the shape of another Beta density.

Proof

The likelihood of observing $k$ successes in $n$ trials given $\theta$ is $L(k\mid\theta)=\binom{n}{k}\theta^k(1-\theta)^{n-k}$ . By the posterior-proportionality theorem above, $\pi(\theta\mid k)\propto L(k\mid\theta)\pi(\theta)=\binom{n}{k}\theta^k(1-\theta)^{n-k}\cdot\dfrac{\theta^{\alpha-1}(1-\theta)^{\beta-1}}{B(\alpha,\beta)}$ .

The factors $\binom{n}{k}$ and $B(\alpha,\beta)$ do not depend on $\theta$ , so they can be absorbed into the proportionality: $\pi(\theta\mid k)\propto\theta^{k}(1-\theta)^{n-k}\cdot\theta^{\alpha-1}(1-\theta)^{\beta-1}=\theta^{(\alpha+k)-1}(1-\theta)^{(\beta+n-k)-1}$ .

This last expression is exactly the kernel (the $\theta$ -dependent part) of a $\mathrm{Beta}(\alpha+k,\beta+n-k)$ density. Since a probability density on $(0,1)$ with this kernel has a unique normalizing constant (namely $1/B(\alpha+k,\beta+n-k)$ , by definition of the Beta function), the posterior must be exactly $\pi(\theta\mid k)=\dfrac{\theta^{(\alpha+k)-1}(1-\theta)^{(\beta+n-k)-1}}{B(\alpha+k,\beta+n-k)}$ , i.e. $\theta\mid k\sim\mathrm{Beta}(\alpha+k,\beta+n-k)$ .

Example: A/B testing a website's click-through rate

Before running an experiment, an analyst places a weakly informative $\mathrm{Beta}(2,2)$ prior on a button's true click-through rate $\theta$ (centered at 0.5, but not very confident). After showing the button to $n=20$ visitors, $k=7$ click it. Find the posterior distribution and its mean.

Solution

By Beta-Binomial conjugacy, the posterior is $\mathrm{Beta}(\alpha+k,\ \beta+n-k)=\mathrm{Beta}(2+7,\ 2+13)=\mathrm{Beta}(9,15)$ .

The mean of a $\mathrm{Beta}(a,b)$ distribution is $a/(a+b)$ , so the posterior mean is $9/(9+15)=9/24=0.375$ : after seeing the data, the analyst's best point estimate for the click-through rate moved from the prior mean of $0.5$ down to $0.375$ , while still being pulled toward — not equal to — the raw sample proportion $7/20=0.35$ , because the prior still contributes some weight.

UndergraduateCredible intervals vs confidence intervals

Definition: Credible interval

A $(1-\alpha)$ credible interval for $\theta$ is any interval $[L,U]$ with $\int_L^U \pi(\theta\mid x)\,d\theta=1-\alpha$ : it is a direct probability statement about $\theta$ , computed from the posterior after the data $x$ has already been observed.

This is a genuinely different object from a frequentist confidence interval. A confidence interval is built from a procedure that, applied to many hypothetical samples, would capture the true (fixed) $\theta$ a known fraction of the time; once a specific interval is computed from the specific data at hand, $\theta$ is either in it or not — there is no probability left to talk about. A credible interval instead treats $\theta$ itself as having a distribution, so "the probability $\theta$ is in $[L,U]$ " is always a meaningful statement.

Credible interval vs confidence interval
Aspect	Bayesian credible interval	Frequentist confidence interval
What is random	$\theta$ is treated as random, with a posterior distribution; the interval is fixed once the data is observed	$\theta$ is a fixed unknown constant; the interval itself is the random object, varying from sample to sample
Interpretation	Given the observed data, the probability that $\theta$ lies in the interval is $1-\alpha$	Over repeated sampling, $1-\alpha$ of the intervals constructed this way would contain the true $\theta$
Depends on a prior	Yes — the prior $\pi(\theta)$ enters directly into the posterior	No — computed only from the likelihood and the sampling distribution

AdvancedBeyond conjugacy: sampling the posterior

Conjugate priors are elegant, but most realistic models — many parameters, hierarchical structure, non-standard likelihoods — have no conjugate form, so the normalizing constant $m(x)$ has no closed-form integral. Markov chain Monte Carlo (MCMC) methods sidestep the integral entirely: they construct a Markov chain whose stationary distribution is exactly the posterior $\pi(\theta\mid x)$ , then simulate that chain and use the resulting samples to approximate any posterior quantity (mean, credible interval, or anything else) by simple averages over the samples, without ever computing $m(x)$ .

UndergraduateReal-world applications and worked examples

Bayesian updating shows up wherever a belief needs to be revised in light of noisy evidence: doctors interpreting a diagnostic test, spam filters classifying an email, search-and-rescue teams updating where to look, and spacecraft navigation systems fusing sensor readings all run some version of "posterior $\propto$ likelihood $\times$ prior."

Example: Why a positive test is not proof of disease

A rare disease affects $1\%$ of a population ( $P(D)=0.01$ ). A test has $99\%$ sensitivity ( $P(+\mid D)=0.99$ ) and a $5\%$ false-positive rate ( $P(+\mid \lnot D)=0.05$ ). A random person tests positive. Find $P(D\mid +)$ .

Solution

By the law of total probability, $P(+)=P(+\mid D)P(D)+P(+\mid\lnot D)P(\lnot D)=0.99\times0.01+0.05\times0.99=0.0099+0.0495=0.0594$ .

Bayes' theorem then gives $P(D\mid +)=\dfrac{P(+\mid D)P(D)}{P(+)}=\dfrac{0.0099}{0.0594}\approx0.167$ .

Despite a test that looks very accurate (99% sensitivity, only 5% false positives), a positive result still means only about a $16.7\%$ chance of actually having the disease — because the disease is rare, false positives from the large healthy population outnumber true positives from the tiny diseased population. This is exactly Bayesian updating in action: the low prior $P(D)=0.01$ pulls the posterior far below the test's own accuracy figures.

Example: From a flat prior to a skewed posterior

To estimate a coin's bias $\theta$ , a skeptic starts with a completely uninformative $\mathrm{Beta}(1,1)$ prior (uniform on $(0,1)$ ). After flipping the coin 10 times and seeing 8 heads, find the posterior distribution, its mean, and its mode.

Solution

By Beta-Binomial conjugacy with $\alpha=\beta=1$ , $n=10$ , $k=8$ : the posterior is $\mathrm{Beta}(1+8,\ 1+2)=\mathrm{Beta}(9,3)$ .

The posterior mean is $9/(9+3)=9/12=0.75$ . The mode of a $\mathrm{Beta}(a,b)$ distribution with $a,b>1$ is $(a-1)/(a+b-2)$ , giving $8/10=0.8$ here.

The prior was a flat, symmetric bump with no preferred value; the posterior is a sharply asymmetric bump peaking near $0.8$ , exactly the "peak shift away from a symmetric prior" that the widget at the top of this page illustrates schematically — 8 heads out of 10 flips has genuinely pulled belief toward a biased coin, even though the sample size is small.

Research frontier as of 2026

Conjugate-prior arithmetic solves only a narrow slice of real Bayesian models. Modern practice for complex models (many parameters, hierarchical structure, non-conjugate likelihoods) relies on Markov chain Monte Carlo algorithms such as Hamiltonian Monte Carlo and its self-tuning variant, the No-U-Turn Sampler, implemented in probabilistic programming languages like Stan, PyMC and NumPyro, which let a practitioner write down a model and let the software handle posterior sampling. Active research directions include: scaling MCMC to the huge datasets common in machine learning (via stochastic-gradient and subsampling variants, which trade some exactness for speed); diagnosing when a sampler has not converged in high-dimensional or multimodal posteriors, which remains only partially solved; and faster approximate alternatives such as variational inference, which turns posterior computation into an optimization problem at the cost of an approximation whose accuracy is not always easy to quantify.

With a $\mathrm{Beta}(2,3)$ prior, after observing $k=4$ successes in $n=10$ trials, what is the posterior distribution?

In the medical-test example on this page, a positive result gave only about a $16.7\%$ posterior chance of disease, far below the test's $99\%$ sensitivity. What does this illustrate?

Which statement correctly describes a $95\%$ Bayesian credible interval, in contrast to a $95\%$ frequentist confidence interval?

For a $\mathrm{Beta}(9,3)$ posterior, what is the posterior mean $E[\theta]$ ?

References

Andrew Gelman, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, Donald B. Rubin (2013). Bayesian Data Analysis (3rd ed.)
Matthew D. Hoffman, Andrew Gelman (2014). The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo · arXiv:1111.4246