Chapter 10B: Limit Theorems  
MATH/STAT 394: Probability I  
Arman Jahangiri  
University of Washington Department of Mathematics  
Summer 2026  
Arman Jahangiri  
Chapter 10B: Limit Theorems  
Summer 2026  
1
 
Learning Goals  
By the end of this lecture, students should be able to:  
▶ understand the Law of Large Numbers;  
▶ distinguish convergence in probability from exact equality;  
▶ apply the Weak Law of Large Numbers;  
▶ understand Monte Carlo integration;  
▶ define the empirical CDF;  
▶ understand the Central Limit Theorem;  
▶ standardize sums and averages;  
▶ apply Normal approximations to Binomial and Poisson distributions.  
Arman Jahangiri  
Chapter 10B: Limit Theorems  
Summer 2026  
2
Why Limit Theorems Matter  
Arman Jahangiri  
Chapter 10B: Limit Theorems  
Summer 2026  
3
Order Emerging from Randomness  
Individual random outcomes are unpredictable.  
But large collections of random variables often display remarkable regularity.  
Examples:  
▶ sample averages stabilize;  
▶ proportions settle down;  
▶ histograms become bell-shaped;  
▶ randomness averages out.  
Main message  
Large-scale randomness becomes highly structured.  
Arman Jahangiri  
Chapter 10B: Limit Theorems  
Summer 2026  
4
Two Fundamental Theorems  
The two most important limit theorems are:  
1
Law of Large Numbers (LLN)  
averages converge to expectations;  
2
Central Limit Theorem (CLT)  
normalized sums become approximately Normal.  
Together, these form the foundation of modern statistics.  
Arman Jahangiri  
Chapter 10B: Limit Theorems  
Summer 2026  
5
Law of Large Numbers (LLN)  
Arman Jahangiri  
Chapter 10B: Limit Theorems  
Summer 2026  
6
Weak Law of Large Numbers  
Theorem (WLLN)  
Let X1, X2, . . . be i.i.d. random variables with finite mean µ (and finite variance σ2)  
Then  
ꢀ
ꢁ
ꢀ
ꢁ
¯
¯
∀ ε > 0,  
lim P |Xn − µ| > ε = 0, or equivalently,  
lim P |Xn − µ| ≤ ε = 1.  
n→∞  
n→∞  
Interpretation  
¯
As the sample size grows, the sample mean Xn becomes arbitrarily close to the population mean µ with  
probability approaching 1.  
Notation (Convergence in Probability): This phenomena is also known as convergence in probability, and is  
denoted by  
P
¯
Xn −→ µ.  
Arman Jahangiri  
Chapter 10B: Limit Theorems  
Summer 2026  
7
Proof of WLLN Using Chebyshev’s Inequality  
Recall:  
¯
E(Xn) = µ,  
and  
σ2  
¯
Var(Xn) =  
.
n
Applying Chebyshev:  
¯
Var(Xn)  
¯
P(|Xn − µ| > ε) ≤  
.
ε2  
Thus:  
σ2  
nε2  
¯
P(|Xn − µ| > ε) ≤  
.
As  
n → ∞,  
the right-hand side goes to  
0.  
Arman Jahangiri  
Chapter 10B: Limit Theorems  
Summer 2026  
8
Coin Toss Example  
Suppose:  
(
1, Heads,  
Xi =  
0, Tails.  
Then E(Xi ) = p. The sample mean is:  
n
X
1
¯
Xn =  
Xi = proportion of Heads.  
n
i=1  
P
The LLN says Xn −→ p. Thus, observed proportions stabilize near the true probability p.  
Arman Jahangiri  
Chapter 10B: Limit Theorems  
Summer 2026  
9
LLN’s Connection to Relative Frequency (Probability’s Naive Definition)  
The coin toss example in the previous slide gives a mathematical justification for the frequency interpretation  
of probability (naive probability) introduced earlier in the course.  
Let A ⊆ S be an event. Define the indicator random variables  
(
1, if A occurs on trial i,  
Xi =  
0, otherwise.  
Then Xi ∼ Bernoulli(P(A)), so E(Xi ) = P(A). The sample mean becomes  
n
X
1
#{times A occurs in n trials}  
¯
Xn =  
Xi =  
,
n
n
i=1  
which is the relative frequency of the event A (exactly what we previously defined as (naive) definition of  
probability). Hence, by the LLN,  
P
Xn −→ P(A) ,  
or less formally put,  
P
naive definition of probability −−−−−−−−−−−−−−−−−−−−−−−−−→ Axiomatic Definition of Probability  
number of times we repeat the experiment→∞  
Chapter 10B: Limit Theorems  
Arman Jahangiri  
Summer 2026 10  
Strong Law of Large Numbers  
Strong Law  
Under the same assumptions of WLLN, we have  
¯
P( lim Xn = µ) = 1.  
n→∞  
Equivalently,  
P({s ∈ S : lim X(s) = µ}) = 1  
n→∞  
Notation (Almost-Sure Convergence): This phenomenon is also known as almost-sure convergence or  
almost-everywhere convergence, and is denoted by  
a.s.  
a.e.  
¯
¯
Xn −−→ µ, or Xn −−→ µ  
Remark: Almost-sure convergence implies (i.e., is stronger than) convergence in probability. Thus, SLLN is  
stronger than WLLN. The proof requires graduate-level background of Probability (Borel-Canteli Lemma).  
Arman Jahangiri  
Chapter 10B: Limit Theorems  
Summer 2026 11  
LLN Application: Monte Carlo Integration  
Arman Jahangiri  
Chapter 10B: Limit Theorems  
Summer 2026 12  
Monte Carlo Integration  
Suppose we want to compute  
Z
1
g(x) dx.  
0
i.i.d.  
Let U1, . . . , Un ∼ Unif(0, 1).  
Then  
Z
1
E(g(Ui )) =  
g(x) dx.  
0
Hence,  
Z
n
X
1
1
E[  
g(Uj )] =  
g(x) dx.  
n
0
j=1  
By the WLLN,  
Z
Z
n
X
1
n
1
1
X
1
P
g(Uj ) −→  
g(x) dx, and thus,  
g(Uj ) ≈  
g(x) dx for large n.  
n
n
0
0
j=1  
j=1  
This is called Monte Carlo estimation of integrals. Draw n random points on the interval, evaluate the function g  
on them, and then average. The larger the n the closer the average to the true integral.  
Arman Jahangiri  
Chapter 10B: Limit Theorems  
Summer 2026 13  
LLN Application: Empirical Distribution Function  
Arman Jahangiri  
Chapter 10B: Limit Theorems  
Summer 2026 14  
Empirical CDF  
Given observations:  
X1, . . . , Xn ∼ F  
define the empirical CDF:  
n
X
1
ˆ
Fn(x) =  
I(Xj ≤ x).  
n
j=1  
Interpretation:  
ˆ
Fn(x) = fraction of observations ≤ x.  
For fixed x, the indicators I(Xj ≤ x) are i.i.d. Bernoulli random variables with mean F(x) = P(X ≤ x). Thus, by  
the LLN:  
ˆ
Fn(x) → F(x).  
The empirical CDF approximates the true CDF.  
Arman Jahangiri  
Chapter 10B: Limit Theorems  
Summer 2026 15  
Central Limit Theorem  
Arman Jahangiri  
Chapter 10B: Limit Theorems  
Summer 2026 16  
Central Limit Theorem  
Theorem (CLT)  
Let X1, X2, . . . be i.i.d. random variables with E(Xi ) = µ and Var(Xi ) = σ2 < ∞.  
Then for every z ∈ R,  
ꢂ
ꢃ
¯
(Xn − µ)  
√
∀z ∈ R, lim P  
n→∞  
≤ z = Φ(z),  
σ/  
n
Interpretation  
We previously learned that E[Xn] = µ and Var[Xn] = 2 , which means that the fraction  
¯
X
−µ  
is the  
n
σ
n
n
¯
¯
√
σ/  
¯
¯
standardized version of Xn. CLT says that for large n, the sample mean Xn is approximately normal:  
ꢄ
ꢅ
2
σ
¯
Xn ≈ N µ,  
. This is especially important because we do not assume any distribution for X1, . . . Xn yet  
n
the sample mean follows a normal distribution for large n. This offers that Normality emerges universally  
from averaging.  
Notation (Convergence in Distribution): This phenomenon is also known as convergence in distribution, and is  
¯
d
X
σ/  
−µ  
n
√
denoted by  
−→ N(0, 1).  
n
Arman Jahangiri  
Chapter 10B: Limit Theorems  
Summer 2026 17  
CLT Application: Normal’s Approximation to Binomial  
We have previously seen that a binomial r.v. can be written as a sum of n i.i.d. Bernouli r.v.s. Let X ∼ Bin(n, p).,  
and X = X1 + · · · + Xn, where X1, . . . , Xn ∼ Ber(p).  
Since:  
µ = E[Xi ] = p,  
σ2 = Var[Xi ] = p(1 − p),  
the CLT gives:  
¯
¯
Xn − E[Xn]  
X/n − p  
X − np  
√
p
p
=
=
≈ N(0, 1).  
¯
Var[Xn]  
p(1 − p)/n  
np(1 − p)  
Arman Jahangiri  
Chapter 10B: Limit Theorems  
Summer 2026 18  
Continuity Correction  
A subtle point here is that Binomial distributions are discrete, while Normal distributions are continuous.  
A better approximation uses a continuity correction:  
ꢂ
ꢃ
1
2
P(X ≤ k) ≈ P Y ≤ k +  
,
where  
Y ∼ N(np, np(1 − p)).  
Idea  
The correction accounts for the width of discrete bins.  
Arman Jahangiri  
Chapter 10B: Limit Theorems  
Summer 2026 19  
Poisson to Normal  
Suppose:  
For large  
X ∼ Pois(λ).  
λ,  
X − λ  
√
≈ N(0, 1).  
λ
Equivalently:  
X ≈ N(λ, λ).  
Arman Jahangiri  
Chapter 10B: Limit Theorems  
Summer 2026 20  
Why the CLT Is Remarkable  
The original distribution can be:  
▶ discrete;  
▶ continuous;  
▶ skewed;  
▶ highly non-Normal.  
Yet averages still become approximately Normal.  
Universality  
The CLT explains why Normal distributions appear everywhere.  
Arman Jahangiri  
Chapter 10B: Limit Theorems  
Summer 2026 21  
Summary  
▶ The LLN says averages stabilize.  
▶ Monte Carlo integration is justified by the LLN.  
▶ The empirical CDF approximates the true CDF.  
▶ The CLT says normalized sums become approximately Normal.  
▶ Standard errors decrease like  
1
√
.
n
▶ Normal approximations apply to Binomial and Poisson distributions.  
Next lecture  
Chi-Square and Student t distributions.  
Arman Jahangiri  
Chapter 10B: Limit Theorems  
Summer 2026 22