1 Introduction to Probability
1.1 Introduction
• Ω = { H, T }
• F = {∅, { H }, { T }, { H, T }}
• P(∅) = 0
• P({ H, T }) = 1
1. Ω ∈ F
2. For A ∈ F , Ac ∈ F
3. For Ai ∈ F , i = 1, 2, · · ·, ∪i Ai ∈ F
Remarks
1
Studies in Economic Statistics Jae-Young Kim
1. A ⊂ σ(A).
2. σ(A) is a σ-field.
• Ω = {1,2,3,4,5,6}
• A = {1,3,5}
• A = {A}
⇒ σ(A) = {A, Ac , ∅, Ω}
1. P( A) ≥ 0, ∀ A ∈ F
2. P(Ω) = 1
∪
3. For Ai ∩ Aj = ∅, i ̸= j, P( i Ai ) = ∑i P ( Ai )
Remarks
• The three properties given above are often referred to as the axioms of
probability.
• A probability (measure) has the range on [0, 1], and a measure has the
range on [0, ∞].
• Outer measure of A
2
Studies in Economic Statistics Jae-Young Kim
• Inner measure of A
µ∗ ( A) = 1 − µ∗ ( Ac )
• Lebesque Measure: µ( A) = µ∗ ( A) = µ∗ ( A)
• Extension
∪
n n
P( Ak ) = ∑ P( Ak ) − ∑ P( Ai ∩ A j )+
k =1 k =1 i< j
· · · + (−1) n +1
P ( A1 ∩ A2 ∩ · · · ∩ A n )
• Boole’s inequality
∞
∪ ∞
P( Ai ) ≤ ∑ P ( Ai )
i =1 i =1
3
Studies in Economic Statistics Jae-Young Kim
P(lim En ) = lim P( En )
Proof.
• E0 = ϕ, En : monotone increasing
• Fn = En − En−1 , P( Fi ) = P( Ei ) − P( Ei−1 )
∪n ∪n
• P( i =1 Fi ) = ∑in=1 P( Fi ) = P( En ) = P( i =1 Ei )
Definition 1.7 (Limit Supremum and Limit Infimum of Events). For a se-
quence of events En , define
∞ ∪
∩ ∞
lim sup En = Ek (∀n ≥ 1, ∃k ≥ n such that ω ∈ Ek , En infinitely often)
n n =1 k = n
∪∞ ∩ ∞
lim inf En = Ek (∃n ≥ 1 such that ∀k ≥ n, ω ∈ Ek , En eventually)
n
n =1 k = n
4
Studies in Economic Statistics Jae-Young Kim
Proof.
∞ ∪
∩ ∞ ∞
∪ ∞
P(lim sup En ) = P( Ek ) ≤ P( Ek ) ≤ ∑ P(Ek ) → 0
n =1 k = n k=n k=n
Remarks
Note that if P( En ) → 0, P(lim inf En ) = 0
Lemma 1.2 (2nd Borel - Cantelli Lemma). Let { En } be a independent sequence
of events.
∞
If ∑ P(Ei ) = ∞, then P(lim sup En ) = 1
i =1
• If A ⊥ B, then P( A ∩ B) = P( A) P( B).
• If A ⊥ B, then P( A | B) = P( A).
P( A∩ B) P( A) P( B)
• P( A | B) = P( B)
= P( B)
= P( A)
Remarks
If A or B is empty, then they are always independent.
Definition 1.10 (Pairwise Independence).
• Let Γ be a class of subsets of Ω.
5
Studies in Economic Statistics Jae-Young Kim
P( A∩ B) P( B| A) P( A)
• P( A | B) = P( B)
= P( B| A) P( A)+ P( B| Ac ) P( Ac )
• Ai , i = 1, 2, . . . , n
• Ai , i = 1, 2, . . . , n, a partition of Ω, P( Ai ) > 0
• Events H ∈ F , P(· | H ) = PH
• Let B ⊂ Ω be observable.
P( Hi ) P( B| Hi )
• P( Hi | B) = ∑in=1 P( Hi ) P( B| Hi )
Y = Xβ + ε
• Bayesian Approach
6
Studies in Economic Statistics Jae-Young Kim
1. PX ( R) = 1
2. PX ( B) ≥ 0 for any B ∈ B
3. For Bi , i = 1, 2, . . . , with Bi ∩ Bj = ∅
PX (∪i Bi ) = ∑ P( Bi )
i
7
Studies in Economic Statistics Jae-Young Kim
FX ( x2 ) − FX ( x1 ) = PX {( x1 , x2 ]}
1. limx→−∞ FX ( x ) = 0, limx→+∞ FX ( x ) = 1
Remarks
A distribution function is not necessarily left continuous.
8
Studies in Economic Statistics Jae-Young Kim
ρ( x0 , x ) < δ ⇒ ρ′ [ f ( x0 ), f ( x )] < ϵ
• We say that f is uniformly continuous if for any given ϵ > 0, ∃δ > 0 such
that, for any x1 , x2 ∈ X,
ρ( x1 , x2 ) < δ ⇒ ρ′ ( f ( x1 ), f ( x2 )) < ϵ.
Remarks
k k
∑ ( a i , bi ) < δ ⇒ ∑ | f (bi ) − f (ai )| < ϵ
i =1 i =1
Remarks
9
Studies in Economic Statistics Jae-Young Kim
- If Q( A) = 0 ⇒ P( A) = 0 ∀ A ∈ F
Example 2.1.
∫
• P( A) = A
f dQ, A ∈ F
∫x
• FX ( x ) = −∞ f (t)dt
• f X ( xi ) > 0 for x = xi , i = 1, . . . , k
• f X ( x ) = 0 for x ̸= xi
• ∑ f X ( xi ) = 1
Remarks
• Some other names of p.m.f are Discrete density function, discrete fre-
quency function, and probability function.
• f X ( x ) ≥ 0, ∀ x
∫∞
• −∞ f X ( x )dx = 1
10
Studies in Economic Statistics Jae-Young Kim
Remarks
• Some other names of p.d.f are Density function, continuous density func-
tion, and integrating density function.
• P [ X = xi ] = 0
dFX ( x )
• f X (x) = dx
∫b
• P( a < X ≤ b) = F (b) − F ( a) = a
f ( x )dx
Proof.
11
Studies in Economic Statistics Jae-Young Kim
∫ ∫
Then, E[ g1 ( X )] − E[ g2 ( X )] = g1 ( x ) f ( x )dx − g2 ( x ) f ( x )dx
∫
= [ g1 ( x ) − g2 ( x )] f ( x )dx ≤ 0.
Remarks
4. If X ⊥ Y, then E( XY ) = E( X ) E(Y )
Proof.
∫ ∫
1. c f ( x )dx = c f dx = c · 1 = c
2. Trivial.
∫∫ ∫∫ ∫∫
3. E( X + Y ) = ( x + y) f ( x, y)dxdy = x f ( x, y)dxdy + y f ( x, y)dxdy
∫ ∫ ∫ ∫ ∫ ∫
= x [ f ( x, y)dy]dx + y[ f ( x, y)dx ]dy = x f ( x )dx + y f (y)dy
= E( X ) + E(Y ))
12
Studies in Economic Statistics Jae-Young Kim
Example 2.2.
1
1. E( X ) = ∑i xi f i , X̄ = n ∑ xi
Theorem 2.7. For 0 < s < r, if E[| X |r ] exists, then E[| X |s ] < ∞.
Remarks
∫
• There must exist h > 0 such that MX (t) = E[etx ] = etx f ( x )dx for
−h < t < h.
• The moment generating function (mgf) does not always exist for a
random variable X.
Example 2.3.
13
Studies in Economic Statistics Jae-Young Kim
Remarks
• MX ( t ) → mr
14
Studies in Economic Statistics Jae-Young Kim
Measures of Dispersion
Skewness
Kurtosis
15
Studies in Economic Statistics Jae-Young Kim
2.5 Inequalities
Theorem 2.8 (Markov Inequality). Let X be a random variable and g(·) a non-
negative Borel measurable function. Then, for every k > 0,
E[ g( X )]
P[ g( X ) ≥ k] ≤
k
Proof.
∫ ∫ ∫
E[ g( X )] = g( x ) f ( x )dx = g( x ) f ( x )dx + g( x ) f ( x )dx
X:g( x )≥k X:g( x )<k
∫ ∫
≥ g( x ) f ( x )dx ≥ k f ( x )dx
X:g( x )≥k X:g( x )≥k
∫
≥k f ( x )dx = kP[ g( X ) ≥ k]
X:g( x )≥k
Example 2.4.
• g( x ) =| X |, g( x ) =| X |α
⇒ ( E[ X ]) = l ( E[ X ]) = E[l ( X )] ≤ E[ g( X )]
1 1
E[ XY ] ≤ E[| X | p ] p E[| Y |q ] q
16
Studies in Economic Statistics Jae-Young Kim
Example 2.5.
∴ −1 ≤ ρ XY = √ Cov( X,Y )
√ ≤1
Var ( X ) Var (Y )
• i.e. X −1 ( I ) = {ω : X1 (ω ) ≤ x1 , · · · , Xn (ω ) ≤ xn } ∈ F .
for each ( x1 , · · · , xn ) ∈ Rn
17
Studies in Economic Statistics Jae-Young Kim
c f . lim0<h→0 F ( x + h, y) = lim0<h→0 F ( x, y + h) = F ( x, y)
4. F ( x2 , y2 ) − F ( x2 , y1 ) − F ( x1 , y2 ) + F ( x1 , y1 ) ≥ 0 (∵ P[ x1 ≤ X ≤ x2 , y1 ≤
Y ≤ y2 ] ≥ 0)
• f X ( x ) > 0 for x = ai , i = 1, . . . , k
• f X ( x ) = 0 for x ̸= ai
• ∑i f X ( ai ) = 1
Remarks
• f ( x1 , . . . , xn ) ≥ 0,∀( x1 , . . . , xn )
∂n F ( x ,...,x )
• f ( x1 , . . . , xn ) = ∂x1 ···
1
∂xn
n
∫∞ ∫∞
• −∞ · · · −∞ f (t1 , t2 , . . . , tn )dt1 · · · dtn = 1
18
Studies in Economic Statistics Jae-Young Kim
• (Discrete case)
f X ( x ) = ∑nj=1 f ( xi , y j )
f Y (y) = ∑in=1 f ( xi , y j )
• (Continuous case)
∫
f X ( x ) = f ( x, y)dy
∫
f Y (y) = f ( x, y)dx
FX |Y ( x | y) = P( X ≤ x | Y = y)
∂FX |Y ( x | y)
f X |Y ( x | y ) = (Continuous)
∂x
f X |Y ( x | y ) = P ( X = x | Y = y ) (Discrete)
∫ x
FX |Y ( x | y) = f (u | y)du
−∞
Remarks
∫x f X,Y (u,y)
• FX |Y ( x | y) = −∞ f Y (y)
du
f X,Y ( x, y)
f X |Y ( x | y ) = if f Y (y)>0
f Y (y)
19
Studies in Economic Statistics Jae-Young Kim
P( A x ∩ By )
f X |Y ( x | y) = P( X = x | Y = y) = P( A x | By ) =
P( By )
P({w : X (w) = x, Y (w) = y}) f ( x, y)
= = X,Y
P({w : Y (w) = y}) f Y (y)
f X,Y ( x, y) = f X ( x ) f Y (y) ( P( A x ∩ By ) = P( A x ) P( By ))
FX,Y ( x, y) = FX ( x ) FY (y) ∀( x, y) ∈ R2
Proof.
⇐) By partial differentiations
⇒) FX,Y ( x, y)
20
Studies in Economic Statistics Jae-Young Kim
Proof.
21
Studies in Economic Statistics Jae-Young Kim
µr1 ,r2 = E[( X1 − µ1 )r1 ( X2 − µ2 )r2 ]: (r1 , r2 )th cross central moment
Remarks
∂r1 +r2 MX,Y (t1 ,t2 )
µr′ 1 ,r2 = r r | t1 = t2 =0
∂t11 ∂t22
MX1 ,X2 ,··· ,Xn (t1 , t2 , · · · tn ) = MX1 (t1 ) MX2 (t2 ) · · · MXn (tn )
E[ g1 ( X ) g2 (Y )] = E[ g1 ( X )][ g2 (Y )]
Remarks
If X1 , X2 , . . . , Xn are independent,
n
Var (S) = ∑ a2i Var(Xi )
i =1
22
Studies in Economic Statistics Jae-Young Kim
Remarks
1. E[c|Y ] = c, c : consant
3. P[ X ≥ 0] = 1 ⇒ E [ X |Y ] ≥ 0
4. P[ X1 ≥ X2 ] = 1 ⇒ E [ X1 | Y ] ≥ E [ X 2 | Y ]
23
Studies in Economic Statistics Jae-Young Kim
Proof.
∫ ∞ ∫ ∞
h( x ) f X,Y ( x, y)dxdy
−∞ −∞
∫ ∞ ∫ ∞
f X,Y ( x, y)
= [ h( x ) dx ] f Y (y)dy
−∞ −∞ f Y (y)
∫ ∞
= E[h( X )|y] f Y (y)dy = E[ E[h( X )|Y ]]
−∞
= E[h( X )]
1. Var ( X |Y ) = E[ X 2 |Y ] − ( E[ X |Y ])2
Proof.
2. E[Var ( X |Y )] = E[ E[ X 2 |Y ] − ( E[ X |Y ])2 ]
=E[ X 2 ] − ( E[ X ])2 − ( E[( E[ X |Y ])2 ] − ( E[ X ])2 )
=Var ( X ) − Var ( E[ X |Y ])
∴ Var ( X ) = E[Var ( X |Y )] + Var ( E[ X |Y ])
24