Lecture Notes 1

36-705
Brief Review of Basic Probability
I assume you already know basic probability. Chapters 1-3 are a review. I will assume
you have read and understood Chapters 1-3. If not, you should be in 36-700.

Random Variables

Let be a sample space (a set of possible outcomes) with a probability distribution (also
called a probability measure) P . A random variable is a map X : R. We write
P (X A) = P ({ : X() A})
and we write X P to mean that X has distribution P . The cumulative distribution
function (cdf ) of X is
FX (x) = F (x) = P (X x).
A cdf has three properties:
1. F is right-continuous. At each x, F (x) = limn F (yn ) = F (x) for any sequence
yn x with yn > x.
2. F is non-decreasing. If x < y then F (x) F (y).
3. F is normalized. limx F (x) = 0 and limx F (x) = 1.
Conversely, any F satisfying these three properties is a cdf for some random variable.
If X is discrete, its probability mass function (pmf ) is
pX (x) = p(x) = P (X = x).
If X is continuous, then its probability density function function (pdf ) satisfies
Z
Z
p(x)dx
pX (x)dx =
P (X A) =
A

and pX (x) = p(x) = F 0 (x). The following are all equivalent:

X P,

X F,

X p.

Suppose that X P and Y Q. We say that X and Y have the same distribution
if P (X A) = Q(Y A) for all A. In that case we say that X and Y are equal in
d
distribution and we write X = Y .

Lemma 1 X = Y if and only if FX (t) = FY (t) for all t.

1

Expected Values

The mean or expected value of g(X) is

Z
E (g(X)) =

Z
g(x)dF (x) =

( R
g(x)p(x)dx if X is continuous

g(x)dP (x) = P g(xj )p(xj )

if X is discrete.
j

Recall that:
P
P
1. E( kj=1 cj gj (X)) = kj=1 cj E(gj (X)).
2. If X1 , . . . , Xn are independent then
E

n
Y

!
Xi

i=1

E (Xi ) .

3. We often write = E(X).

4. 2 = Var (X) = E ((X )2 ) is the Variance.
5. Var (X) = E (X 2 ) 2 .
6. If X1 , . . . , Xn are independent then
Var

n
X

!
ai X i

i=1

a2i Var (Xi ) .

7. The covariance is
Cov(X, Y ) = E((X x )(Y y )) = E(XY ) X Y
and the correlation is (X, Y ) = Cov(X, Y )/x y . Recall that 1 (X, Y ) 1.

The conditional expectation of Y given X is the random variable E(Y |X) whose
value, when X = x is
Z
E(Y |X = x) = y p(y|x)dy
where p(y|x) = p(x, y)/p(x).

The Law of Total Expectation or Law of Iterated Expectation:

Z


E(Y ) = E E(Y |X) = E(Y |X = x)pX (x)dx.
The Law of Total Variance is




Var(Y ) = Var E(Y |X) + E Var(Y |X) .
The moment generating function (mgf ) is


MX (t) = E etX .
d

If MX (t) = MY (t) for all t in an interval around 0 then X = Y .

(n)

Exercise (potential test question): show that MX (t)|t=0 = E (X n ) .

Transformations

Let Y = g(X) where g : R R. Then

Z
FY (y) = P(Y y) = P(g(X) y) =

pX (x)dx
A(y)

where
Ay = {x : g(x) y}.
The density is pY (y) = FY0 (y). If g is monotonic, then


 dh(y) 

pY (y) = pX (h(y)) 
dy 
where h = g 1 .
Example 2 Let pX (x) = ex for x > 0. Hence FX (x) = 1 ex . Let Y = g(X) = log X.
Then
FY (y) = P (Y y) = P (log(X) y)
y
= P (X ey ) = FX (ey ) = 1 ee
y

and pY (y) = ey ee for y R.

Example 3 Practice problem. Let X be uniform on (1, 2) and let Y = X 2 . Find the
density of Y .
3

Let Z = g(X, Y ). For example, Z = X + Y or Z = X/Y . Then we find the pdf of Z as

follows:
1. For each z, find the set Az = {(x, y) : g(x, y) z}.
2. Find the CDF
Z Z
FZ (z) = P (Z z) = P (g(X, Y ) z) = P ({(x, y) : g(x, y) z}) =

pX,Y (x, y)dxdy.

Az

3. The pdf is pZ (z) = FZ0 (z).

Example 4 Practice problem. Let (X, Y ) be uniform on the unit square. Let Z = X/Y .
Find the density of Z.

Independence

X and Y are independent if and only if

P(X A, Y B) = P(X A)P(Y B)
for all A and B.
Theorem 5 Let (X, Y ) be a bivariate random vector with pX,Y (x, y). X and Y are independent iff pX,Y (x, y) = pX (x)pY (y).
X1 , . . . , Xn are independent if and only if
P(X1 A1 , . . . , Xn An ) =

n
Y

P(Xi Ai ).

i=1

Qn

Thus, pX1 ,...,Xn (x1 , . . . , xn ) = i=1 pXi (xi ).

If X1 , . . . , Xn are independent and identically distributed we say they are iid (or that they
are a random sample) and we write
X1 , . . . , X n P

or

X1 , . . . , X n F

or

Important Distributions

Normal (Gaussian). X N (, 2 ) if
1
2
2
p(x) = e(x) /(2 ) .
2
4

X1 , . . . , Xn p.

If X Rd then X N (, ) if


1
T 1
exp (x ) (x ) .
p(x) =
(2)d/2 ||
2
P
Chi-squared. X 2p if X = pj=1 Zj2 where Z1 , . . . , Zp N (0, 1).
1

Bernoulli. X Bernoulli() if P(X = 1) = and P(X = 0) = 1 and hence

p(x) = x (1 )1x

x = 0, 1.

Binomial. X Binomial() if
 
n x
p(x) = P(X = x) =
(1 )nx
x

x {0, . . . , n}.

Uniform. X Uniform(0, ) if p(x) = I(0 x )/.

x

Poisson. X Poisson() if P (X = x) = e x! x = 0, 1, 2, . . .. The E (X) = Var (X) =

t
and MX (t) = e(e 1) . We can use the mgf to show: if X1 Poisson(1 ), X2 Poisson(2 ),
independent then Y = X1 + X2 Poisson(1 + 2 ).
Multinomial. The multivariate version of a Binomial is called a Multinomial. Consider
drawing a ball from an urn with has balls with kPdifferent colors labeled color 1, color
2, . . . , color k. Let p = (p1 , p2 , . . . , pk ) where
j pj = 1 and pj is the probability of
drawing color j. Draw n balls from the urn (independently and with replacement) and let
X = (X1 , X2 , . . . , Xk ) be the count of the number of balls of each color drawn. We say that
X has a Multinomial (n, p) distribution. The pdf is


n
p(x) =
px1 1 . . . pxkk .
x1 , . . . , x k
Exponential. X exp() if pX (x) = 1 ex/ , x > 0. Note that exp() = (1, ).
Gamma. X (, ) if
pX (x) =
for x > 0 where () =

R
0

1
x1 ex/
()

1 1 x/
x e
dx.

Remark: In all of the above, make sure you understand the distinction between random
variables and parameters.

More on the Multivariate Normal. Let Y Rd . Then Y N (, ) if



1
1
T 1
p(y) =
exp (y ) (y ) .
(2)d/2 ||1/2
2
5

Then E(Y ) = and cov(Y ) = . The moment generating function is



tT t
T
M (t) = exp t +
.
2
Theorem 6 (a). If Y N (, ), then E(Y ) = , cov(Y ) = .
(b). If Y N (, ) and c is a scalar, then cY N (c, c2 ).
(c). Let Y N (, ). If A is p n and b is p 1, then AY + b N (A + b, AAT ).
Theorem 7 Suppose that Y N (, ). Let
 
 


Y1
1
11 12
Y =
, =
, =
.
Y2
2
21 22
where Y1 and 1 are p 1, and 11 is p p.
(a). Y1 Np (1 , 11 ), Y2 Nnp (2 , 22 ).
(b). Y1 and Y2 are independent if and only if 12 = 0.
(c). If 22 > 0, then the condition distribution of Y1 given Y2 is
1
Y1 |Y2 Np (1 + 12 1
22 (Y2 2 ), 11 12 22 21 ).

(1)

Lemma 8 Let Y N (, 2 I), where Y T = (Y1 , . . . , Yn ), T = (1 , . . . , n ) and 2 > 0 is a

scalar. Then the Yi are independent, Yi N1 (, 2 ) and
 T 
Y TY

||Y ||2
2
=
n
.
2
2

2
Theorem 9 Let Y N (, ). Then:
(a). Y T 1 Y 2n (T 1 ).
(b). (Y )T 1 (Y ) 2n (0).

Sample Mean and Variance

Let X1 , . . . , Xn P . The sample mean is

Xn =

1X
Xi
n i

and the sample variance is

Sn2 =

1 X
(Xi X)2 .
n1 i

The sampling distribution of X n is

Gn (t) = P(X n t).
Practice Problem. Let X1 , . . . , Xn be iid with = E(Xi ) = and 2 = Var(Xi ) = 2 .
Then
2
E(X n ) = , Var(X n ) = , E(S 2 ) = 2 .
n
6

Theorem 10 If X1 , . . . , Xn N (, 2 ) then
2

(a) X n N (, n ).
(b)

2
(n1)Sn
2

2n1 .

(c) X n and Sn2 are independent.