54 tayangan

Judul Asli: Lecture Notes-March06 (1)

Diunggah oleh rukma

Lecture Notes-March06 (1)

(Stochastic Models and Methods)

© All Rights Reserved

- UWO 1228 Final Exam 2010
- veerarajan
- Introduction to Econometrics with R
- Probability Distributions
- Bunge.1981.Four concepts of probability.pdf
- Review of Probability
- E_C-_8_1_.3.10
- lecture1-converted.pptx
- Statistics Probability Midterm Cheat Sheet
- 2019 03 Exam p Syllabi
- JQT - Planned Replacement - Some Theory and Its Application
- Statistic Tables
- 100 problems in stochastic processes
- l 05 Stats i Discrete Prob Dist s
- SSM-UNIT 5 6.doc
- Probability Concepts, Models and Axioms
- Probability - Free Guide
- class03_7
- Lecture 6
- 05 Random Signal

Anda di halaman 1dari 43

by Hossein Abouee Mehrizi, University of Waterloo

Remark: These are point-form summary of the lectures for MSCI 431. There is no

guarantee for completeness and accuracy, and therefore, they should not be regarded as a

substitutable for attending course lectures. Lectures are based on the book titled Introduction to Probability Models by Sheldon M. Ross.

1. Introduction to Probability Theory:

Lecture 1:

Probability

Experiment: any process whose outcome is not known in advance.

Flip a coin

Roll a die

Sample space: set of all possible outcomes.

Example. Flipping a coin: S = {H, T }

Example. Rolling a die: S = {1, 2, ..., 6}

Example. Flipping two coins: S = {(H, H), (H, T ), (T, H), (T, T )}

Example: Rolling two dice: S = {(m, n) : 1 m, n 6}

Event: subset of the sample space.

Example. Flipping a coin: E = {H}, the event that a head appears.

Example. Rolling a die: E = {2, 4, 6}, the event that an even number appears.

S

Union of events E and F (E F ): all outcomes that are either in E, or F , or both.

S

Example. Flipping a coin: E = {H}, F = {T }. Then, E F = {H, T }.

T

Intersection of events E and F (E F ): all outcomes that are in both E and F .

T

Example. Rolling a die: E = {1, 3, 5}, F = {1, 2, 3}. Then, E F = {1, 3}.

Consider the events E1 , E2 , .... Then,

S

Union of these events,

i=1 Ei , is a new events that includes all outcomes that

are in En for one value of n = 1, 2, ....

T

Intersection of these events,

i=1 Ei , is a new events that includes all outcomes

that are in all En for n = 1, 2, ...

1

Complement of E (E c ): outcomes that are in the sample space S and are not in E.

Probability: Consider an experiment with the sample space S. For an event E S,

we assume that P (E) is defined and satisfies

(i) 0 P (E) 1,

(ii) P (S) = 1,

(iii) ForSa sequenceP

of the events E1 , E2 , ... that are mutually exclusive (En

P ( i=1 Ei ) =

i=1 P (Ei ).

Em = , n 6= m),

Example. Rolling a die: P ({1, 3, 5}) = P ({1}) + P ({3}) + P ({5}) = 1/2.

Two basic properties of probability:

P (E c ) = 1 P (E).

P (E F ) = P (E) + P (F ) P (E F ).

Example. Flipping two coins:

E = {(H, H), (H, T )}: head appears on the first coin

E = {(H, H), (T, H)}: head appears on the second coin

P (E F ) = 1/2 + 1/2 1/4 = 3/4.

Conditional Probability

Example. Rolling two dice: Suppose we observe that 4 has appeared on the first die.

What is the probability that the sum is 6?

E: event that 4 appears on the first die

F : event that sum of two dice is 6

P (E|F ): the probability that the sum of two dice is 6 given that 4 has appeared

on the first die.

The sample space reduces to {(4, 1), (4, 2), (4, 3), (4, 4), (4, 5), (4, 6)} given that 4

has appeared on the first die. Therefore, P (E|F ) = 1/6.

P (E|F ) =

P (E F )

.

P (F )

Example. Suppose cards are numbered 1 to 10 and they are placed in a hat and one

of them is drawn. We are told that the number on the drawn card is at least 5. What

is the probability that the number on the drawn card is 10?

E: event that number on the drawn card is 10

F : event that number on the drawn card is at least 5

P (E F )

1/10

P (E|F ) =

=

= 1/6.

P (F )

6/10

Lecture 2:

Conditional Probability (continue ...)

Example. A family has two children. What is the probability that both are boys given

that at least one of them is a boy?

E: event that both are boys.

F : event that at least one of them is a boy.

P (E F )

1/4

P (E|F ) =

=

= 1/3.

P (F )

3/4

Example. Suppose that an urn contains 7 black balls and 5 white balls. We draw two

balls from the urn without replacement. What is the probability that both balls are

black?

E: event that the second one is black.

F : event that the first one is black.

P (E F ) = P (F )P (E|F ) = (7/12)(6/12) = 42/132.

Example. Bev can take a course in computers or chemistry. If Bev takes the computer

course, then she will receive an A grade with probability 1/2. If she takes the chemistry

course, then she will receive an A grade with probability 1/3. Bev decides to base her

decision on the flip of a fair coin. What is the probability that Bev will get an A in

chemistry?

E: event that she receives an A.

F : event that she takes chemistry.

P (E F ) = P (F )P (E|F ) = (1/2)(1/3) = 1/6.

Example. Suppose that each of three men at a party throws his hat into the center of

the room. Each man randomly selects a hat. What is the probability that none of the

three men selects his hat?

S S

T

T

Remark.

P

(E

E

E

)

=

P

(E

)+P

(E

)+P

(E

)P

(E

E

)P

(E

E3 )

1

2

3

1

2

3

1

2

1

T

T T

P (E2 E3 ) + P (E1 E2 E3 ).

Ei : event that ith man selects his own hat.

S S

P (E1 E2 E3 ): probability that at least one of them selects his own hat.

S S

1 P (E1 E2 E3 ): probability that none of them selects his own hat.

P (Ei ) = 1/3, i = 1, 2, 3

Since each man is equally likely to select any of them.

T

P (E1 E2 ) = P (E1 )P (E2 |E1 ) = (1/3)(1/2) = 1/6.

Since given that the first man has selected his own hat, there remain two hats

that the second man may select.

3

Ej ) = (1/3)(1/2) = 1/6, i 6= j.

T

T

T

P (E1 E2 E3 ) = P (E1 E2 )P (E3 |E1 E2 ) = (1/6)(1) = 1/6.

T

P (E3 |E1 E2 ) = 1: since given that the first two men get their own hats, it

follows that the third man must also get his own hat.

S S

T

T

T

P (E1 T E2 T E3 ) = P (E1 )+P (E2 )+P (E3 )P (E1 E2 )P (E1 E3 )P (E2 E3 )+

P (E1 E2 E3 ) = 1/3 + 1/3 + 1/3 1/6 1/6 1/6 + 1/6 = 2/3.

S S

1 P (E1 E2 E3 ) = 1 2/3 = 1/3.

P (Ei

Independent Events

Two events E and F are independent if P (E

F ) = P (E)P (F ).

Example. Suppose we toss two fair dice.

Ei : event that sum of the two dice is 6.

F : event that first die is 4.

T

P (E1 F ) = P ({4, 2}) = 1/36.

P (E1 )P (F ) = (5/36)(1/6) = 5/216.

T

P (E1 F ) 6= P (E1 )P (F )!

Lecture 3:

Independent Events

Two events E and F are independent if P (E

F ) = P (E)P (F ).

Example. Suppose we toss two fair dice.

Ei : event that sum of the two dice is 6.

F : event that first die is 4.

T

P (E1 F ) = P ({4, 2}) = 1/36.

P (E1 )P (F ) = (5/36)(1/6) = 5/216.

T

P (E1 F ) 6= P (E1 )P (F )!

Since our chance of getting a total of six depends on the outcome of the first

die.

E2 : event that sum of the dice is 7.

T

P (E1 F ) = P ({4, 3}) = 1/36.

P (E1 )P (F ) = (1/6)(1/6) = 1/36.

T

P (E1 F ) = P (E1 )P (F )!

Bayes Formula

Let E and F be events. Then,

T S T

E = (E F ) (E F c ).

Since a point is is E if it is either in both E and F , or in E and not F .

T

T

P (E) = P (E F ) + P (E F c ) = P (F )P (E|F ) + P (F c )P (E|F c ).

Example. Consider 2 urns. The first contains 2 white and 7 black balls, and the second

contains 5 white and 6 black balls. We flip a fair coin and then draw a ball from the

first urn or second urn depending on whether the outcome was heads or tails. What

is the probability that the outcome of the toss was heads given that a white ball was

selected?

W : Event that a white ball is drawn.

H: Event that the coin comes up heads.

P (H|W ): probability that the outcome of the coin is heads given that a white

ball was selected.T

P (H W )

.

P (H|W ) =

W

T

P (H W ) = P (H)P (W |H) = (1/2)(2/9).

5

T

(1/2)(2/9)

P (H W )

=

= 22/67.

P (H|W ) =

W

(1/2)(2/9) + (1/2)(5/11)

Example. In answering a question with five-choice test, a student knows the correct

answer with probability 1/2 and guess with probability 1/2. Assume that a student

who guesses at the answer will be correct with probability 1/5. What is the probability

that a student knew the correct answer given that she answered it correctly?

C: event that the student answers it correctly

K: event that the

T student knows the answer

P (K)P (C|K)

(1/2)(1)

P (K C)

=

=

=

P (K|C) =

c

c

C

P (K)P (C|K) + P (K )P (C|K )

(1/2)(1) + (1/2)(1/5)

5/6.

2. Random Variables:

In performing an experiment we are often interested in some functions of the outcome

as opposed to the the outcome itself.

Example. Rolling two dice: we interested in sum of the two dice.

Random variables: real-valued functions defined on the sample space.

Example. Flipping two fair coins: Let Y denote the number of heads appearing.

P (Y = 0) = P ({T, T }) = 1/4.

P (Y = 1) = P ({T, H}, {H, T }) = 2/4.

P (Y = 2) = P ({H, H}) = 1/4.

P (Y = 0) + P (Y = 1) + P (Y = 2) = 1.

until the first head appears. Let N denote the number of flips required, and

assume that the outcome of successive flips are independent.

P (N = 1) = P ({H}) = p.

P (N = 2) = P ({T, H}) = (1 p)p.

P (N = 3) = P ({T, T, H}) = (1 p)2 p.

variable X is defined for any real number b , b , by F (b) = P (X b).

6

F (b) denotes the probability that the random variable X takes on a value less

than or equal to b.

Example. Flipping two fair coins: Let Y denote the number of heads

F (0) = P (Y 0) = P (Y = 0) = 1/4.

F (1) = P (Y 1) = P (Y = 0) + P (Y = 1) = 3/4.

F (2) = P (Y 2) = P (Y = 0) + P (Y = 1) + P (Y = 2) = 1.

A random variable that can take on at most a countable number of possible values is

said to be discrete.

P (a) = P (X = a): probability mass function of X.

Example. Flipping two fair coins: Let Y denote the number of heads

probability mass function of Y : P (0) = 1/4, P (1) = 2/4, P (2) = 1/4.

CDF of Y : F (0) = 1/4, F (1) = 3/4, F (2) = 1.

Lecture 4:

Discrete Random Variables:

Discrete random variables are often classified according to their probability mass functions.

The Bernoulli Random Variable

Consider a trial, or an experiment whose outcome can be classified as either a

success or failure.

Let X equal 1 if the outcome is a success and 0 if the outcome is a failure.

Let 0 p 1 denote the probability that the trial is a success.

The probability mass function of X is

P (0) = P (X = 0) = 1 p,

P (1) = P (X = 1) = p.

X is a Bernoulli random variable with parameter p.

Example. Flipping a fair coin: consider heads as a success and tails as a failure.

P (0) = 1/2, P (1) = 1/2.

The Binomial Random Variable

Suppose that n independent trails, each of which results in a success with

probability p and in a failure with probability 1 p, are to be performed.

Let X represent the number of successes that occur in the n trials.

X is a binomial random variable with parameters (n, p).

Probability mass function of a binomial random variable having parameters (n, p):

n!

pi (1 p)ni , i = 0, 1, , n.

P (i) = ni pi (1 p)ni =

(n i)!i!

Example. Suppose that each patient in a hospital is discharged on day t with

probability p. What is the distribution of the number of discharged patients on

day t given that there are n patients in the hospital on that day?

X: number of successes (discharged patients).

There are n trials and each of them is a success with probability p.

Therefore, the distribution of the number of discharged patients is binomial

n!

pi (1p)ni , i = 0, 1, , n.

with parameters (n, p): ni pi (1p)ni =

(n i)!i!

Example. It is known that any item produced by a certain machine will be

defective with probability 0.1, independently of any other item. What is the

probability that in a sample of three items, at most one item will be defective?

X: number of defective items in the sample.

8

Probability at most one defective in the sample: P (X = 0) + P (X = 1).

P (X = 0) + P (X = 1) = 30 (0.1)0 (1 0.1)3 + 31 (0.1)1 (1 0.1)2 = 0.972.

The Geometric Random Variable

Suppose that independent trials, each having probability p of being a success, are

performed until a success occurs.

Let X denote the number of trails required until the first success.

X is geometric random variable with parameter p.

Probability mass function of a geometric random variable having parameters p:

P (n) = (1 p)n1 p, n = 1, 2, , n.

Example. Suppose that we flip a coin having a probability p of coming up heads

until the first head appears. Let N denote the number of flips required, and

assume that the outcome of successive flips are independent.

N : a geometric random variable.

Therefore, P (N = n) = (1 p)n1 p,

n 1.

A random variable X taking on one of the values 0, 1, 2, , is said to be a Poisson

random variable with parameter , if for some > 0,

i

P (i) = P (X = i) = e , i = 0, 1, .

i!

Example. Pedestrian death (true story): In January 2010 there were 7 pedestrian

deaths in Toronto (14 in GTA). On average there are 2.66 pedestrian deaths per

month in Toronto.

Suppose that the distribution of the number of pedestrian deaths in Toronto is

Poisson with parameter 2.66. What is the probability of having 7 pedestrian

deaths in a month in Toronto?

(2.66)7

= 0.013077 or 1.3%.

P (= 7) = e2.66

7!

Probability of having 7 or more pedestrian deaths in a month in Toronto:

j

P

2.66 (2.66)

P (X 7) =

e

= 0.019.

j=7

j!

Poisson random variable may be used to approximate a binomial random variable

binomial parameter n should be large,

binomial parameter p should be small.

The Poisson random variable that approximates a binomial random variable

with parameters (n, p) has parameter = np.

Example. Number of fires in Toronto per day.

This is caused by a very large n number of buildings.

9

Then, the number of fires per day can be approximated by a Poisson

random variable with parameter = np.

Suppose number of building is n = 100000 and each has a probability

2.5

of having a fire.

100000

Let N denote number of fires.

2.5k

.

P (N = k) = e2.5

k!

Continuous Random Variables:

X is a continuous random variable if there exists a nonnegative function f (x), define

for all real x R (, ) having the property that for any set B of real numbers

P (X B) = B f (x)x.

f (x): probability density function.

R

P (X (, )) = f (x)dx = 1.

Let B = [a, b]. Then, P (a X b) =

Ra

P (X = a) = a f (x) = 0.

Ra

F (a) = P (X (, a]) = f (x)dx

Rb

a

10

f (x)dx.

Lecture 5:

Several Important Continuous Random Variables:

The Uniform Random Variable

A random variable is said to be uniformly distributed over the interval (0, 1) if its

probability density function (pdf) is

(

1, 0 < x < 1

f (x) =

0, otherwise

f (x)dx =

R1

0

(1)dx = 1.

Rb

Rb

P (a X b) = a f (x)dx = a (1)dx = b a.

The probability that X is in any particular subinterval of (0, 1) equal the

length of that subinterval.

In general, X is a uniform random variable on the interval (, ) if its probability

density function (pdf) is

1 , <x<

f (x) =

0,

otherwise

Example. Calculate the cumulative distribution function (cdf) of a random variable uniformly distributed over (, ).

Ra

1 Ra

F (a) = P (X (, a]) = f (x)dx =

(1)dx. Then,

0,

a

a

, <a<

F (a) =

1,

a>

Example. If X is uniformly distributed over (0, 10), calculate the probability that

X < 3, X > 7, 1 < X < 6.

1 R3

3

P (X < 3) =

(1)dx = .

0

10

10

1 R 10

3

P (X > 7) =

(1)dx = .

7

10

10

1 R6

5

P (1 < X < 6) =

(1)dx = .

1

10

10

Exponential Random Variable

11

some > 0, by

(

ex , x 0

f (x) =

0,

x<0

is said to be an exponential random variable with parameter .

Ra

Cumulative distribution function (cdf): F (a) = 0 ex = 1 ea , a 0.

Example. The dollar amount of damage involved in a car accident is an exponential random variable with parameter 1/1000. Of this, the insurance company

only pays that amount exceeding (the deductible amount of) 400. What is the

probability that the insurance company pays at least 400 to a person who has an

accident?

Since the company pays the amount exceeding 400, the damage should be at

least 800 in order to the company pays at least 400. Then,

P (X 800) = 1 P (X < 800) = 1 F (800) = 1 (1 e800/1000 ) =

e800/1000 = 0.4493.

Expectation of a Random Variable:

The Discrete Case

Consider a discrete random variable X with probability mass function P (x).

Then, the expected value of X is defined by

X

E[X] =

xP (x).

x:P (x)>0

can take on.

Example. Find E[X] where X is the outcome when we roll a fair die.

1

1

1

1

1

7

1

E[X] = 1( ) + 2( ) + 3( ) + 4( ) + 5( ) + 6( ) = .

6

6

6

6

6

6

2

Example. Calculate E[X] when X is a Bernoulli random variable with parameter

p.

E[X] = 0(1 p) + 1(p) = p.

Example. Calculate E[X] when X is a binomial random variable with parameters

(n, p).

P

P

P

n!

E[X] = ni=0 iP (i) = ni=0 i ni pi (1 p)ni = ni=0 i

pi (1 p)ni

(n i)!i!

= np.

Example. Calculate E[X] when X is a geometric random variable with parameter

p.

P

P

P

1

i1

i1

E[X] =

p=p

= .

i=1 iP (i) =

i=1 i(1 p)

i=1 i(1 p)

p

12

.

P

P ! P i

= i=1 e

E[X] =

iP

(i)

=

i=0

i=0 ie

i!

(i 1)!

P i1

= i=1 e

= .

(i 1)!

13

Lecture 6:

Discrete Case (continue ...)

Example. Suppose that teams A and B are playing a series of games. Team A

wins each game independently with probability 2/3 and Team B wins each game

independently with probability 1/3. The winner of the series is the first team to

win 2 games. Find the expected number of games that are played.

X: number of games

P (X = 2) = P (X = 2, A wins 2 of the first 2)+P (X = 2, B wins 1 of the first 2)

2 2

1

5

2

+

=

=

3

3

9

P (X = 3) = P (X = 3, A wins 1 of the first 2)+P (X = 3, B wins 1 of the first 2)

2 1 1 1 2

1 1 2 1 1

12

2

2

= 1

+ 1

= .

3

3

3

3

3

3

27

66

E [X] = 2P (X = 2) + 3P (X = 3) = .

27

Continuous Case

Consider a continuous random variable X with probability density function f (x).

Then, the expected value of X is defined by

Z

xf (x)dx.

E[X] =

over (, ).

R

1

2 2

( )( + )

( + )

E[X] = x(

)dx =

=

=

.

2( )

2( )

2

Example. Calculate E[X] when X is an exponential random variable with parameter .

R

E[X] = 0 x(ex )dx.

Integrating by parts (dv = ex , u = x) yields to

Z

Z

1

1

x

x

E[X] =

x(e )dx = xe |0 +

(ex )dx = 0 (ex )|

.

0 =

0

0

Expectation of a Function of a Random Variable:

Suppose we are interested in a function of X, say g(X).

If X is a discrete random variable with probability mass function P (x), then for any

real-valued function g(x),

X

E [g(X)] =

g(x)P (x).

x:P (x)>0

14

Example. Suppose X has the following probability mass function P (0) = 0.2, P (1) =

0.5, P (2) = 0.3. Calculate E[X 2 ].

E[X 2 ] = (0)2 (0.2) + (1)2 (0.5) + (2)2 (0.3) = 1.7.

If X is a continuous random variable with probability density function f (x), then for

any real-valued function g(x),

Z

E [g(X)] =

g(x)f (x)dx.

Example. The dollar amount of damage involved in a car accident is an exponential random variable with expected value of 1000. The insurance company pays

the whole damage if it is more than 400 and 0 otherwise. What is the expected

value that the company pays per accident?

Let define g(X) as

(

0, 0 < X < 400

g(X) =

X, 400 < X < 1000

Then,

Z

E [g(X)] =

400

g(x)f (x)dx =

(0)

Z

1 x/1000

1 x/1000

e

e

dx+

(x)

dx

1000

1000

400

1 1/1000x

e

, u = x) yields to

1000

Z

1/1000x

E[g(X)] = xe

|400 +

(e1/1000x )dx = 400e400/1000 +1000(e400/1000 ).

400

E[aX + b] = aE[X] + b.

Remark. V ar(X) = E [(X E(X))2 ]: the variance of X measures the expected

square of deviation of X from its expected value.

V ar(X) = E [X 2 ] (E[X])2 .

Example. Calculate V ar(X) when X is the outcome of rolling a fair die.

1

1

1

1

1

1

91

E [X 2 ] = 12 ( ) + 22 ( ) + 32 ( + 42 ( ) + 52 ( ) + 62 ( ) = ( )

6

6

6

6

6

6

6

7

E[X] = (It is obtained in Lecture 5.)

2

91

7

35

V ar(X) = E [X 2 ] (E[X])2 =

( )2 = .

6

2

12

15

Lecture 7:

Expectation of a Function of a Random Variable...

Remark. If a and b are constants, then

E[aX + b] = aE[X] + b.

Remark. V ar(X) = E [(X E(X))2 ]: the variance of X measures the expected square

of deviation of X from its expected value.

V ar(X) = E [X 2 ] (E[X])2 .

Example. Calculate V ar(X) when X is the outcome of rolling a fair die.

1

1

1

1

1

1

91

E [X 2 ] = 12 ( ) + 22 ( ) + 32 ( + 42 ( ) + 52 ( ) + 62 ( ) = ( )

6

6

6

6

6

6

6

7

E[X] = (It is obtained in Lecture 5.)

2

91

7

35

V ar(X) = E [X 2 ] (E[X])2 =

( )2 = .

6

2

12

Joint Distribution Functions:

If X and Y are discrete random variables, the joint probability mass function of X

and Y is defined by

P (x, y) = P (X = x, Y = y).

The probability mass function of X can be obtained from P (x, y) by

X

PX (x) =

P (x, y).

y:P (x,y)>0

X

PY (y) =

P (x, y).

x:P (x,y)>0

Example. Suppose X and Y are discrete random variables with the probability

mass function P (x, y),

P (1, 1) = 1/4, P (1, 2) = 1/8, P (1, 3) = 1/16, P (1, 4) = 1/16,

P (2, 1) = 1/16, P (2, 2) = 1/16, P (2, 3) = 1/4, P (2, 4) = 1/8.

What is the probability that Y = 3?

PY (3) = P (Y = 3, X = 1) + P (Y = 3, X = 2) =

1

1

5

+ = .

16 4

16

Remark. For discrete random variables X and Y , and real-valued function g(X, Y )

XX

E[g(X, Y )] =

g(x, y)P (x, y).

y

16

E[a1 X1 + a2 X2 + + an Xn ] = a1 E[X1 ] + a2 E[X2 ] + + an E[Xn ].

Example. Calculate the expected sum obtained when three fair dice are rolled.

Let X denote the sum obtained.

Let Xi denote the value of the ith die.

X = X1 + X2 + X3 . Thus,

7

7

21

7

E[X] = E[X = X1 +X2 +X3 ] = E[X1 ]+E[X2 ]+E[X3 ] = ( )+( )+( ) = .

2

2

2

2

Example. Suppose that there are 20 patients of type A and 15 patients of type

B in a hospital. Each patient of type A is discharged today with probability 2/3

independent of other patients. Also, each patient of type B is discharged today

with probability 1/3 independent of other patients. What is the expected number

of patients who are discharged today?

Let X denote the total number of discharged patients.

(

1, if the ith patient of type A is discharged today

Xi =

0, otherwise

(

1, if the ith patient of type B is discharged today

Yi =

0, otherwise

2

2

1

E[Xi ] = (0)( ) + (1)( ) = .

3

3

3

2

1

1

E[Yi ] = (0)( ) + (1)( ) = .

3

3

3

X = X1 + X2 + + X20 + Y1 + Y2 + + Y15 .

Thus,

E[X] = E[X1 + X2 + + X20 + Y1 + Y2 + + Y15 ]

= E[X1 ] + E[X2 ] + + E[X20 ] + E[Y1 ] + E[Y2 ] + + E[Y15 ]

2

1

55

= (20)( ) + + (15)( ) = .

3

3

3

Independent Random Variables:

The random variable X and Y are independent if for all a and b,

P (X a, Y b) = P (X a)P (Y b).

If X and Y are independent, then for any functions g(X) and h(X)

E[g(X)h(X)] = E[g(X)]E[h(X)].

17

Example. Two fair dice are rolled. What is the expected value of the product of

their outcomes?

Let Z denote the product of their outcomes.

Let X denote the outcome of the first die.

Let Y denote the outcome of the second die.

49

7 7

E[Z] = E[XY ] = E[X]E[Y ] = ( )( ) = .

2 2

4

18

Lecture 8:

Conditional Probability: Discrete Case

Recall that for any two events E and F , the conditional probability of E given F is

defined, as long as P (F ) > 0, by

T

P (E F )

P (E|F ) =

.

P (F )

If X and Y are discrete random variables, the conditional probability mass function of

X given that Y = y is defined by,

P (X = x|Y = y) =

P (x, y)

P (X = x, Y = y)

=

.

P (Y = y)

P (y)

Example. Suppose that P (x, y), the joint probability mass function of X and Y ,

is given by

P (1, 1) = 0.5, P (1, 2) = 0.1, P (2, 1) = 0.1, P (2, 2) = 0.3

Calculate the conditional probability mass function of X given that Y = 1.

P (Y = 1) = P (1, 1) + P (2, 1) = 0.6

P (X = 1|Y = 1) =

P (1, 1)

5

P (X = 1, Y = 1)

=

= .

P (Y = 1)

P (Y = 1)

6

P (X = 2|Y = 1) =

P (X = 2, Y = 1)

P (2, 1)

1

=

= .

P (Y = 1)

P (Y = 1)

6

y such that P (Y = y) > 0, by

X

F (x|y) = P (X x|Y = y) =

P (a|y).

ax

X

X

E[X|Y = y] =

xP (X = x|Y = y) =

xP (x|y).

x

Example. The joint probability mass function of X and Y , P (x, y), is given by

1

1

1

P (1, 1) = , P (2, 1) = , P (3, 1) = ,

9

3

9

1

1

P (1, 2) = , P (2, 2) = 0, P (3, 2) = ,

9

18

1

1

P (1, 3) = 0, P (2, 3) = , P (3, 3) = .

6

9

19

= (1)

P (X = 2, Y = 2)

P (X = 3, Y = 2)

P (X = 1, Y = 2)

+ (2)

+ (3)

P (Y = 2)

P (Y = 2)

P (Y = 2)

1

1

0

5

= (1) 9 + (2) + (3) 18 = .

1

1

1

3

6

6

6

Example. If X1 and X2 are independent binomial random variables with respective parameters (5, 0.4) and (10, 0.4), calculate the conditional probability mass

function of X1 given that X1 + X2 = 8.

Remark. If X1 and X2 are binomial random variables with parameters (n1 , p)

and (n2 , p), respectively, then X1 + X2 is a binomial random variable with

parameters (n1 + n2 , p).

P (X1 = k|X1 +X2 = 8) =

P (X1 = k, X1 + X2 = 8)

=

P (X1 + X2 = 8)

P (X1 + X2 = 8)

5

10

k

5k

8k

108+k

(0.4)

(1

0.4)

(0.4)

(1

0.4)

8k

= k

=

15

8 (1 0.4)158

(0.4)

8

5

k

10

8k

15

8

,

0 k 5.

Example. There are n components. On a rainy day, component i will function with

probability pi . On a nonrainy day, component i will function with probability qi ,

for i = 1, n. It will rain tomorrow with probability . Calculate the conditional

expected number of components that function tomorrow given it rains.

Let define Xi and Y as

(

1, if component i functions tomorrow

Xi =

0, otherwise

(

1, if it rains tomorrow

Y =

0, otherwise

Then,

"

E

n

X

#

Xi |Y = 1 =

i=1

n

X

E [Xi |Y = 1] =

i=1

20

n

X

i=1

pi .

X and Y are jointly continuous if there exists a function f (x, y), defined for all real x

and y, having the property that for all sets A and B of real numbers

Z Z

P (X A, Y B) =

f (x, y)dxdy.

B

The function f (x, y) is called the joint probability density function of X and Y .

The probability density function of Y can be obtained from f (x, y) by

Z

Z Z

fY (y)dy,

f (x, y)dxdy =

P (Y B) = P (X (, ), Y B) =

B

where fY (y) =

f (x, y)dx.

If X and Y have a joint density function f (x, y), then the conditional probability

density function of X, given that Y = y, is defined for all values of y such that fY (y),

by

f (x, y)

f (x|y) =

fY (y)

21

Lecture 10:

Conditional Probability: Continuous Case ...

X and Y are jointly continuous if there exists a function f (x, y), defined for all real x

and y, having the property that for all sets A and B of real numbers

Z Z

P (X A, Y B) =

f (x, y)dxdy.

B

The function f (x, y) is called the joint probability density function of X and Y .

The probability density function of Y can be obtained from f (x, y) by

Z Z

Z

P (Y B) = P (X (, ), Y B) =

f (x, y)dxdy =

fY (y)dy,

B

where fY (y) =

f (x, y)dx.

(

6xy(2 x y), 0 < x < 1, 0 < y < 1

f (x, y) =

0,

otherwise

Calculate fY (y).

fY (y) =

0

Z Z

E[g(X, Y )] =

g(x, y)f (x, y)dxdy.

(

6xy(2 x y), 0 < x < 1, 0 < y < 1

f (x, y) =

0,

otherwise

Calculate E[g(X, Y )] where g(X, Y ) = Y .

E[g(X, Y )] =

Z

y6xy(2xy)dxdy =

Z

=

0

(6x2 y 2 2x3 y 2 3x2 y 3 )|10 dy

3

7

4

6y 2 2y 2 3y 3 dy = ( y 3 y 4 )|10 = .

3

4

12

22

If X and Y have a joint density function f (x, y), then the conditional probability

density function of X, given that Y = y, is defined for all values of y such that fY (y),

by

f (x, y)

.

f (x|y) =

fY (y)

Example. Suppose the joint probability density of X and Y is given by

(

6xy(2 x y), 0 < x < 1, 0 < y < 1

f (x, y) =

0,

otherwise

Calculate the probability density function of X given that Y = y.

f (x|y) =

6xy(2 x y)

f (x, y)

6xy(2 x y)

6x(2 x y)

=

= R1

=

.

fY (y)

y(4 3y)

(4 3y)

6xy(2 x y)dx

0

Z

xf (x|y)dx.

E[X|Y = y] =

(

6xy(2 x y), 0 < x < 1, 0 < y < 1

f (x, y) =

0,

otherwise

Calculate the conditional expectation of X given that Y = y.

E[X|Y = y] =

xf (x|y)dx =

(2 y)2

6x2 (2 x y)

dx =

(4 3y)

4 3y

6

4

5 4y

.

8 6y

Suppose X and Y are two random variables. If Y is a discrete random variable, then

the expected value of X can be obtained by

X

E[X] =

E[X|Y = y]P (Y = y).

y

by

Z

E[X] =

E[X|Y = y]fY (y)dy.

23

Example. Sam will read either one chapter of his probability book or one chapter

of his history book. Suppose the number of misprints in a chapter of his probability book is Poisson distributed with mean 2 and the number of misprints in his

history chapter is Poisson distributed with mean 5. Assume that Sam is equally

likely to choose either book. What is the expected number of misprints that Sam

will come across?

X: the number of misprints.

(

1, if Sam chooses his history book

Y =

2, if Sam chooses his probability book

Then,

1

7

1

E[X] = E[X|Y = 1]P (Y = 1) + E[X|Y = 2]P (Y = 2) = 5( ) + 2( ) = .

2

2

2

Example. A miner is trapped in a mine containing three doors. The first door

leads to a tunnel that takes him to safety after two hours of travel. The second

door leads to a tunnel that returns him to the mine after three hours of travel.

The third door leads to a tunnel that returns him to his mine after five hours.

Assuming that the miner is at all times equally likely to choose any one of the

doors, what is the expected length of time until the miner reaches safety?

X: the time until the miner reaches safety.

Y : the door he initially chooses.

Then,

E[X] = E[X|Y = 1]P (Y = 1)+E[X|Y = 2]P (Y = 2)+E[X|Y = 3]P (Y = 3).

E[X|Y = 1] = 2.

E[X|Y = 2] = 3 + E[X].

E[X|Y = 3] = 5 + E[X].

Therefore,

E[X] = E[X|Y = 1]P (Y = 1)+E[X|Y = 2]P (Y = 2)+E[X|Y = 3]P (Y = 3)

1

1

1

E[X] = (2) + (3 + E[X]) + (5 + E[X]) E[X] = 10.

3

3

3

Let E denote an arbitrary event and define X as

(

1, if E occurs

X=

0, otherwise

Then,

E[X] = (1)P (E) + (0)(1 P (E)) = P (E).

Therefore, P (E) = E[X].

24

Lecture 11:

Computing Probabilities by Conditioning ...

Let E denote an arbitrary event and Y denote a discrete random variable. Then, the

probability of event E can be obtained by

X

P (E) =

P (E|Y = y)P (Y = y).

y

by

Z

P (E) =

P (E|Y = y)fY (y)dy.

rainy day is a Poisson random variable with mean 9, whereas on a dry day it

is a Poisson random variable with mean 3. Let X denote the number of traffic

accidents tomorrow. Suppose it will rain tomorrow with probability 0.6. Calculate

P (X = 0).

Let

(

1, if it rains tomorrow

Y =

0, otherwise

Then,

P (X = 0) = P (X = 0|Y = 1)P (Y = 1) + P (X = 0|Y = 0)P (Y = 0)

0

9 (9)

= (0.6)(e

0!

0

3 (3)

) + (1 0.6)(e

0!

) = (0.6)(e9 ) + (0.4)(e3 ).

Markov Chains

Stochastic Processes: A discrete-time stochastic process {Xn , n = 0, 1, } is a collection of random variables.

For each n = 0, 1, , Xn is a random variable.

The index n is often interpreted as time and, as a result, we refer to Xn as the

state of the process at time n.

For example,

Xn might be the total number of customers that have entered a supermarket

by time n.

Xn might be the number of customers in the supermarket at time n.

A stochastic process is a family of random variables that describes the evolution

through time of some process.

Example (Frog Example). Suppose 1000 lily pads are arranged in a circle. A frog

starts at pad number 1000. Each minute, she jumps either straight up, or one

pad clockwise, or one pad counter-clockwise, each with probability 1/3.

25

1

P (at pad # 1 after 1 step) = .

3

1

P (at pad # 1000 after 1 step) = .

3

1

P (at pad # 999 after 1 step) = .

3

P (at pad # 428 after 987 steps)?

Markov Chain: a discrete-time Markov chain is a discrete-time stochastic process specified by

A state space S: any non-empty finite or countable set.

In frog example, 1000 lily pads.

Transition probabilities {Pij }i,jS : probability of jumping to j if you start at i.

probability that the process will next

is in state i.

P

Pij 0, and j Pij = 1 for all i.

In frog example,

1/3,

1/3,

1/3,

Pij =

1/3,

1/3,

0,

ij =0

ij =1

ji=1

i j = 999

j i = 999

otherwise

P (Xn+1 = j|X0 = i0 , X1 = i1 , , Xn1 = in1 , Xn = i) = P (Xn+1 = j|Xn = i) = Pij .

This is called Markov property.

Example (Gamblers ruin). Consider a gambling game in which on any turn you

win $1 with probability 0.4 or lose $1 with probability 0.6. Suppose further that

you adopt the rule that you quit playing if your fortune reaches $5.

Let Xn be the amount of money you have after n plays.

State space S = {0, 1, 2, 3, 4, 5}.

Xn has the Markov property since given the current state Xn any other information about the past is irrelevant for predicting the next state, Xn+1 .

Transition probabilities,

P (Xn+1 = i+1|X0 = i0 , , Xn = i) = P (Xn+1 = i+1|Xn = i) = 0.4,

0 < i < 5.

0 < i < 5.

P (Xn+1 = 0|X0 = i0 , , Xn = 0) = P (Xn+1 = 0|Xn = 0) = 1.

26

Transition matrix

0

0

0.6 0 0.4 0

0 0.6 0 0.4 0

0

P =

0 0.6 0 0.4 0

0

0

0

0 0.6 0 0.4

0

0

0

0

0

1

Example (Inventory Chain). Consider an (s, S) inventory control policy. That is,

when the stock on hand at the end of the day falls to s or below, we order enough

to bring it back up to S. For simplicity, we assume it happens at the beginning

of the next day.

27

Lecture 12:

Markov Chains ...

Example (Inventory Chain). Consider an (s, S) inventory control policy. That is, when

the stock on hand at the end of the day falls to s or below, we order enough to bring it

back up to S. For simplicity, we assume it happens at the beginning of the next day.

Suppose that s = 1 and S = 5. Also, assume that the distribution of the demand on

day n + 1 is

P (Dn+1 = 0) = 0.3, P (Dn+1 = 1) = 0.4, P (Dn+1 = 2) = 0.2, P (Dn+1 = 3) = 0.1.

Xn : the amount of stock on hand at the end of day n.

State space S = {0, 1, 2, 3, 4, 5}.

Transition probabilities,

P (Xn+1 = 0|Xn = 0): when stock on hand is zero at the end of day n, 5 units

will be ordered and therefore there will be 5 units available at the beginning

of day n + 1. Since the maximum demand on day n + 1 is 3, there will be

at least 1 unit available at the end of the day n + 1. This means that given

Xn = 0, Xn+1 is greater than zero, or

P (Xn+1 = 0|Xn = 0) = P (Dn+1 5) = 0.

P (Xn+1 = 1|Xn = 0) = P (Dn+1 = 4) = 0

(similar to the above discussion).

P (Xn+1 = 2|Xn = 0): when stock on hand is zero at the end of day n, 5 units

will be ordered and therefore there will be 5 units available at the beginning

of day n + 1. If the demand on day n + 1 is exactly 3, there will be 2 units

available at the end of the day n + 1.

P (Xn+1 = 2|Xn = 0) = P (Dn+1 = 3) = 0.1

Similarly,

P (Xn+1 = 3|Xn = 0) = P (Dn+1 = 2) = 0.2.

P (Xn+1 = 4|Xn = 0) = P (Dn+1 = 1) = 0.4.

P (Xn+1 = 5|Xn = 0) = P (Dn+1 = 0) = 0.3.

Similarly, for Xn = 1:

P (Xn+1 = 0|Xn = 1) = P (Dn+1 5) = 0.

P (Xn+1 = 1|Xn = 1) = P (Dn+1 = 4) = 0.

P (Xn+1 = 2|Xn = 1) = P (Dn+1 = 3) = 0.1

P (Xn+1 = 3|Xn = 1) = P (Dn+1 = 2) = 0.2.

P (Xn+1 = 4|Xn = 1) = P (Dn+1 = 1) = 0.4.

P (Xn+1 = 5|Xn = 1) = P (Dn+1 = 0) = 0.3.

28

P (Xn+1 = 0|Xn = 2): when stock on hand is 2 at the end of day n, 0 units

will be ordered and there will be 2 units available at the beginning of day

n + 1. Therefore,

P (Xn+1 = 0|Xn = 2) = P (Dn+1 2) = P (Dn+1 = 2) + P (Dn+1 = 3) = 0.3.

P (Xn+1 = 1|Xn = 2) = P (Dn+1 = 1) = 0.4.

P (Xn+1 = 2|Xn = 2) = P (Dn+1 = 0) = 0.3.

P (Xn+1 = 3|Xn = 2) = 0.

P (Xn+1 = 4|Xn = 2) = 0.

P (Xn+1 = 5|Xn = 2) = 0.

Similarly, for Xn = 3:

P (Xn+1 = 0|Xn = 3) = P (Dn+1 3) = P (Dn+1 = 3) = 0.1.

P (Xn+1 = 1|Xn = 3) = P (Dn+1 = 2) = 0.2.

P (Xn+1 = 2|Xn = 3) = P (Dn+1 = 1) = 0.4.

P (Xn+1 = 3|Xn = 3) = P (Dn+1 = 0) = 0.3.

P (Xn+1 = 4|Xn = 3) = 0.

P (Xn+1 = 5|Xn = 3) = 0.

Similarly, for Xn = 4:

P (Xn+1 = 0|Xn = 4) = P (Dn+1 4) = 0.

P (Xn+1 = 1|Xn = 4) = P (Dn+1 = 3) = 0.1.

P (Xn+1 = 2|Xn = 4) = P (Dn+1 = 2) = 0.2.

P (Xn+1 = 3|Xn = 4) = P (Dn+1 = 1) = 0.4.

P (Xn+1 = 4|Xn = 4) = P (Dn+1 = 0) = 0.3.

P (Xn+1 = 5|Xn = 4) = 0.

Similarly, for Xn = 5:

P (Xn+1 = 0|Xn = 5) = P (Dn+1 5) = 0.

P (Xn+1 = 1|Xn = 5) = P (Dn+1 4) = 0.

P (Xn+1 = 2|Xn = 5) = P (Dn+1 = 3) = 0.1.

P (Xn+1 = 3|Xn = 5) = P (Dn+1 = 2) = 0.2.

P (Xn+1 = 4|Xn = 5) = P (Dn+1 = 1) = 0.4.

P (Xn+1 = 5|Xn = 5) = P (Dn+1 = 0) = 0.3.

29

Transition matrix

0

0

0

P =

0

0.1 0.2 0.4 0.3 0

0

0 0.1 0.2 0.4 0.3

Example (Repair Chain). A machine has three critical parts that are subject to failure,

but can function as long as two of these parts are working. When two are broken, they

are replaced and the machine is back to working order the next day. Assume that

parts 1, 2, and 3 fail with probabilities 0.01, 0.02, and 0.04, but no two parts fail on

the same day. Formulate the system as a Markov chain.

Xn : the parts that are broken.

State space S = {0, 1, 2, 3, 12, 13, 23}.

Transition probabilities:

P (Xn+1 = 0|Xn = 0) = 1 0.01 0.02 0.04 = 0.93.

P (Xn+1 = 1|Xn = 0) = 0.01.

P (Xn+1 = 2|Xn = 0) = 0.02.

P (Xn+1 = 3|Xn = 0) = 0.04.

If we continue, we get the transition Matrix as,

0

1

2

3

12

13

23

0

0.93 0.01 0.02 0.04

0

0

0

1

0.94

0

0

0.02 0.04

0

0

2

0

0.95

0

0.01

0

0.04

0

0

0

0

0.97

0

0.01

0.02

P = 3

12

1

0

0

0

0

0

0

13

1

0

0

0

0

0

0

23

1

0

0

0

0

0

0

P (Xn+1 = j|Xn = i) = Pij gives the probability of going from i to j in one step.

What is the probability of going from i to j in m > 1 steps?

P (Xn+m = j|Xn = i) = Pijm =?

30

Chapman-Kolmogorov Equations:

Pijn+m

m

Pikn Pkj

,

n, m 0, all i, j.

k=0

and then from k to j in m steps.

Theorem. The m step transition probability P (Xn+m = j|Xn = i) is the mth power

of the transition matrix P ,

P (Xn+m = j|Xn = i) = Pijm = (P m )ij .

31

Lecture 13:

Multistep Transition Probabilities ...

Theorem. The m step transition probability P (Xn+m = j|Xn = i) is the mth power

of the transition matrix P ,

P (Xn+m = j|Xn = i) = Pijm = (P m )ij .

Example. Suppose that if it rains today, then it will rain tomorrow with probability 0.7; and if it does not rain today, then it will rain tomorrow with probability

0.4. Calculate the probability that it will rain two days from today given that it

is raining today. Also, calculate the probability that it will rain four days from

today given that it is raining today.

We model the problem as a Markov chain.

State space S = {0, 1} where 0 denotes that it rains and 1 denotes that it

does not rain.

Transition matrix

"

#

0.7 0.3

P =

0.4 0.6

Then,

"

P2 =

#

0.61 0.39

0.52 0.48

2

= 0.61.

The desired probability is P00

To calculate the probability that it will rain four days from today given that

it is raining today, we consider

#

"

0.5749

0.4251

P4 =

0.5668 0.4332

4

= 0.5749.

The desired probability is P00

To obtain the mth power of a matrix, you can use WWW.WOLFRAMALPHA.COM.

For example, copy {{0.7, 0.3}, {0.4, 0.6}}4 in this website to get P 4 .

8

What about P00

?

"

#

0.5714

0.4286

P8 =

0.5714 0.4286

8

The desired probability is P00

= 0.5714.

10

What about P00 ?

"

#

0.5714 0.4286

10

P =

0.5714 0.4286

Rows are identical! It says that the probability that it will rain in 10

days, or 2o days, ... is 0.5714.

32

10

The desired probability is P00

= 0.5714.

Assume that the distribution of the demand on day n + 1 is

P (Dn+1 = 0) = 0.3, P (Dn+1 = 1) = 0.4, P (Dn+1 = 2) = 0.2, P (Dn+1 = 3) = 0.1.

Suppose that today is day 0. What is the probability of having 3 units on hand

at the end of day 20 given that there are 2 units available at the end of today?

From last session we have,

0

0

0

P =

0

0.1 0.2 0.4 0.3 0

0

0 0.1 0.2 0.4 0.3

0

20

. Therefore,

We are looking for P23

P 20 =

0.0909 0.1556 0.231

Rows are identical! It says that the probability that there will be 3 units of

inventory on hand in 20 days, or 25 days, ... is 0.2156.

20

= 0.2156.

The desired probability is P23

Classification of States:

State j is said to be accessible from state i if Pijn > 0 for some n 0.

Example. Consider a Markov chain with the following transition matrix.

P =

1

2

1

2

0.2 0.8

0 1.0

Example. Consider a Markov chain with the following transition matrix.

1

2

3

P = 2 0 0.9 0.1

3 0.4 0 0.6

33

2 and 3 are accessible from 1. 1 is accessible from 2 since with probability 0.1 we

can go from 2 to 3, and with probability 0.4 we can go from 3 to 1. Similarly, 2

is accessible from 3.

Two states i and j that are accessible to each other are said to communicate, and we

write i j.

Example. Consider a Markov chain with the following transition matrix.

P =

1

2

1

2

0.2 0.8

0 1.0

Example. Consider a Markov chain with the following transition matrix.

1

2

3

P = 2 0 0.9 0.1

3 0.4 0 0.6

Two states that communicate are said to be in the same class.

The Markov chain is said to be irreducible if there is only one class, that is, if all states

communicate with each other.

Example. Consider a Markov chain with the following transition matrix.

P =

1

2

1

2

0.2 0.8

0 1.0

The Markov chain has two classes, {1} and {2}. Therefore, it is not irreducible

or it is reducible.

Example. Consider a Markov chain with the following transition matrix.

1

2

3

P = 2 0 0.9 0.1

3 0.4 0 0.6

The Markov chain has one class, {1, 2, 3}. Therefore, the Markov chain is irreducible.

State i is said to be recurrent if starting in state i the process will ever reenter state i

with probability 1. Otherwise, state i is called transient.

34

State i is said to have period d if Piin = 0 whenever n is not divisible by d, and d is the

largest integer with this property.

For instance, starting in i, it may be possible for the process to enter state i only

at times 2, 4, 6, 8, in which case state i has period 2.

A state with period 1 is said to be aperiodic.

35

Lecture 14:

Multistep Transition Probabilities ...

State i is said to be recurrent if starting in state i the process will ever reenter state i

with probability 1. Otherwise, state i is called transient.

Suppose state i is recurrent. Then it is positive recurrent if, starting in i, the expected

time until the process returns to state i is finite.

Remark. Every irreducible Markov chain with a finite state space is positive recurrent.

State i is said to have period d if Piin = 0 whenever n is not divisible by d, and d is the

largest integer with this property.

For instance, starting in i, it may be possible for the process to enter state i only

at times 2, 4, 6, 8, in which case state i has period 2.

A state with period 1 is said to be aperiodic.

Remark. An irreducible Markov chain is aperiodic if there is a state i for which

Pii > 0.

Example. Consider a MC with the following transition matrix.

0

1

2

3

4

5

1

0

0

0.5

0

0.5

0

2

0.3

0

0.4

0

0.3

0

P =

3 0

0.3

0

0.4

0

0.3

4 0

0

0.5

0

0.5

0

5 0.5

0

0

0

0

0.5

Is this MC irreducible?

All states communicate with each other. Therefore, the MC is irreducible.

Long-run Behavior (Limiting Behavior):

Theorem. If a Markov chain is irreducible, positive recurrent, and aperiodic, then

the long-run proportion of time that the process will be in state j, j is

j = lim Pijn , j 0.

n

Remark. j is called stationary probabilities.

36

Example. Suppose that if it rains today, then it will rain tomorrow with probability 0.7; and if it does not rain today, then it will rain tomorrow with probability

0.4. Calculate the probability that it will rain two days from today given that it

is raining today. In long-run what fraction of time it rains.

We model the problem as a Markov chain.

State space S = {0, 1} where 0 denotes that it rains and 1 denotes that it

does not rain.

Transition matrix

"

#

0.7 0.3

P =

0.4 0.6

Then,

"

P 20 =

#

0.5714 0.4286

0.5714 0.4286

Example. Three of every four trucks on the road are followed by a car, while only

one of every five cars is followed by a truck. What fraction of vehicles on the road

are trucks?

Let Xn denote the type of the nth vehicle. Then, S = {C, T } where C and T

denote car and truck, respectively.

"

#

0.25 0.75

P =

0.2 0.8

Then,

"

P 20 =

#

4/19 15/19

4/19 15/19

Suppose that a Markov chain has the following transition matrix

P =

Then, 1 =

1

2

1

2

1a

a

b

1b

a

b

and 2 =

.

a+b

a+b

Example. A rapid transit system has just started operating. In the first month

of operation, it was found that 25% of commuters are using the system while

75% are travelling by automobile. Suppose that each month 10% of transit users

go back to using their cars, while 30% of automobile users switch to the transit

system. What fraction of people will eventually use the transit system?

37

"

P =

#

0.9 0.1

0.3 0.7

0.1

0.4

= 0.75 and 2 =

= 0.25.

0.3 + 0.1

0.3 + 0.1

Example. Market research suggests that in a five year period 8% of people with

cable television will get rid of it, and 26% of those without it will sign up for it.

What is the long run fraction of people with cable TV?

Then, 1 =

P =

Then, Cable =

Cable

No

Cable No

0.92 0.08

0.26 0.74

26

0.26

=

= 0.7647.

0.26 + 0.08

34

38

Lecture 15:

Long-run Behavior (Limiting Behavior) ...

Example. Consider an (s, S) inventory control policy. Assume that the distribution of

the demand on day n is

P (Dn = 0) = 0.3, P (Dn = 1) = 0.4, P (Dn = 2) = 0.2, P (Dn = 3) = 0.1.

Suppose that sales produce a profit of $12 but it costs $2 a day to keep unsold units

in the store overnight. What are the optimal values of s and S that maximize the

long-run net profit?

The objective is to maximize the long-run net profit, i.e.,

E [net profit] = E [sales] E [holding costs] .

Let I denote the inventory level at the beginning of the day. Conditioning on the

inventory level at the beginning of the day, we have

X

E [net profit] =

E [net profit|I = k] P (I = k).

k

Note that P (I) is the long-run probability of having I units at the beginning of

the day.

Since it is impossible to sell 4 units in a day, and it costs us to have unsold

inventory we should never have more than 3 units on hand.

Based on the above discussion the inventory level at the beginning of a day is

either 3, 2, or 1. We consider them separately.

Suppose that the inventory level at the beginning of a day is 3, i.e., I = 3.

Then the sales of the day is

E [sales|I = 3] = E [sales|I = 3, Dn = 0] P (Dn = 0)+E [sales|I = 3, Dn = 1] P (Dn = 1)

+E [sales|I = 3, Dn = 2] P (Dn = 2) + E [sales|I = 3, Dn = 3] P (Dn = 3)

= [0 (12)] P (Dn = 0)+[1 (12)] P (Dn = 1)+[2 (12)] P (Dn = 2)+[3 (12)] P (Dn = 3)

= [0 (12)] (0.3) + [1 (12)] (0.4) + [2 (12)] (0.2) + [3 (12)] (0.1) = 13.2.

The holding costs of the day is

E [costs|I = 3] = E [costs|I = 3, Dn = 0] P (Dn = 0)+E [costs|I = 3, Dn = 1] P (Dn = 1)

+E [costs|I = 3, Dn = 2] P (Dn = 2) + E [costs|I = 3, Dn = 3] P (Dn = 3)

= [3 (2)] P (Dn = 0)+[2 (2)] P (Dn = 1)+[1 (2)] P (Dn = 2)+[0 (2)] P (Dn = 3)

= [3 (2)] (0.3) + [2 (2)] (0.4) + [1 (2)] (0.2) + [0 (2)] (0.1) = 3.8.

39

E [net profit|I = 3] = E [sales|I = 3] E [costs|I = 3] = 13.2 3.8 = 9.4.

Suppose that the inventory level at the beginning of a day is 2, i.e., I = 2.

Then the sales of the day is

E [sales|I = 2] = E [sales|I = 2, Dn = 0] P (Dn = 0)+E [sales|I = 2, Dn = 1] P (Dn = 1)

+E [sales|I = 2, Dn = 2] P (Dn = 2) + E [sales|I = 2, Dn = 3] P (Dn = 3)

= [0 (12)] P (Dn = 0)+[1 (12)] P (Dn = 1)+[2 (12)] P (Dn = 2)+[2 (12)] P (Dn = 3)

= [0 (12)] (0.3) + [1 (12)] (0.4) + [2 (12)] (0.2) + [2 (12)] (0.1) = 12.

The holding costs of the day is

E [costs|I = 2] = E [costs|I = 2, Dn = 0] P (Dn = 0)+E [costs|I = 2, Dn = 1] P (Dn = 1)

+E [costs|I = 2, Dn = 2] P (Dn = 2) + E [costs|I = 2, Dn = 3] P (Dn = 3)

= [2 (2)] P (Dn = 0)+[1 (2)] P (Dn = 1)+[0 (2)] P (Dn = 2)+[0 (2)] P (Dn = 3)

= [2 (2)] (0.3) + [1 (2)] (0.4) + [0 (2)] (0.2) + [0 (2)] (0.1) = 2.0.

The net profit of the day is

E [net profit|I = 2] = E [sale|I = 2] E [costs|I = 2] = 12 2 = 10.

Suppose that the inventory level at the beginning of a day is 1, i.e., I = 1.

Then the sales of the day is

E [sale|I = 1] = E [sale|I = 1, Dn = 0] P (Dn = 0)+E [sale|I = 1, Dn = 1] P (Dn = 1)

+E [sale|I = 1, Dn = 2] P (Dn = 2) + E [sale|I = 1, Dn = 3] P (Dn = 3)

= [0 (12)] P (Dn = 0)+[1 (12)] P (Dn = 1)+[1 (12)] P (Dn = 2)+[1 (12)] P (Dn = 3)

= [0 (12)] (0.3) + [1 (12)] (0.4) + [1 (12)] (0.2) + [1 (12)] (0.1) = 8.4.

The holding costs of the day is

E [costs|I = 1] = E [costs|I = 1, Dn = 0] P (Dn = 0)+E [costs|I = 1, Dn = 1] P (Dn = 1)

+E [costs|I = 1, Dn = 2] P (Dn = 2) + E [costs|I = 1, Dn = 3] P (Dn = 3)

= [1 (2)] P (Dn = 0)+[1 (2)] P (Dn = 1)+[0 (2)] P (Dn = 2)+[0 (2)] P (Dn = 3)

= [1 (2)] (0.3) + [0 (2)] (0.4) + [0 (2)] (0.2) + [0 (2)] (0.1) = 0.6.

The net profit of the day is

E [net profit|I = 1] = E [sale|I = 1] E [costs|I = 1] = 8.4 0.6 = 7.8.

40

P

To obtain E [net profit] = 3k=0 E [net profit|I = k] P (I = k), we need to calculate P (I = k) which depends on the inventory control policy.

Since it is impossible to sell 4 units in a day, and it costs us to have unsold

inventory we should never have more than 3 units on hand, we compare the profit

of (2, 3), (1, 3), (0, 3), (1, 2), and (0, 1) inventory policies.

Consider (2, 3) inventory policy. In this case we always start a day with 3 units,

therefore,

P =

0.1

0.2

0.4

0.3

and

P 20

0.1

0.1

=

0.1

0.1

Therefore, under the (2, 3) inventory control policy, the long-run probabilities

of having 0, 1, 2, and 3 units at the end of the day are 0 = 0.1, 1 = 0.2,

2 = 0.4, and 3 = 0.3, respectively.

Also, under the (2, 3) inventory control policy, the inventory at the beginning

of a day is always 3. Therefore,

E [net profit] =

3

X

E [net profit|I = k] P (I = k) =

k=1

= (7.8) (0) + (12) (0) + (9.4) (1) = 9.4.

Consider (1, 3) inventory policy. Then,

0.1 0.2

0.1 0.2

P =

0.3 0.4

0.1 0.2

and

P 20

19/110

19/110

=

19/110

19/110

0.4 0.3

0.3 0.4

0.3 0

0.4 0.3

41

Therefore, under the (1, 3) inventory control policy, the long-run probabilities of

having 0, 1, 2, and 3 units at the end of the day are 0 = 19/110, 1 = 30/110,

2 = 40/110, and 3 = 21/110, respectively.

Under the (1, 3) inventory control policy, the inventory at the beginning of a day

is either 2 or 3. The long-run probability that the inventory level at the beginning

of a day is 2 is P (I = 2) = 2 = 40/110, and P (I = 3) = 0 + 1 + 3 = 70/110.

Therefore,

3

X

E [net profit] =

E [net profit|I = k] P (I = k) =

k=1

70

40

+ (9.4)

= 9.61818.

= (7.8) (0) + (10)

110

110

Consider (0, 3) inventory policy. Then,

0

0.7 0.3 0

P =

0.3 0.4 0.3 0

and

P 20

343/1070

343/1070

=

343/1070

343/1070

Therefore, under the (0, 3) inventory control policy, the long-run probabilities of

having 0, 1, 2, and 3 units at the end of the day are 0 = 343/1070, 1 = 300/1070,

2 = 280/1070, and 3 = 147/1070, respectively.

Under the (0, 3) inventory control policy, the inventory at the beginning of a day

is either 1, 2 or 3. Therefore, P (I = 1) = 1 = 300/1070, P (I = 2) = 2 =

280/1070, and P (I = 3) = 0 + 3 = 490/1070. Therefore,

E [net profit] =

3

X

E [net profit|I = k] P (I = k) =

k=1

300

280

490

= (7.8)

+ (10)

+ (9.4)

= 9.108.

1070

1070

1070

42

0.3 0.4 0.3

Therefore, under the (1, 2) inventory control policy, the long-run probabilities of

having 0, 1, and 2 units at the end of the day are 0 = 0.3, 1 = 0.4, and 2 = 0.3,

respectively. Then,

E [net profit] =

2

X

E [net profit|I = k] P (I = k) =

k=1

Consider (0, 2) inventory policy. Then,

P = 0.7 0.3 0

0.3 0.4 0.3

Therefore, under the (0, 2) inventory control policy, the long-run probabilities of

having 0, 1, and 2 units at the end of the day are 0 = 49/110, 1 = 40/110, and

2 = 21/110, respectively. Then,

E [net profit] =

2

X

E [net profit|I = k] P (I = k) =

k=1

= (7.8)

40

110

+ (10)

70

110

= 9.2.

Remark. Let P be the transition matrix of a Markov chain. Then, the stationary

distribution of the Markov chain is the solutions of P = where is the vector the

stationary distribution.

43

- UWO 1228 Final Exam 2010Diunggah olehCarol Shereen
- veerarajanDiunggah olehManish Vohra
- Introduction to Econometrics with RDiunggah olehHervé Dakpo
- Probability DistributionsDiunggah olehcooooool1927
- Bunge.1981.Four concepts of probability.pdfDiunggah olehthiago
- Review of ProbabilityDiunggah olehSyed Ali
- E_C-_8_1_.3.10Diunggah olehGovind Mantry
- lecture1-converted.pptxDiunggah olehTaqsim Rajon
- Statistics Probability Midterm Cheat SheetDiunggah olehJeff Farmer
- 2019 03 Exam p SyllabiDiunggah olehPittnanPanichakan
- JQT - Planned Replacement - Some Theory and Its ApplicationDiunggah olehbriandpc
- Statistic TablesDiunggah olehMaria Firdausiah
- 100 problems in stochastic processesDiunggah olehscrat4acorn
- l 05 Stats i Discrete Prob Dist sDiunggah olehyeshimebet
- SSM-UNIT 5 6.docDiunggah olehachutha795830
- Probability Concepts, Models and AxiomsDiunggah olehSam Okpeh
- Probability - Free GuideDiunggah olehnieotyagi
- class03_7Diunggah olehDr. Ir. R. Didin Kusdian, MT.
- Lecture 6Diunggah olehgobigupta
- 05 Random SignalDiunggah olehlokesh_harami_kurm
- Chapter4_Lecturer.pdfDiunggah olehham.karim
- Stigler's stat notes part 1Diunggah olehplxnospam
- Discrete Random Variables LectureDiunggah olehHarumoto
- Small Mathematical ExpectationDiunggah olehARUPARNA MAITY
- IPMA ICB 3 w edukacji - Czechy.pdfDiunggah olehTomasz Wiatr
- Methodology StatisticsDiunggah olehPatricia Amillos
- simRDiunggah olehsipho23
- Concur 2013 KatoenDiunggah olehHugo Paquet
- ma3215_exam2_soln.pdfDiunggah olehPei Jing
- Lecture_12_ch8_222_w05_s2.pdfDiunggah olehPaulojoy Buenaobra

- EMTSession5DecisionMaking.pptDiunggah olehnoreen sultan
- Solved ProblemsDiunggah olehOmar Ahmed Elkhalil
- VIT CSE BTech Course PlanDiunggah olehApoorv Krishak
- Lesson 1 - Money, Money, Money With Student Responses and Added ReflectionDiunggah olehKeri Kerby
- UntitledDiunggah olehapi-127299018
- IntroductoryStatistics.pdfDiunggah olehM M Pasha
- Actuary PS3Diunggah olehappleduck
- Periodic Review Inventory Model WithDiunggah olehDr-Mohammed Farid
- Assignment 1 GV9037 C Henrique G FreitasDiunggah olehCarlos
- CH2 Prob SuppDiunggah olehchangazi354
- A Definition of Causal EffectDiunggah olehSamuel Andrés Arias
- Signal Detection Theory and PsychophysicsDiunggah olehSnikeron
- Edexcel S1 Notes.pdfDiunggah olehÅzmâñ Khäñ
- cba4Diunggah olehmihai37
- New method for Dynamic Treatment RegimesDiunggah olehchengmasta
- Brownian Motion in the Stock MarketDiunggah olehmiguelangelnicolas
- Effects of Behavioural Finance on Emerging Capital MarketsDiunggah olehChandrika Das
- Utkal Bca SyllabusDiunggah olehakbisoi1
- Probability and ComputingDiunggah olehNanjunda Swamy
- rubricDiunggah olehapi-302940298
- PobDiunggah oleh0169514311
- Paper RASD2010 005 Halfpenny KihmDiunggah olehjeanpremier
- encsyllDiunggah olehmanjunath
- Zivot - Introduction to Computational Finance and Financial EconometricsDiunggah olehjrodasch
- elementary probability theory.pdfDiunggah olehDaniel Maruri
- prm_exam_2Diunggah olehHasan Tariqul Islam
- Challenging problems in MathematicsDiunggah olehKathiresan Manickam
- Decision Making Under ConflictDiunggah olehJamesIgro
- Decision TreesDiunggah olehMarketing Expert
- Chapter OneDiunggah olehrawatrajeev15