INDR 343

Stochastic Models
Department of Industrial Engineering
Koç University

Chapter 29
Markov Chains

Süleyman Özekici
ENG 119, Ext: 1723
sozekici@ku.edu.tr

Course Topics

• Markov Chains (Chapter 29)

• Queueing Models (Chapter 17)
• Inventory Models (Chapter 18)
• Markov Decision Models (Chapter 19)

S. Özekici INDR 343 Stochastic Models 2

1
 Markov Chains and Processes
• Stochastic processes
• Introduction to Markov chains
• Transient analysis
• Classsification of states
• Ergodic and potential analysis of MC
• Introduction to Markov processes
• Ergodic and potential analysis of MP

S. Özekici INDR 343 Stochastic Models 3

Stochastic Processes
• Many stochastic models in operations research are represented using a collection of
random variables that are indexed by time, so that
Xt = state of the system at time t
• Some examples are
– Xt = the number of shoppers who arrived at a supermarket until time t
– Xt = the amount of inventory in stock at the end of week t
– Xt = the number of patients in the emergency room of an hospital at
time t
– Xt = the price of a share of common stock that is traded at Istanbul
Stock Exchange at the close of day t
– Xt = the functional state of a workstation at time t
– Xt = the number of vehicles that arrive at the Boğaziçi bridge during day t
– Xt = the time of arrival of the tth customer to a bank during a given day
– Xt = the number of wins of a soccer team in t matches played

S. Özekici INDR 343 Stochastic Models 4

2
 Stochastic Analysis
• A stochastic process X is a collection of random variables

X  {X t ; t  T }
• If T = {0, 1, 2, 3, ...}, then X is a dicrete-time process
• If T = [0, +∞), then X is a continuous-time process
• If Xt is a discrete random variable, taking values in a set like {0, 1, 2, ... },
then X is a discrete-state process
• If Xt is a continuous random variable, taking values in a set like [0, +∞), then
X is a continuous-state process
• We need to determine the probability law of X, and analyze it to detemine
P{Xt = i} (Transient analysis)
limt+∞ P{Xt = i} (Ergodic analysis)
 + 
E  t f ( X t )  (Potential analysis)
 t 0 
S. Özekici INDR 343 Stochastic Models 5

Markov Chain: Definition

• The discrete time and state stochastic process X = {Xt; t = 0, 1, 2, ...} is said to be a
Markov chain if it satisfies the following so-called Markov property
P{Xt+1 =j | X0 = k0, ..., Xt-1 = kt-1, Xt = i} = P{Xt+1 = j | Xt = i}

for i,j in {0, 1, 2, ..., M}

• We will suppose in this course that these conditional probabilities do not depend on
time t, so that the transition probabilities are given by the following transition matrix
P{Xt+1 = j | Xt = i} = Pij
• In general, the n-step transition matrix is denoted by
P{Xt+n = j | Xt = i} = Pij(n)
• We know that (n)
Pij 0


M
j 0
Pij( n )  1

S. Özekici INDR 343 Stochastic Models 6

3
 An Inventory Example
• Suppose that the weekly demand for cameras in a store D1, D2, ... are
independent and identically distributed random variables that have a Poisson
distribution with mean 1. This means
P{Dt = n}= e-11n/n!
• The number of cameras is observed at the close of the working day every
Saturday and if there are no cameras left 3 new cameras are ordered. The
order is received at the beginning of the week on Monday morning
(immediate delivery). If there are 1, 2 or 3 cameras in the store, no new order
is placed.
• This ordering policy is known as the (s, S) policy where s = 0 and S = 3 in
this example (i.e., order up to S units whenever you have s or less units left in
stock).
• If there is no stock left when a customer arrives, then the sale is lost.
• Let Xt be the number of cameras left in the store at the end of week t

S. Özekici INDR 343 Stochastic Models 7

The Transition Matrix

max{3-Dt +1 , 0} if X t  0
X t +1  
max{X t -Dt +1 , 0} if X t  1

0 0.080 0.184 0.368 0.368

1 0.632 0.368 0 0 
P
2 0.264 0.368 0.368 0 
 
3 0.080 0.184 0.368 0.368
(1) 2 e 1
P01  P{Dt +1  2}   0.184
2!
(1) 0 e 1
P10  P{Dt +1  1}  1  P{Dt +1  0}  1   0.632
0!

S. Özekici INDR 343 Stochastic Models 8

4
 Transition Diagram

S. Özekici INDR 343 Stochastic Models 9

Stock Example
• Yt = closing price of a share of common stock that is traded in an exchange at the end
of day t
• Define the process X such that

0 if Yt  Yt 1
Xt  
1 if Yt  Yt 1
• This means that Xt is either 0 or 1 depending on whether the price goes up or
down during the day
• If the future value of X depends only on the last observed value only given
all the past values, then X is a Markov chain with some transition matrix

0 0.7 0.3
P 
1 0.5 0.5

S. Özekici INDR 343 Stochastic Models 10

5
 Another Stock Example
• Suppose that the value of X depends on the previous 2 values of X only, then
Zt = (Xt, Xt-1) is a Markov chain with states

0 = (0, 0): the stock increased both today and yesterday

1 = (1, 0): the stock increased today and decreased yesterday
2 = (0, 1): the stock decreased today and increased yesterday
3 = (1, 1): the stock decreased both today and yesterday

• The transition matrix is

0 0.9 0 0.1 0 
1 0.6 0 0.4 0 
P
2  0 0.5 0 0.5
 
3  0 0.3 0 0.7 

S. Özekici INDR 343 Stochastic Models 11

Gambling Example
• Suppose that a player has $1 and with each play of a game wins $1 with
probability p or loses $1 with probability (1 - p). The game ends when the
fortune of the player becomes $3 or when he goes broke.
• Let Xt be the amount of money that the player has at the end of the tth game,
then X is a Markov chain with states {0, 1, 2, 3} and transition matrix

0 1 0 0 0
1 1  p 0 p 0 
P
2 0 1 p 0 p
 
3 0 0 0 1

S. Özekici INDR 343 Stochastic Models 12

6
 Transient Analysis
• Chapman-Kolmogorov Equations
M
Pij( n )   Pik( m ) Pkj( n  m )
k 0

P ( n)  P ( m) P ( nm)
• This implies

P (1)  P
P ( 2 )  P (1)  P (1)  P  P  P 2
P (3)  P (1)  P ( 2 )  P  P 2  P 3

P ( n )  P (1)  P ( n 1)  P  P n 1  P n

S. Özekici INDR 343 Stochastic Models 13

Inventory Example
• For n = 2 and 4
0.080 0.184 0.368 0.368 0.080 0.184 0.368 0.368
0.632 0.368 0 0  0.632 0.368 0 0 
P2   
0.264 0.368 0.368 0  0.264 0.368 0.368 0 
   
0.080 0.184 0.368 0.368 0.080 0.184 0.368 0.368
0.249 0.286 0.300 0.165
0.283 0.252 0.233 0.233

 0.351 0.319 0.233 0.097
 
0.249 0.286 0.300 0.165

0.249 0.286 0.300 0.165 0.249 0.286 0.300 0.165

0.283 0.252 0.233 0.233 0.283 0.252 0.233 0.233
P4  P2P2   
 0.351 0.319 0.233 0.097   0.351 0.319 0.233 0.097 
   
0.249 0.286 0.300 0.165 0.249 0.286 0.300 0.165
0.289 0.286 0.261 0.164
0.282 0.285 0.268 0.166

0.284 0.283 0.263 0.171
 
0.289 0.286 0.261 0.164

S. Özekici INDR 343 Stochastic Models 14

7
 Inventory Example (MATLAB)

S. Özekici INDR 343 Stochastic Models 15

Unconditional Probabilities
• Given the initial distribution P{X0 = i}, we can compute

P{ X n  j}  i 0 P{ X 0  i}Pijn
M

• In the inventory example, suppose that P{X0 = 0}= 0.10, P{X0 = 1} = 0.25,
P{X0 = 2} = 0.30 and P{X0 = 3} = 0.35, then

P{ X 2  3}  P{ X 0  0}P032 + P{ X 0  1}P132 + P{ X 0  2}P232 + P{ X 0  3}P332

 (0.10)0.165 + (0.25)0.233 + (0.30)0.097 + (0.35)0.165
 0.161

S. Özekici INDR 343 Stochastic Models 16

8
 Classification of States
• State j is accessible from state i if Pijn > 0 for some n
• States i and j communicate if j is accessible from i and i is accessible from j
• A class consists of all states that communicate with each other
• The Markov chain is irreducible if it consists of a single class, or if all states
communicate
• State i is transient if, upon entering this state, the process may never return to it
• State i is recurrent if, upon entering this state, the process definitely will return to it
• State i is absorbing if, upon entering this state, the process will never leave it
• State i is periodic with period t >1, if Piin = 0 for all values of n other than t, 2t, 3t, 4t,
...; otherwise, it is aperiodic
• State i is ergodic if it is recurrent and aperiodic
• A Markov chain is ergodic if all of its states are ergodic

S. Özekici INDR 343 Stochastic Models 17

Examples
• In the inventory and stock examples, the Markov chain is ergodic
• In the gambling example, the Markov chain is not ergodic. States 0 and 3 are
both absorbing, and states 1 and 2 are transient.
• In the following example,

0 1 / 4 3 / 4 0 0 0
1 1 / 2 1 / 2 0 0 0
P  2 0 0 1 0 0
 
3 0 0 1/ 3 2 / 3 0
4  1 0 0 0 0

– State 2 is absorbing
– States 0 and 1 form a class of recurrent and aperiodic states
– States 3 and 4 are transient

S. Özekici INDR 343 Stochastic Models 18

9
 Ergodic Analysis
• If the Markov chain is ergodic, then the limiting distribution
limt+∞ P{Xt = j|X0 = i} = limt+∞ Pijt = πj
exists, and it is the unique solution of the following system of linear
equations
M
 j    i Pij for j  0,1,2,, M (  P)
i 0

  j 1 ( 1  1)
M
j 0

• The πj are called steady-state or stationary probabilities since if P{X0 = j} =

πj , then
P{Xn = j} = πj

S. Özekici INDR 343 Stochastic Models 19

Inventory Example
• In the inventory example, the Markov chain is ergodic and the limiting
distribution satisfies
0  0.080 0 + 0.632 1 + 0.264 2 + 0.080 3
1  0.184 0 + 0.368 1 + 0.368 2 + 0.184 3
2  0.368 0 + 0.368 2 + 0.368 3
3  0.368 0 + 0.368 3
1  0 + 1 + 2 + 3

• The solution is

π0 = 0.286, π1 = 0.285, π2 = 0.263, π3 = 0.166

S. Özekici INDR 343 Stochastic Models 20

10
 Inventory Example (MATLAB)

S. Özekici INDR 343 Stochastic Models 21

Average Cost
• Suppose that the Markov chain incurs the cost C(i) everytime state i is visited, then
the average cost per time is

1 n  1 n
lim E  t 1 C ( X t )  lim t 1 C ( X t ) C   j 0  j C ( j )
M

n +
n  n+ n

• Suppose that there is a storage cost charged at the end of each week for items held in
stock so that C(0) = 0, C(1) = 2, C(2) = 8 and C(3) = 18, then the average storage
cost per week is

1 n
lim
n +
 C ( X t ) 0.286(0) + 0.285(2) + 0.263(8) + 0.166(18)  5.662
n t 1

S. Özekici INDR 343 Stochastic Models 22

11
 Complex Cost Function
• Suppose that cost depends on the present state and random occurences in the next
time period (given by C(Xt-1, Dt)), then

1 n  1 n
lim E  t 1 C ( X t 1 , Dt )  lim t 1 C ( X t 1 , Dt ) k   j 0  j k ( j )
M

n  +
n  n+ n
where
k ( j )  E[C ( j , Dt )]

• In the inventory example, if z cameras are ordered then the cost incurred is 10 + 25z
where the fixed cost of ordering is $10 and the purchase cost is $25 for each camera.
For each unsatisfied demand due to shortage there is a penalty of $50. The cost in
week t is

10 + 25(3) + 50max{ Dt  3,0} if X t 1  0

C ( X t 1 , Dt )  
50max{ Dt  X t 1 ,0} if X t 1  1

S. Özekici INDR 343 Stochastic Models 23

• Numerical calculations give

k (0)  E[C (0, Dt )]  85 + 50 E[max{Dt  3,0}]
 85 + 50[ PD (4) + 2 PD (5) + 3PD (6) + ]
 85 + 50[0.015 + 2(0.003) + 3(0.001)]
 86.2
since
PD (n)  e 1 / n!

One can similarly compute

k (1)  18.4, k (2)  5.2, k (3)  1.2

• The average cost per week is

  j k ( j )  86.2(0.286) + 18.4(0.285) + 5.2(0.263) + 1.2(0.166)  $31.46

M
j 0

S. Özekici INDR 343 Stochastic Models 24

12
 Potential Analysis
• For any Markov chain, if there is a periodic discount factor 0 ≤ α < 1, the expected
total discounted cost function is


g (i)  E t 0  t C ( X t ) | X 0  i
+

• It is the unique solution of the system of linear equations

g (i )  C (i ) +   j 0 Pij g ( j )
M

or
g  C +  Pg
or
g  ( I   P) 1 C

S. Özekici INDR 343 Stochastic Models 25

Inventory Example
• In the inventory example, if the weekly discount factor is α = 0.90, then the sytem of
linear equations become

g (0)  86.2 + 0.90[0.080 g (0) + 0.184 g (1) + 0.368 g (2) + 0.368 g (3)]
g (1)  18.4 + 0.90[0.632 g (0) + 0.368 g (1)]
g (2)  5.2 + 0.90[0.264 g (0) + 0.368 g (1) + 0.368 g (2)]
g (3)  1.2 + 0.90[0.080 g (0) + 0.184 g (1) + 0.368 g (2) + 0.368 g (3)]

• The solution is

g(0) = 91.913 , g(1) = 105.68 , g(2) = 92.764 , g(3) = 6.913

S. Özekici INDR 343 Stochastic Models 26

13
 First Passage Time
• Let Tj be the time of first passage to state j and denote its distribution by
fij (n) = P{Tj = n| X0 = i}
and the mean of the first passage time by
μij = E[Tj | X0 = i] = ∑nn fij (n)
• The distribution can be determined recursively as
f ij(1)  Pij
f ij( n )  k  j Pik f kj( n 1)
• The mean can be computed by solving the system of linear equations

 ij  1 + k  j Pik  kj
• It also follows that the mean recurrence time is
1
 ii 
i
S. Özekici INDR 343 Stochastic Models 27

Inventory Example
• The probability distribution of the first passage time to state j = 0 from state
i = 0, 1, 2, 3 is can be obtained as follows
f30(1)  P30  0.080, f 20(1)  P20  0.264, f10(1)  P10  0.632, f 00(1)  P00  0.080
0.080 
0.632 
fi 0(1)  Pi 0   
0.264 
 
0.080 
f30(2)  P31 f10(1) + P32 f 20(1) + P33 f 30(1)  0.184(0.632) + 0.368(0.264) + 0.368(0.080)  0.243

0 0.184 0.368 0.368 0.080   0.243

0 0.368 0 0  0.632   0.233
fi 0(2)  
0 0.368 0.368 0  0.264  0.330 
    
0 0.184 0.368 0.368 0.080   0.243

S. Özekici INDR 343 Stochastic Models 28

14
 Inventory Example (MATLAB)

S. Özekici INDR 343 Stochastic Models 29

Inventory Example
• The means can be computed by solving

 00  1 0 0.184 0.368 0.368  00 

   1 0 0.368 0 0   10 
 10     + 
 20  1 0 0.368 0.368 0   20 
     
 30  1 0 0.184 0.368 0.368  30 

00  1 + 0.18410 + 0.36820 + 0.36830

10  1 + 0.36810
20  1 + 0.36810 + 0.36820
30  1 + 0.18410 + 0.36820 + 0.36830

00  3.50 weeks, 10  1.58 weeks, 20  2.51 weeks, 30  3.50 weeks

S. Özekici INDR 343 Stochastic Models 30

15
 Inventory Example (MATLAB)

S. Özekici INDR 343 Stochastic Models 31

Absorbing States
• If k is an absorbing state among possibly several others, then one is
interested in the probability that the process will eventually be absorbed in
state k given that the initial state is i
• Denoting this absorption probability by fik, the Markov property gives the
following system of linear equations

f ik   j 0 Pij f jk
M

subject to the conditions

f kk  1 and f ik  0 if state i is recurrent and i  k

S. Özekici INDR 343 Stochastic Models 32

16
 Gambling Example
• In the gambling example, suppose that p = 0.4, then the probability that the gambler
will eventaully reach the $3 target without going broke is f13 given that the initial
fortune is $1
• The system of linear equations is
f 03  (1) f 03
f13  0.6 f 03 + 0.4 f 23
f 23  0.6 f13 + 0.4 f 33
f 33  (1) f 33
• The additional conditions are f03 = 0 and f33 = 1, so the equations become

f13  0.4 f 23
f 23  0.6 f13 + 0.4
• The solution is
f13 = 0.21 , f23 = 0.53

S. Özekici INDR 343 Stochastic Models 33

Gambling Example (MATLAB)

S. Özekici INDR 343 Stochastic Models 34

17
 Markov Process: Definition
• The continuous time and discrete state stochastic process X = {X(t); t ≥ 0} is
said to be a Markov process (or continuous time Markov chain) if it satisfies
the following so-called Markov property
P{X(s+t) = j | X(u); u ≤ s, X(s)= i} = P{X(s+t) = j | X(s) = i}
for i,j in {0, 1, 2, ..., M}
• We will suppose in this course that these conditional probabilities do not
depend on time s, so that the transition probabilities are given by the
following continuous time transition probability function
P{X(s+t) = j | X(s) = i} = Pij(t)
• We know that
Pij (t )  0


M
j 0
Pij (t )  1

S. Özekici INDR 343 Stochastic Models 35

Structure of a Markov Process

• Let Sn be the time of the nth jump and Yn be the nth state visited by the
Markov process X
• The relationship between the processes X and (Y, S) is described as
X t  Yn whenever S n  t  S n+1
• The stochastic process Y = {Y0, Y1, Y2, ...} is a Markov chain with some
transition matrix P with Pii = 0
• The amount of time spent in the nth state has the exponential distribution
with rate qi if the state is i, in other words

P{S n+1  S n  t | Yn  i}  P{Ti  t}  1  e  qit

• Here, Ti is a generic random variable that represents the amount of time

spent in state i
• The Markov process jumps out of a state i exponentially with rate qi and
goes to some other state j with probability Pij
S. Özekici INDR 343 Stochastic Models 36

18
 Transient Analysis
• The probability law of X is described by the probability laws of Y and S
• Putting the transition matrix P and the jump rate vector q together, we obtain
the so-called transition rate matrix

qi if j  i
Qij  
qij  qi Pij if j  i

• It is quite difficult to find the transition function P(t) explicitly, but we can
show that it satisfies the following system of differential equations
dP(t)/dt = QP(t) = P(t)Q
where Q is the transition rate matrix (Kolmogorov’s equations)
• The solution is matrix exponential given as
+ n
tn n  t 
P(t )  eQt   Q  limn+  I + Q 
n 0 n !  n 
S. Özekici INDR 343 Stochastic Models 37

Ergodic Analysis
• A pair of states i and j are said to communicate with each other if there are
times t1 and t2 such that Pij(t1) > 0 and Pji(t2) > 0
• If the Markov process is ergodic (i.e., all states communicate with each other
so that the Markov process is irreducible), then the limiting distribution
limt+∞ P{X(t) = j|X(0) = i} = limt+∞ Pij(t) = πj
exists, and it is the unique solution of the following system of linear
equations (also known as balance equations)
 j q j    i qij for j  0,1,2,, M (0  Q)
i j

  j 1 ( 1  1)
M
j 0

• The πj are called steady-state or stationary probabilities since if P{X(0) = j}

= πj , then
P{X(t) = j} = πj

S. Özekici INDR 343 Stochastic Models 38

19
 Reliability Example
• A shop has 2 identical machines that are operated continuously except when
they are broken down. There is a full-time maintenance person who repairs
a broken machine. The time to repair a machine is exponentially distributed
with a mean of 0.5 day. The amount of time a repaired machine works until
next failure is also exponentially distributed with a mean of 1 day.
• Let Xt denote the number of machines that are not functioning at time t, then
X is a Markov process with the following transition rate matrix and diagram

0  2 2 0
qij  1  2  3 1 
2  0 2  2

S. Özekici INDR 343 Stochastic Models 39

Ergodic Analysis
• In the reliability example, the Markov process is ergodic and the limiting
distribution satisfies

2 0  2 1
3 1  2 0 + 2 2
2 2   1
 0 + 1 +  2  1

• The solution is

π0 = 2/5 = 0.4, π1 = 2/5 = 0.4, π2 = 1/5 = 0.2

S. Özekici INDR 343 Stochastic Models 40

20
 Reliability Example (MATLAB)

S. Özekici INDR 343 Stochastic Models 41

Average Cost
• Suppose that the Markov process incurs the cost C(i) per unit time while it is in state
i, then the average cost per time is

1 t  1 t
lim E   C ( X s )ds   lim  C ( X s )ds  C   j 0  j C ( j )
M

t +
t 0  t + t 0

• In the reliability example, suppose that there is a cost associated with the downtime
of each machine so that C(0) = 0, C(1) = 100 and C(2) = 200 per day, then the
average cost per day is

1 t 
lim E   C ( X s )ds   0.4(0) + 0.4(100) + 0.2(200)  80
t +
t 0 

S. Özekici INDR 343 Stochastic Models 42

21
 Potential Analysis
• For any Markov process, if there is a continuous discount factor α > 0, the
expected total discounted cost function is

g (i)  E  e s C ( X s )ds | X 0  i 

+

 0 

• It is the unique solution of the system of linear equations

 g (i )  C (i ) +  j 0 qij g ( j )
M

or
 g  C + Qg
or
g  ( I  Q) 1 C

S. Özekici INDR 343 Stochastic Models 43

Reliability Example
• In the reliability example, if the discount factor is α = 0.95, then the sytem of linear
equations become

0.95 g (0)  0  2 g (0) + 2 g (1)

0.95 g (1)  100 + 2 g (0)  3g (1) + g (2)
0.95 g (2)  200 + + 2 g (1)  2 g (2)

• The solution is

g(0) = 59.371 , g(1) = 87.572 , g(2) = 127.17

S. Özekici INDR 343 Stochastic Models 44

22
 Homework 1 & 2
• Homework 1 • Homework 2
– 29.2-2 – 29.5-4
– 29.2-3 – 29.5-9
– 29.3-2 – 29.6-5
– 29.4-2
– 29.8-1
– 29.4-5
– 29.8-2
• Review Exercises 1
– 29.4-3
– 29.5-5
– 29.6-1
– 29.7-1
– 29.7-2

S. Özekici INDR 343 Stochastic Models 45

23