3 Discrete Markov Chain (Long Run)

Long-Run Behavior of
Markov Chain
(Bab 4 Taylor dan Karlin)
NUR Iriawan
Statistika FMIPA-ITS
Long-Run Behavior (Steady State) of Markov Chains 1

Limiting Probability

Regular Markov Chain

Example

Def: Regular Matrix

Proof (1)

Proof (2)

Example 1

By Applying Theorem 1.1

Solution

Contoh 2: Spiders and Fly
2-14
14
2-15
15
2-16 n-step transition probabilities into state 1
16
2-17 Sequence of Transition Prob Matrix
17
2-18 Steady state condition
18
Doubly Stochastic

Example: Doubly Stochastic

Limit Theorems
Theorem 1: Irreducible aperiodic Markov chain
For every state j, the following limit
j lim P{ X n j | X 0 i}, i 0,1,2,...
n
exists and is independent of initial state i

Nj(k): number of visits to state j up to time k
N j (k )
P j lim X0 i 1
k k
j: frequency the process visits state j

Limiting Distribution

Not All Markov Chains are Regular
Some examples:
Pn = P
Pn = P, if n is odd
Pn = identity matrix,
Long-Run Behavior (Steady State) of Markov Chains if n is even 25
1 1
2 2
On the long run, n

Stationary
Definition: A stochastic process is said to be

stationary if
Pr{X1 = x1, X2 = x2, , Xn= xn}
= Pr{X1+i = x1, X2+i = x2, , Xn+i= xn}
for every shift i.
E.g. Pr{Xn = 1} = Pr{Xn+1 = 1}

= Pr{Xn+2 = 1}
= 1
Classification of States
It is important to understand the nature of
states in a Markov Chain in order to tell
how the process will evolve
Here we will define the terminology we are
going to use in classifying the states

Accessible States
State j is said to be accessible from state i if pij(n) >
0 for some n 0
This means starting from i, it is possible that the
process enters state j at some point in time
In the inventory example pij(2) > 0 for all i,j. Thus,
every state is accessible from every other state
In the random walk example interior states {1,,
M-1} are not accessible from the boundary states
{0,M}. But not vice versa

Communication
If state i is accessible from state j and state j is accessible from state
i, then states i and j are said to communicate [ i j]
Properties of Communicating States:
Every state communicates with itself because pii(0) = P{X0 = i | X0 = i} = 1
[ i i]
If state i communicates with state j then state j communicates with state
i [(i j) (j i)]
If state i communicates with state j and state j communicates with state
k then state i communicates with state k
[(i j) and (j k) (i k)]
Therefore, communication divides the state space up into mutually
exclusive classes.
If all the states communicate, the Markov chain is irreducible.

Equivalence Classes
The states in an equivalence class are
those that communicate with each other
It may be possible, starting in one class, to
enter some other class with positive
probability; but after that, it is impossible to
return back to the previous class
Examples

Recurrence vs. Transience
Let fi be the probability that, starting in state i, the process
will ever reenter state i. If fi = 1, the state is recurrent,
otherwise it is transient.
If state i is recurrent then, starting from state i, the
process will reenter state i infinitely often (w/prob. 1).
If state i is transient then, starting in state i, the
number of periods in which the process is in state i
has a geometric distribution with parameter 1 fi.
Or, state i is recurrent if n1 Pii and transient if n1 ii
n
n P
Recurrence (transience) is a class property: If i is recurrent
(transient) and i j then j is recurrent (transient).
A special case of a recurrent state is if Pii = 1 then i is
absorbing.
Transient States
A state is said to be a transient state if, upon
leaving this state, the process may never return
to this state again
State i is transient iff there exists a state j
(j i) that is accessible from state i but not vice
versa
If state i is transient, there is a nonzero
probability that the process will never return to i
A transient state will be visited only a finite
number of times

Recurrent State
A state is said to be recurrent state if, upon
leaving this state, it is certain that the process
returns to this state at some point in time
A state is recurrent iff it is not transient
If state i is recurrent, wherever the process goes
upon exiting i, it is certain that it will return to
state i again
A recurrent state will be visited infinitely number
of times

Recurrence, Transience and Other
Properties
Not all states in a finite Markov chain can be transient (why?).
All states of a finite irreducible Markov chain are recurrent.
If Pii 0 whenever n is not divisible by d, and d is the largest
n
integer with this property, then state i is periodic with period d.

If a state has period d = 1, then it is aperiodic.
If state i is recurrent and if, starting in state i, the expected time until
the process returns to state i is finite, it is positive recurrent
(otherwise it is null recurrent).
A positive recurrent, aperiodic state is called ergodic.

Contoh 1

Markov Chain Decomposition

Irreducibility
If all states communicate with each other then
they all form a single class and the Markov chain
is said to be irreducible. See Camera inventory
problem, is it irreducible?.
State 0 1 2 3
0 0.080 0.184 0.368 0.368
1 0.632 0.368 0 0
2 0.264 0.368 0.368 0
3 0.080 0.184 0.368 0.368

Reducible Markov Chains
Transient Irreducible
Set T Set S1
Irreducible
Set S2
In steady state, we know that the Markov chain will
eventually end in an irreducible set and the previous
analysis still holds, or an absorbing state.
The only question that arises, in case there are two or more
irreducible sets, is the probability it will end in each set
Reducible Markov Chains
Transient
Set T s1
r Irreducible
sn
Set S
i
Suppose we start from state i. Then, there are two ways to

go to S.
In one step or
Go to r T after k steps, and then to S.
Define
i S Pr X k S | X 0 i , k 1, 2,...
Example
Irreducible Markov Chain

p01 p12 p22
p00 0 1 2
p10 p21
Reducible Markov Chain

p01 p12 p23
0 1 2 p32 3
p00 p10
p14
Absorbing p22 p33
State 4
Closed irreducible set
Absorbing State
A state is said to be an absorbing state if,
upon entering this state, the process will
never leave this state again.
This occurs iff pii = 1

Markov Chain decomposition into
recurrent classes and transient classes

Markov Chain decomposition into
recurrent classes and transient classes

Some Deduction Properties
All states of a Markov chain are either
recurrent of transient
Recurrence and Transience are class
properties. In a class all states are either
recurrent or transient (Why?)
In a finite-state state Markov chain, not all
states can be transient (Why?)
All states in an irreducible finite state
Markov chain are recurrent
Periodicity
The period of state i is defined to be an integer t
(t > 1) such that pii(n) = 0 for all values of n other
than t, 2t, 3t, and t is the largest integer with
this property
This means that we may visit state i only at
multiples of t
In the gambling example, it is only possible to
enter state 1 (starting from 1) at times 2, 4, ,
so state 1 has a period of 2

Periodicity
Definition (periodic): if state i is periodic with

period d, then
(n)
Pii = 0
where n is not divisible by d, and d is the largest
integer with this property.
E.g. Random walk with Pi, i+1 = = 1 Pi, i-1. In

this case, period is 2.

Example 1
0 1 2 3
0 0 1 0 0
1 0 0 1 0
2 0 0 0 1
3 0 0
p00 = 0, p00(2) = 0, p00(3) = 0, p00(4) = ,

p00(5) = 0, p00(6) = , .
Therefore, the period of state 0 is 2

Example
1 0.5
0 1 2
0.5 1 0 1 0
Periodic State d = 2 P 0.5 0 0.5
0 1 0

Example 2

Aperiodicity, Ergodicity
If there are two consecutive periods s and
s+1 at which the process may visit state i,
the state i is called aperiodic state.
Recurrent states that are aperiodic are
called ergodic states
A Markov chain is said to be ergodic if all
its states are ergodic states

Example
0 1 2 3 4 5
0 1/3 0 1/3 0 0 1/3
1 0 0 0
2 0 0 0 0 1 0
3 0 0
4 0 0 1 0 0 0
5 0 0 0 0 0 1
Equivalence classes:
{5} is an absorbing class
{2, 4}: recurrent class
(each of them has a period of 2)
{0}, {1}, {3}: transient classes
Long Run Properties
Assume that a particular customer
switches brands among products A, B, C,
and D according to the following Markov
chain: State A B C D
A 0.4 0.2 0.2 0.2
P= B 0.05 0.05 0.7 0.2
C 0.05 0.25 0.4 0.3
D 0.1 0.1 0.6 0.2
State A B C D
A 0.200 0.160 0.420 0.220
P2 = B 0.078 0.208 0.445 0.270
C 0.083 0.153 0.525 0.240
3 0.095 0.195 0.450 0.260

Long Run Properties
State A B C D
A 0.108 0.172 0.475 0.245
P4 = P2 * P2 = B 0.094 0.176 0.480 0.250
C 0.094 0.172 0.486 0.248
3 0.096 0.175 0.480 0.249
State A B C D
A 0.096 0.173 0.482 0.248
P8 = P4* P4= B 0.096 0.173 0.482 0.248
C 0.096 0.173 0.482 0.248
3 0.096 0.173 0.482 0.248

Long-Run (Steady-State)
Probability Existence

1 2 3 4
1 0 0 0
2 0 0 0

3 0 0 0

4 0 0 0
Reducible
MC
1 2 3
1 0 1 0
2 0 0

Long-Run Behavior (Steady State) of Markov Chains
3 0 0 60
0 1 2
0 0 1 0
1 0 0 1

Periodic MC
2 1 0 0
Theorem
For any irreducible ergodic Markov chain
( n)
lim p ij exists and is independent of i.
n
Furthermore, n ij j 0 where j
( n)
lim p
satisfy the following steady state equations
M
j i pij for j 0,1,..., M
i 0
M

j 0
j 1.

Partial Proof (?)
Given that j exists and is independent of
the initial state i
M
PX n 1 j PX n 1 j | X n iPX n i
i 0
M
pij PX n i
i 0
Letting n PX n 1 j j , PX n i i
M
j i pij
i 0
Linear Algebra Notation
Let 0 1 .. M for a finite MC.
Then the steady-state equations can be
written as
P
and
1 1 where 1 1 1 ... 1
T

Steady State Probabilities
j's are called steady state probabilities
In an irreducible ergodic MC., whichever the
starting state is, after making sufficiently large
number of transitions, the process will be in state
j with some probability and this probability is
given by j
After the steady state has been reached, the
process still continues to make transitions with
probabilities pij but on the average the process
will be at state j, j percent of the time

Example
State A B C D
A 0.4 0.2 0.2 0.2
P= B 0.05 0.05 0.7 0.2
C 0.05 0.25 0.4 0.3
D 0.1 0.1 0.6 0.2
For the above Markov chain:

1 = 0.41 + 0.05 2 + 0.05 3 + 0.14
2 = 0.21 + 0.052 + 0.253 + 0.14
3 = 0.21 + 0.72 + 0.43 + 0.64
4 = 0.21 + 0.22 + 0.33 + 0.24
1 = 1 + 2 + 3 + 4
5 equations with 4 unknowns.
Solve the above set of equations, you would obtain:
1 = 0.096, 2 = 0.173, 3 = 0.482, 4 = 0.248
Additional remarks on Steady State Probabilities:

If state i and j are recurrent states belonging to
different equivalence classes then pij(n) = 0 for all
n. (Why?)
If state j is a transient state then lim pij 0
( n)
n

( n)
For periodic states n ij may not exist
lim p
0 1
Example: P
1 0
However, for an irreducible (ergodic or not)
Markov Chain the following limit always exist.
1 (k )
n
lim pij j
~
k 1
n n

This limit is the long-run average proportion of
time spent in state j.
Moreover, this limit satisfies the steady-state
equations although it is not the steady-state
probability M
~ j ~i pij for j 0,1,..., M
i 0
M
j
~ 1.

j 0
If the MC is ergodic ~i i

Reducible MC

Example 1
Given the probability transition as

Example 2
Given the probability transition as

Combining these partial answers in matrix form, we have the solution for

Expected Average Cost

Long-Run Expected Average
Cost Per Unit Time
Let C(Xn) be the cost incurred when the process is in
state Xn at period n
If Xn = j C(Xn) = C(j) (the cost of being in state j for a
period)
The expected average cost per period incurred in the
first k periods is 1 k
E C X k
k n 1
It can be shown that the long-run average expected cost
per period is .
1 k M ~
lim E C X k j C ( j )
k
k n 1 j 0
Proof
1 k 1 k
E C X n E C X n
k n 1 k n 1
1 k M 1 k M
C j PX n j | X 0 i C j pij
(n)
k n 1 j 0 k n 1 j 0
1 k 1 k M
lim E C X n lim C j pij( n )
k
k n 1 k k n 1 j 0
M
1 k (n) M
C j lim pij C j ~ j
k k
j 0 n 1 j 0

Long-Run Average Cost Per
Unit Time
The long-run actual average cost is
1 k
lim C X k
n 1
k k

It can be shown that for an irreducible MC. this
limit converges to a single number. Thus
1 k M ~
lim C X k j C ( j )
n 1
k k
j 0

Inventory Example
Xn is defined as the inventory in stock at the end
of week n. Assume that cost of carrying
inventory from one week to the other is $2 per
unit. Let C(Xn) be the cost incurred when the
Markov chain is at state Xn at time t
Recall that Xt = {0, 1, 2, 3}. Therefore C(0) = 0,
C(1) = 2, C(2) = 4, and C(3) = 6
Xt and C(Xt) are random variables

Inventory Example (modified)
In the previous inventory example, let us
consider a different cost structure. For
every time we place an order, $3 is
charged. For each item ordered there is a
unit cost of $25. Moreover, for every
customer lost, we incur a penalty cost of
$50 for lost sales. What is the expected
average cost of this system?

Unit Time
Question: What if the cost to be incurred
cannot be expressed as a function of the state at
time n?
In the modified inventory example:
Ordering Cost = 10 +25z (when z units ordered)
Shortage Cost = 50s ( for s units demand not
satisfied)
Then the cost for the nth week is
10 (25)(3) 50 max Dn 3,0 if X n 1 0
C X n 1 , Dn
50 max Dn X n 1 ,0 if X n 1 1

Unit Time
Under this scenario the long-run average cost is
1
k M
lim C X , D k j where k j E C j , D
n 1 n j n
k k
n 1 j 0
Dn
In the inventory problem:

K(0) = E[C(0,Dn)] = 85 + 50[PD(4) + 2PD(5) + 3PD(6) + ] =
85 + 50[0.015 + 2*0.003 + 3*0.001] = 86.2
K(1) = E[C(1,Dn)] = 50[PD(2) + 2 PD(3) + 3 PD(4) + ] = 18.4
K(2) = E[C(2,Dn)] = 50[PD(3)+2 PD(4)+3 PD(5)+] = 5.2
K(2) = E[C(3,Dn)] = 50[PD(4)+2 PD(5)+3 PD(6)+] = 1.2
Then the long-run average cost is
3
k j j 86.2(0.286) 18.4(0.285) 5.2(0.263) 1.2(0.166)

j 0
$31.46
Unit Time
The method described can be employed
given that:
The cost can be written as
1 k
lim C X n , Dn m where m0
k k
n 1
P{Xn, n 0} is an irreducible MC.
{Dn}s form a sequence of random variables
which are independent and identically
distributed (iid)

Existence of Stationary
Distribution
Theorem 2: Irreducible aperiodic Markov chain. There
are two possibilities for scalars:
j lim P{ X n j | X 0 i} lim Pijn
n n
1. j = 0, for all states j No stationary distribution

2. j > 0, for all states j is the unique stationary
distribution
Remark: If the number of states is finite, case 2 is the
only possibility to be as an Irreducible
aperiodic Markov chain

Ergodic Markov Chains
Markov chain with a stationary distribution
j 0, j 0,1, 2,...
States are positive recurrent: The process returns to
state j infinitely often
A positive recurrent and aperiodic Markov chain is called
ergodic
Ergodic chains have a unique stationary distribution
j lim Pijn
n
Ergodicity Time Averages = Stochastic Averages

Calculation of Stationary
Distribution
A. Finite number of states B. Infinite number of states
Solve explicitly the system of Cannot apply previous methods
equations to problem of infinite dimension
m
Guess a solution to recurrence:
j i Pij , j 0,1,..., m
m
i 0
j i Pij , j 0,1,...,
i 1
i 0
i 0
i 1
Numerically from Pn which i 0
converges to a matrix with

rows equal to
Suitable for a small number
of states

Example: Finite Markov Chain
(Absent-minded professor)
1 p
Absent-minded professor
0 2 1 1 p
uses two umbrellas when
commuting between home 1 p p
and office. If it rains and an Markov chain formulation
umbrella is available at her i is the number of umbrellas
location, she takes it. If it available at her current
does not rain, she always location
forgets to take an umbrella. Transition matrix
Let p be the probability of
rain each time she 0 0 1
commutes. What is the P 0 1 p p

probability that she gets 1 p p 0
wet on any given day?
1 p
0 0 1
0 2 1 1 p P 0 1 p p

1 p p 0
1 p p
0 (1 p ) 2
P (1 p ) p
1 1 p 1 1

1 2
, ,

i i 1 2 0 p 1
0
3 p
1
3 p
2
3 p
0 1 2 1
1 p
P{gets wet} 0 p p
3 p

Taking p = 0.1:
1 p 1 1
, , 0.310, 0.345, 0.345
3 p 3 p 3 p
0 0 1
P 0 0.9 0.1

0.9 0.1 0
Numerically determine limit of Pn
0.310 0.345 0.345
lim P n 0.310 0.345 0.345 ( n 150)
n
0.310 0.345 0.345
Effectiveness depends on structure of P

Global Balance Equations
Markov chain with infinite number of states
Global Balance Equations (GBE)

j Pji i Pij j Pji i Pij , j0
i 0 i 0 i j i j
j Pji is the frequency of transitions from j to i

Frequency of Frequency of
transitions out of j transitions into j

Intuition: j visited infinitely often; for each transition
out of j there must be a subsequent transition into j
with probability 1

Alternative Form of GBE

P P ,
jS
j
iS
ji
iS
i
jS
ij S 0,1,2,...
If a probability distribution satisfies the GBE,

then it is the unique stationary distribution of
the Markov chain
Finding the stationary distribution:
Guess distribution from properties of the system
Verify that it satisfies the GBE
Special structure of the Markov chain simplifies
taskState) of Markov Chains
Long-Run Behavior (Steady 97
Proof

j i Pij and P ji 1
i 0 i 0

j Pji i Pij j Pji i Pij
i 0 i 0 i j i j

j Pji i Pij j Pji i Pij
i 0 i 0 jS i 0 jS i 0

j jiP Pji i ij i ij
P P
jS iS iS jS iS iS
P P
jS
j
iS
ji
iS
i
jS
ij

Interpretation of Balanced Equation

Bird-Death Processes
(BDP)

Transition probability graph for BDP

Birth-Death Process
S Sc
P01 Pn 1,n Pn ,n 1
0 1 2 n n+1
P10 Pn ,n 1 Pn 1,n
P00 Pn ,n
One-dimensional Markov chain with transitions only

between neighboring states: Pij=0, if |i-j|>1
Detailed Balance Equations (DBE)
n Pn ,n 1 n 1Pn 1,n n 0,1,...
Proof: GBE with S ={0,1,,n} give:
n n
P P
j 0 i n 1
j ji
j 0 i n 1
i ij n Pn ,n 1 n 1Pn 1,n

Example 1: Random walk with
reflecting barriers

Example: Discrete-Time Queue
In a time-slot, one arrival with probability p or zero
arrivals with probability 1-p
In a time-slot, the customer in service departs with
probability q or stays with probability 1-q
Independent arrivals and service times
State: number of customers in system
p p(1 q ) p p(1 q )
0 1 2 n n+1
q(1 p) q(1 p) q(1 p)
(1 p) (1 p)(1 q) pq (1 p)(1 q) pq

p p(1 q ) p(1 q ) p(1 q )
0 1 2 n n+1
q(1 p) q(1 p) q(1 p)
(1 p) (1 p)(1 q) pq (1 p)(1 q) pq
p/q
0 p 1q(1 p) 1 0
1 p
p(1 q)
n p(1 q) n 1q(1 p) n 1 n , n 1
q(1 p)
p(1 q)
Define: p / q,
q(1 p)

1 0
1 p n n 1 0 , n 1
, n 1 1 p
n 1 n

Have determined the distribution as a function of 0

n n 1 0 , n 1
1 p
How do we calculate the normalization constant 0?

Probability conservation law:
1 1
n 1 1
n 0 n 1 p n 1

1 0 1 1
(1 p) (1 )

Noting that
1 p 1 1 p q(1 p) p(1 q) q p 1
q(1 p) q
0 1

n (1 ) , n 1
n 1

Detailed Balance Equations
General case:
j Pji i Pij i , j 0,1,...
Imply the GBE
Need not hold for a given Markov chain
Greatly simplify the calculation of stationary distribution
Methodology:
Assume DBE hold have to guess their form
Solve the system defined by DBE and ii = 1
If system is inconsistent, then DBE do not hold
If system has a solution {i: i=0,1,}, then this is the unique
stationary distribution

Limiting Probability (1)
Definition (limiting probability): Only when all

states in a Markov Chain is ergodic (aperiodic,
positive recurrent) do limn Pijn exists and we
denote
j = limn Pijn
as the limiting probability.

Limiting probability has two role

In the long run, the probability that the process is in
state j.
The long run proportion of time the the process is in
state j.
Limiting probability is the unique non-negative

solution that satisfy the following:

j = i =0 P , j >= 0
i ij
0 j = 1
Example 1: a transition prob. Matrix P is said to be

doubly stochastic if sum over each column equals
one; that is
i = 0 Pij = 1, for all j
Suppose such a chain is irreducible and ergodic and
consist of M+1 states, 0, 1, 2, , M. What is the
limiting probability, then?
Solution:

We suppose j = 1/(M+1) for all j is a solution. Clearly,
j = 0 M j = 1
j = 1/(M+1),
i =0 i Pij
= (1/M+1) i =0 Pij
= 1/(M+1)
Since limiting probabilities is the unique positive solution. The
limiting probability for all states is 1/(M+1).

Limiting Probabilities
Theorem: For an irreducible ergodic Markov chain, j lim Pij
n
n
exists for all j and is independent of i. Furthermore, j is the
unique nonnegative solution of

j i Pij , j 0
i 0

j 0
j 1
The probability j also equals the long run proportion of time

that the process is in state j.
If the chain is irreducible and positive recurrent but periodic,
the same system of equations can be solved for these long run
proportions.

Limiting Probabilities 2
The long run proportions j are also called stationary
probabilities because if P X 0 j j
then P X n j j for all n, j 0
Let mjj be the expected number of transitions until the
Markov chain, starting in state j, returns to state j
j
m jj 1Then
(finite if state j is positive recurrent).
If X n , n 0 is an irreducible Markov chain with
stationary probabilities , and r is a bounded function
on the state space. Then with probability 1,
r Xn
N

Long run
lim n 1
r j j
N N j 0
average reward

Transient Analysis
Suppose a finite Markov chain with m states has some transient
states. Assume the states are numbered so that T = {1, 2, , t}
is the set of transient states, and let PT be the matrix of transition
probabilities among these states.
Let R be the t x (m-t) matrix of one-step transition probabilities
from transient states to the recurrent states and PR be the (m-t) x
(m-t) matrix of transition probabilities among the recurrent states:
the overall one-step transition probability matrix can be written
as
PT R
P
0 PR
If the recurrent states are all
absorbing then PR = I.

Transient Analysis 2
If the process starts in a transient state, how long does it spend

among the transient states?
What are the probabilities of eventually entering a given recurrent
state?
Define dij = 1 if i = j and 0 otherwise.
For i and j in T, let sij be the expected number of periods that the
Markov chain is in state j given that it started in state i.
T
Condition on the first transition,
sij d ij Pik skj
k 1
and note that transitions from recurrent
states to transient states are impossible

Transient Analysis 3
Let S be the matrix of sij values. Then S = I + PTS. Or,

I PT S I
S I PT
1
For i and j in T, let fij be the probability that the Markov

chain ever makes a transition into j, starting from i.
sij d ij
fij
s
For i in T and j in Tc, the matrix of these probabilities isjj
F I PT R
1

Time Reversibility
One approach to estimate transition probabilities from each state
is by looking at transitions into states and tracking what the
previous state was.
How do we know this information is reliable?
How do we use it to estimate the forward transition probabilities?
Consider a stationary ergodic Markov chain.
Trace the sequence of states going backwards: Xn, Xn-1,, X0
This is a Markov chain with transition probabilities:
j Pji
Qij P X m j X m1 i
i
If Qij = Pij for all i, j, then the Markov chain is time reversible.

Time Reversibility 2
Another way of writing the reversibility equation is:

i Qij j Pji
Proposition: Consider an irreducible Markov chain with
transition probabilities Pij. If one can find positive
numbers i summing to 1 and a transition probability
matrix Q such that the above equation holds for all i, j,
then Qij are the transition probabilities for the reversed
chain and the i are the stationary probabilities for both
the original and the reversed chain.
Use this, thinking backwards, to guess at transition

probabilities of reversed chain.
Contoh-2

Example
A flea moves around the vertices of a triangle in the

following manner: Whenever it is at vertex i it moves to its
clockwise neighbor vertex with probability pi and to the
counterclockwise neighbor with probability qi = 1- pi, i = 1,
2, 3.
(a) Find the proportion of time that the flea is at each of
the vertices?
(b) How often does the flea make counterclockwise move
which is then followed by 5 consecutive clockwise moves?
Solution:

Example
|| 0 p1 q1 ||
|| ||
P = ||q2 0 p2 ||
|| ||
||p3 q3 0 ||
(a) Solve the following equation:
1 + 2 + 3 = 1,
1 = q2 2 + p3 3
2 = p1 1 + q3 3

|| 0 p1 q1 ||
|| ||
P = ||q2 0 p2 ||
Example || ||
||p3 q3 0 ||
Long run proportion in vertex 1 is 1. How often

flea at vertex 1 go anticlockwise follow by 5
clockwise is
a = 1q1p3p1p2p3p1 = 1q1(p1)2p2(p3)2

clockwise is
b = 2q2p1p2p3p1p2 = 2q2(p1)2 (p2)2p3

clockwise Is
c = 3q3p2p3p1p2p3 = 3q3p1 (p2)2 (p3)2
How often is a + b + c
Example
0 The trader is free
1 The trader is handling an
institutional account in its first minute
2 The trader is handling the
institutional account in its last second
minute
3 The trader is handling an individual
account

Say that we decide to give priority to
institutional accounts. In this case, the
transition probabilities are :
P00 (1 1 / 3)(1 1 / 2) 1 / 3
P01 1 / 3
P02 0; P03 (1 1 / 3)(1 / 2) 1 / 3
P10 0; P11 0; P12 1; P13 0
P20 (1 1 / 2)(1 1 / 3) 1 / 3; P21 1 / 3; P22 0; P23 (1 1 / 3)(1 / 2) 1 / 3
P30 (1 1 / 2)(1 1 / 3) 1 / 3; P31 1 / 3; P32 0; P33 (1 1 / 3)(1 / 2) 1 / 3

The Markov chain is thus
1 / 3 1 / 3 0 1 / 3
0 0 1 0
P
1 / 3 1 / 3 0 1 / 3

1 / 3 1 / 3 0 1 / 3
2 / 9 2 / 9 3 / 9 2 / 9
3 / 9 3 / 9 0 / 9 3 / 9
P2
2 / 9 2 / 9 3 / 9 2 / 9

2 / 9 2 / 9 3 / 9 2 / 9

And in the long run
1 / 4 1 / 4 1 / 4 1 / 4
1 / 4 1 / 4 1 / 4 1 / 4

1 / 4 1 / 4 1 / 4 1 / 4

1 / 4 1 / 4 1 / 4 1 / 4

As a result, when priority is given to institutional
accounts, the trader will be free of the time.
If we give priority to individual accounts, it is simple

to verify that the corresponding markov chain is :
1 / 3 1 / 6 0 1 / 2
0 0 1 0
P
1 / 3 1 / 6 0 1 / 2

1 / 3 1 / 6 0 1 / 2
And in the long run
2 / 7 1 / 7 1 / 7 3 / 7
2 / 7 1 / 7 1 / 7 3 / 7

2 / 7 1 / 7 1 / 7 3 / 7

2 / 7 1 / 7 1 / 7 3 / 7
Insurance Firms market share
The current market position of an
insurance firm as well as its two
competitors is given by the following :
1 El Rose Ins 12 %
2 Cte dOr Ins 40 %
3 Majestic Ins 48 %

Based on industry data, clients switch from year to year
from one fund to another. The reasons are numerous but
include as well deceptions and below expectations results.
The switching fund matrix is calculated to be
.1 .3 .6

P .1 .5 .4
.1 .3 .6
Given this matrix, we wish to predict the yearly
profit of each fund (from transactions and
percentage of the funds gains) when the initial
market share of each of the funds is given by
b(0) .12 .40 .48

For simplicity, assume that there are
altogether 100 major accounts while the
profit per account is on the average 30
million dollars per year. In the long run
what market share would each of the
funds have ? Of course, initially the funds
share are
b(0) .12 .40 .48

Then by the Chapman-Kolmogorov
equation, at the nth period
b(n) b(0)P n

At the first year
b1 (1) b1 (0) p11 b2 (0) p21 b3 (0) p31
b (1) b (0) p b (0) p b (0) p
2 1 12 2 22 3 32
b3 (1) b1 (0) p13 b2 (0) p23 b3 (0) p33
or
b1 (1) .12(.1) .40(.1) .48(.1) 010
.
b (1) .12(.3) .40(.5) .48(.3) 0.38
2
b3 (1) .12(.6) .40(.4) .48(.6) 0.52

El Rose Fund: (30)(100,000(.12 + 0.10+...) =....
Cote d' Or Fund: (30)(100,000(.40+.0.38+...) =.....

Majectic Fund: (30)(100,000(.48 + 0.52+...) =.....

In the long run
1 P11 2 P21 3 P31 1
1 P12 2 P22 3 P32 2
1 P13 2 P23 3 P33 3
1 2 3 1
1 (01
. ) 2 (01
. ) 3 (01
. ) 1
1 (0.3) 2 (0.5) 3 (0.3) 2
1 (01
. ) 2 (0.3) 3 (0.6) 3
1 2 3 1
1 0.1; 2 0.375; 3 0.525

Problem
The Surest insurance co. has devised a premium
formula based on claiming records of insured clients.
If no accident occurs in the last two years, the
insurance premium is $400. If there was an accident
in one of the past two years, the premium to be paid
is $750. If there were two accidents in the last two
year, the premium to be paid is $1200. Data analysis
based on past claim records indicates (a) If an
accident has occurred in the last year, there is a 12%
chance of an accident this year. (b) If no accident
has occurred last year, then there is a 4% chance of
an accident this year. What is the average premium
paid by an insured to the company. What is the
probability that an insured will not claim in 5 years,
that it will claim twice in 4 years.
A Gamblers problem

Insurance for a car fleet
(Tijms)
: A transport firm insures its cars. The
insurance premium is paid in the
beginning of each year and there are 4
premium classes,
The premiumPi , i paid 4 Pi 1 Pi on the past
1,2,3,depends
claims. If the premium is
and no claims is made in Pthe i previous
year, then the current premium is
Pi 1

otherwise it equals
. Since the firms has P1many cars, it has the
choice to report or not an accident. If an
accident is reported, the insurance firm pays to
the company the amount claimed less a
participation when the premium levelri is at i
Car accident claims are known to follow a known
probability distribution which is independent from
one year to the next. The decision to claim
associated to each class i is given by
1 2 3 4 , i ri
, above which the claim is reported. The
problem we are faced with is to
construct the corresponding Markov
chain. To do so, we let be switching
probability matrix P when moving from
one state to another. It is given by :

pi1 1 F ( i ); i 1,2,3,4
pi ,i 1 F ( i ); i 1,2,3
p44 F ( 4 )
otherwise, pij 0

or
p11 p12 0 0
p 0 p23 0
P
21
p31 0 0 p34

p41 0 0 p44
What are the ergodic probabilities and

what is the expected cost of insurance

And therefore
1 F1 (1 ) F1 (1 ) 0 0
1 F ( ) 0 F ( ) 0
P 2 2 2 2
1 F3 ( 3 ) 0 0 F3 ( 3 )

1 F4 ( 4 ) 0 0 F4 ( 4 )

3 Discrete Markov Chain (Long Run)

Diunggah oleh

Informasi Dokumen

Deskripsi Asli:

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

3 Discrete Markov Chain (Long Run)

Diunggah oleh

Hak Cipta:

Format Tersedia

Long-Run Behavior of

Long-Run Behavior (Steady State) of Markov Chains 1

Long-Run Behavior (Steady State) of Markov Chains 2

Long-Run Behavior (Steady State) of Markov Chains 3

Long-Run Behavior (Steady State) of Markov Chains 4

Long-Run Behavior (Steady State) of Markov Chains 8

Long-Run Behavior (Steady State) of Markov Chains 9

Long-Run Behavior (Steady State) of Markov Chains 10

Long-Run Behavior (Steady State) of Markov Chains 11

Long-Run Behavior (Steady State) of Markov Chains 12

Long-Run Behavior (Steady State) of Markov Chains 13

Long-Run Behavior (Steady State) of Markov Chains 19

Long-Run Behavior (Steady State) of Markov Chains 20

exists and is independent of initial state i

Long-Run Behavior (Steady State) of Markov Chains 21

Long-Run Behavior (Steady State) of Markov Chains 22

On the long run, n

Definition: A stochastic process is said to be

E.g. Pr{Xn = 1} = Pr{Xn+1 = 1}

Long-Run Behavior (Steady State) of Markov Chains 28

Long-Run Behavior (Steady State) of Markov Chains 29

Long-Run Behavior (Steady State) of Markov Chains 30

Long-Run Behavior (Steady State) of Markov Chains 31

Long-Run Behavior (Steady State) of Markov Chains 33

Long-Run Behavior (Steady State) of Markov Chains 34

integer with this property, then state i is periodic with period d.

Long-Run Behavior (Steady State) of Markov Chains 35

Long-Run Behavior (Steady State) of Markov Chains 37

Long-Run Behavior (Steady State) of Markov Chains 38

Long-Run Behavior (Steady State) of Markov Chains 39

Suppose we start from state i. Then, there are two ways to

Irreducible Markov Chain

Reducible Markov Chain

Long-Run Behavior (Steady State) of Markov Chains 44

Long-Run Behavior (Steady State) of Markov Chains 45

Long-Run Behavior (Steady State) of Markov Chains 46

Long-Run Behavior (Steady State) of Markov Chains 48

Definition (periodic): if state i is periodic with

E.g. Random walk with Pi, i+1 = = 1 Pi, i-1. In

Long-Run Behavior (Steady State) of Markov Chains 49

p00 = 0, p00(2) = 0, p00(3) = 0, p00(4) = ,

Long-Run Behavior (Steady State) of Markov Chains 50

Long-Run Behavior (Steady State) of Markov Chains 52

Long-Run Behavior (Steady State) of Markov Chains 53

Long-Run Behavior (Steady State) of Markov Chains 54

Long-Run Behavior (Steady State) of Markov Chains 56

Long-Run Behavior (Steady State) of Markov Chains 57

Long-Run Behavior (Steady State) of Markov Chains 58

Long-Run Behavior (Steady State) of Markov Chains 65

Long-Run Behavior (Steady State) of Markov Chains 67

Long-Run Behavior (Steady State) of Markov Chains 68

For the above Markov chain:

Additional remarks on Steady State Probabilities:

Long-Run Behavior (Steady State) of Markov Chains 70

Long-Run Behavior (Steady State) of Markov Chains 72

Long-Run Behavior (Steady State) of Markov Chains 73

Long-Run Behavior (Steady State) of Markov Chains 76

Long-Run Behavior (Steady State) of Markov Chains 78

Long-Run Behavior (Steady State) of Markov Chains 79

Long-Run Behavior (Steady State) of Markov Chains 81

Long-Run Behavior (Steady State) of Markov Chains 83

Long-Run Behavior (Steady State) of Markov Chains 84

Long-Run Behavior (Steady State) of Markov Chains 85