24

Chapter 4. The Long Run Behavior of Markov Chains

In the long run, we are all equal. —-with apology to John Maynard Keynes

4.1. Regular Markov chains.

Example 4.1 ⋆ Let {Xn } be a M C with two states 0 and 1, and transition matrix:
 
0.33 0.67
P= .
0.75 0.25

Then,
   
2 0.611 0.389 5 0.524 0.476
P = , P = ,
0.438 0.562 0.536 0.464
   
0.528 0.472 0.5294 0.4706
P7 = , 16
.... P = ....
0.530 0.469 0.5294 0.4706

Recall that P(n) = Pn . We get the impression from this example that the n-step transition matrix
converge, actually quite fast in this example, and that limits in the same columns are the same.
The question is: is this a general phenomenon or only pertained to this M C? Note that all the
above transition matrices, one-step or many step, have all positive entries. 
Example 4.2 ⋆ Let {Xn } be a M C with two states 0 and 1, and transition matrix:
 
0 1
P= .
1 0

In other words, from state 1, the M C sure goes to 0 the next step; and from state 0, it sure goes
to 1 the next step. Then
 
1 0
Pk = P for all odd k, Pk = for all even k.
0 1

Clearly, Pn does not converge as n → ∞.

Example 4.3 ⋆ (Wealthiness does not last long.) According to an old Chinese saying: Fu
Bu Guo San Dai (wealthiness does not last beyond third generation—a word-by-word translation).
Indeed, suppose every family in a population assumes one of N + 1 financial classes/states: State
0 is the poorest, 1 the second poorest, ..., N the richest. Let Xn the state of the n-th generation
of a family. It is reasonable to assume a transition matrix with all Pij > 0, i, j = 0, 1, ..., N . Such
transition probability matrix and the M C are so called regular, to be defined below. It will be
shown that, in the long run, the family, no matter it started rich or poor, will turn out ordinary
after many generations. The same is true for, other than wealth, height or IQ or nearly all of the
biological traits. The hundredth generation of Yao, Ming, the star basketball player, and that of
Pan, Changjiang, a Chinese comedy star known for being short and funny, will have about same
distribution of height. The former can very well be shorter than the latter, and, even if taller, it’s
largely by randomness.
Indeed, we shall show a large class of M Cs, called regular M Cs, the transition matrices do converge.
Example 4.1 is one of those and Example 4.2 is not.
Definition (Regular Markov Chains)
(i). A transition probability matrix P is called a regular transition probability matrix, if there
exists a k > 0 such that all entries of Pk are positive.
(ii). A M C is called regular if its transition probability matrix is regular.
 25

The following is another example of three state regular transition matrix:

 
0 1/2 1/2
P =  1/4 0 3/4 
2/3 1/6 1/6
The following are three state non-regular transition matrices:
   
0 1 0 0.5 0.4 0.1
P = 0 0 1 P =  0.2 0.8 0 
1 0 0 0 0 1

DIY: ⋆ verify that the two matrices are non-regular. Hint: for the former, Pk for all k > 0 can be
figured out; for the latter, the third rows of Pk is always 0, 0, 1 for all k > 0.
Given a transition probability matrix, it is often not easy to identify whether it is regular. The
following suf f icient condition offers some ease for.
Suppose (i), for any i and j in the state space, there exists a k¿0 (which depends on i and j) such
(k)
that Pij > 0; and (ii), there exists a state i such that Pii > 0. Then the transition matrix is
regular.
For example, consider
0 1 0 0 0 0 1 0 0 0
   
 0 0 1 0 0  0 0 1 0 0
P= 0 0 0 1 0  is regular , but P =  0 0 0 1 0  is not.
   
0 0 0 0 1 0 0 0 0 1
   
0.99 0 0 0 0.01 1 0 0 0 0
The former can be easily checked by the above sufficient condition, while the latter does not satisfy
the condition since its diagonal elements are all 0.
REMARK. In the definition of regular transition matrix, Pk has all positive elements for some
k > 0. Then, for all n > k, Pn all have all positive elements (Please DIY). In other words, an
(n)
equivalent definition is there exists a k > 0 such that, for all states i, j and all n ≥ k, Pij > 0.
We also note that Part (i) of the above sufficient condition, in which k depends on i and j, is
very different from the universal k in the definition of regular transition matrix. We shall later
see that Part (i) of the above sufficient condition says all states are communicative and Part (ii)
says one state is aperiodic. (Under communicativeness, all states are aperiodic.) In other words,
communicativeness plus aperiodicity of all states ensures the regularity.

4.2. The limit of regular Markov chains.

The long run behavior of regular M Cs is derived from the following theorem, which is the main
result of Chapter 4.
Theorem 4.1. Let P be a regular transition matrix with states {0, 1, ..., N }. Then, for all states
i and j, as n → ∞,
(n)
Pij → πj > 0 and, as a result, P (Xn = j) → πj > 0, (4.1)
where (π0 , π1 , ..., πN ) is the unique solution to
Pn
πj = j=0 πk Pkj , j = 0, 1, ..., N (4.2)

PN
i=0 πi = 1. (4.3)

Equation (4.1) is also the definition of πj . However its implication lies three folds: (a). the limit
(n)
of Pij as n → ∞ exists; (b). the limit is positive; and (c). the limit is irrelevant with the starting
state i. Equations (4.2) can be conveniently written in matrix form
(π0 π1 ... πN ) = (π0 π1 ... πN )P. (4.2′ )
26

It appears to be odd that there are N + 2 equations in (4.2) and (4.3) to solve for N + 1 variables
π0 , ..., πN (Please DIY ⋆ ⋆ ⋆).
A rigorous proof of Theorem 4.1 is technical, especially about the existence of the limit, and is
beyond the pursuit of this course. Here we offer a brief explanation. First, should the limit πj exist,
they must satisfy the equation (4.2). This is because, for j = 0, ..., N ,
N N N
(n) (n−1) (n−1)
X X X
πj ≡ lim Pij = lim Pik Pkj = lim Pik Pkj = πk Pkj .
n→∞ n→∞ n→∞
k=0 k=0 k=0

The interpretation of limit theorem is, in my view, more important than the technical proofs. The
interpretation of (4.1) has two faces that I call the space average and the time average. The space
average interpretation is, at any fixed time long into the future, the MC has about probability πj to
be in state j. This gives the distribution of the M C Xn for a fixed large n in the state space. The
time average interpretation is, over a long period of time, the relative frequency of the MC staying
in state j is about πj .
A little more elaboration on the time average. Suppose there is a payoff cj whenever the MC visits
state j once. Assume X0 = k. The long run average of payoff is
n
X n X
X N
An ≡ cXi /n = cj I{Xi =j} /n.
i=1 i=1 j=0

Heuristically, for large n, An , which is random, would be close to its mean, which is
n X
N n X
N N n N
(i)
X X X X X
E(An ) = cj E(I{Xi =j} )/n = cj P (Xi = j|X0 = k)/n = cj [ Pkj /n] ≈ cj πj .
i=1 j=0 i=1 j=0 j=0 i=1 j=0

Now, consider the special case with cl = 1 for a particular l and the rest c0 , ...cl−1 , cl+1 ..., cN being
PN
all 0. Then An is the fraction of the MC visits of state l, which is indeed close to j=0 cj πj = πl .
Example 4.4 ⋆ ⋆ ⋆ Mr. C spends his life time only in three places Home, Office and Gym. Assume
every time he reaches H, O and G, he spends 12, 10 and 2 hours, respectively, and then transits to
another location according to the following transition probability matrix

H O G
H 0 0.9 .1
P = O  .1 0 .9 
G 0.8 0.2 0

It is easily verified that P2 has all entries positive and therefor P is regular, which is perhaps easier
to see from the this figure.

Gym

Home Office

Solving equation (4.2)-(4.3), we have

πH = 0.309 πO = 0.347 πG = 0.343.

Let Xn be Mr C’s whereabout after n transitions. {Xn : n ≥ 0} is a MC with state space {H, O, G}
and transition matrix P. Let nH be the number of times he stays in state H in a long period with
 27

(a large) n transitions. Likewise define nO and nG . The long run fraction of his life time spent in
office is
nO × 10 10nO /n
=
nH × 12 + nO × 10 + nG × 2 12nH /n + 10nO /n + 2nG /n
10πO
≈ = 0.441.
12πH + 10πO + 2πG
Translating into daily hours, it is 0.441 ∗ 24 = 10.58 hours in office per day. Likewise, the long run
fractions of his life time spent in Home and Gym are, respectively, 0.472 and 0.087, which are 11.32
and 2.09 in daily hours. 
DIY:
Exercise 4.1. ⋆ ⋆ ⋆⋆ Following the above example, What is the average number of times per day
that Mr C goes from Gym to Home?