MATH11029 Stochastic Modelling (MSc) 1

(1) Consider a discrete time Markov chain (DTMC) with the following state diagram

1
0.6 0.4
0.6
2 4
1
0.4
1
0.9
0.1 3 5 6
1

(a) Write down the transition matrix P . [4 marks]

(b) Identify the communicating classes. [3 marks]
(c) For each state show whether it is positive recurrent, null recurrent or transient.
[3 marks]
(d) Which states are periodic, and what is their period? [3 marks]
(n)
(e) Calculate limn→∞ pi,j for all i, j ∈ S if it exists. [12 marks]

[Please turn over]

MATH11029 Stochastic Modelling (MSc) 2

Solution:

(a) Each row sums up to one in

 
0 0.6 0 0.4 0 0
0.6 0 0.4 0 0 0
 
0 0 0.1 0 0.9 0
P =
 
 0 0 0 0 0 1

0 0 1 0 0 0
0 0 0 1 0 0

(b) {1, 2}, {3, 5}, {4, 6}

(c) {3, 5} and {4, 6} are closed and finite, hence they are both positive recurrent.
{1, 2} is not closed hence it’s transient.
(d) Periodicity is a class property, and clearly {3, 5} is aperiodic, while {1, 2} and
{4, 6} has a period 2.
(n)
(e) Since j = 1, 2 are transient states, for these states limn→∞ pi,j = 0 for all i.
(n)
Also, since states j = 4, 6 are periodic, limn→∞ pi,j does not exist for those j
values, and for i = 1, 2, 4, 6, but the limit is zero for i = 3, 5. Hence we only
care about ending up in states j = 3, 5. Consider first the irreducible chain on
the state space {3, 5}, for which the detailed balance equations are 0.9π3 = π5
and π3 + π5 = 1, which has the solution π3 = 10/19 and π5 = 9/19. Hence
(n) (n)
limn→∞ pi,3 = 10/19 = 0.526316 . . . and limn→∞ pi,5 = 9/19 = 0.473684 . . . for
i = 3, 5. If we start from states i = 4, 6 we cannot end up in states j = 3, 5,
(n)
hence limn→∞ pi,j = 0 in these cases.
Let hi be the probability of ever reaching states {3, 5} when starting from state
i = 1, 2. Clearly h5 = h3 = 1 and h4 = h6 = 0. Conditioning on the first step
we write
h1 = 0.6h2
h2 = 0.6h1 + 0.4
which has the solution h1 = 3/8 = 0.375, h2 = 5/8 = 0.625. If reached the class
{3, 5}, it reaches the stationary state, hence, e.g.

(n) 3 10
lim p1,3 = h1 π3 = × = 0.197368
n→∞ 8 19
Arguing similarly for 3 more elements, we arrive at the whole limit matrix
 
0 0 15/76 − 27/152 −
0 0 25/76 − 45/152 −
 
n
0 0 10/19 0 9/19 0
lim P =  
n→∞ 0 0
 0 − 0 −

0 0 10/19 0 9/19 0
0 0 0 − 0 −

[Please turn over]

MATH11029 Stochastic Modelling (MSc) 3

(2) Consider the following DTMC. At time n you have Xn (fair) coins. In each step you
do the following. First, toss all your coins simultaneously. If you get more heads
than tails, then add two more coins to your coins, but if you have more tails than
heads, then keep only one coin and throw away the rest. Suppose you have one coin
initially, so X0 = 1, and the state space is S = {1, 3, 5, . . . }.

(a) Write down the one step transition probabilities. [4 marks]

(b) Identify the communicating classes. [4 marks]
(c) For each state show whether it is transient or recurrent. [4 marks]
(d) Find the stationary solution. [5 marks]
(e) What is the mean time to first return to state 1 from state 1? [4 marks]
(f) Is state 77 positive or null recurrent? [4 marks]

[Please turn over]

MATH11029 Stochastic Modelling (MSc) 4

Solution:

(a) Since heads and tails are symmetric (and there cannot be ties), you have more
tails then heads with probability 1/2. Hence pi,i+2 = pi,1 = 1/2 for all i ∈ S,
and all other transition probabilities are zero.
(b) State i + 2 is accessible from i ∈ S, hence, due to transitivity and induction,
from state 1 you can reach any state j. Also you can reach state 1 from any
state i in one step. Then again, from transitivity, you can reach any state j from
any state i, and vice versa. Hence all states communicate, and form a single
communicating class.
(c) Since transience and recurrence are class properties, and we have a single
communicating class, it is enough to show this for state 1. Starting from state
1 you never return only if you add 2 coins in each step, that is with probability

lim 1/2n = 0
n→∞

Hence state 1 is – and so all states are – recurrent.

(d) We have the detailed balance
P condition πj+2 = πj /2 for j ∈ S, which with the
(j+1)/2
normalization condition j∈S πj = 1 leads to πj = 1/2 for j ∈ S.
(e) This is just m11 = 1/π1 = 2.
(f) Since m11 < ∞, state 1 is positive recurrent, which is a class property, so all
states are positive recurrent. Alternatively, since πj > 0 for all j, all states are
positive recurrent.

[Please turn over]

MATH11029 Stochastic Modelling (MSc) 5

(3) Children arrive at an ice-cream parlour as a non-homogeneous Poisson process at

rate (
3 0≤t≤3
λ(t) =
t 3≤t≤6
where t (a real number) is the time measured in hours between noon and 6 in the
afternoon.

(a) If we know that exactly 20 children arrived between noon and 6, what is the
probability that exactly 3 arrived between 1 and 2? [5 marks]
(b) Each child turns out independently to be a girl with probability 2/3 and a boy
otherwise. What is the probability that exactly 5 girls arrive between 5 and
6? [6 marks]
(c) Boys always buy a single chocolate ice cream, but girls buy a single chocolate ice
cream only with probability 1/2, and vanilla otherwise. What is the probability
that exactly 5 chocolate ice creams are sold between 2 and 4? [6 marks]
(d) If we know that exactly 3 chocolate ice creams were sold between 3 and 4, what
is the probability that there were exactly 6 customers between 2 and 4?
[8 marks]

[Please turn over]

MATH11029 Stochastic Modelling (MSc) 6

Solution: Let {Nt : t ≥ 0} denote the PP with rate t describing the number of
children arriving by time t.

(a) The 20 unordered arrival times Si are independent random variables on [0, 6],
and for each of them the probability
Rt P (Si ∈ [1, 2]) = (Λ(2) − Λ(1))/Λ(6) =
3/22.5 = 2/15, with Λ(t) = 0 λ(s)ds. Hence
 
20
P (N2 − N1 = 3|N6 = 20) = (2/15)3 (13/15)17 = 0.23725 . . .
3

(b) Due to the splitting property, girls arrive as {Gt : t ≥ 0}, a rate 2λ(t)/3 PP,
R 6 boys as {Bt : t ≥ 0}, a rate λ(t)/3 PP. Hence G6 − G5 is a Poisson
and
( 5 2λ(t)/3dt = 11/3) random variable, so

(11/3)5 −11/3
P (G6 − G5 = 5) = e = 0.141177 . . .
5!

(c) Due to the super-positioning property, the number of chocolate ice-cream sold
{C
R 4t : t ≥ 0} is a rate λ(t)/3 + λ(t)/3 = 2λ(t)/3 PP. Hence C4 − C2 is a Poisson
( 2 2λ(t)/3dt = 13/3) random variable, so

(13/3)5 −13/3
P (C4 − C2 = 5) = e = 0.167104 . . .
5!

(d) Customers either buy chocolate ice cream or vanilla. There were 3 customer
buying chocolate between 3 and 4, so the question is the probability that
there were an additional 3 customers who either bought vanilla between 3
and 4, or bought anything between 2 and 3. Here we used that vanilla and
chocolate buying customers arrive independently, and that the increments are
independent. These additional customers arrive as a Poisson process at rate
(
3 2≤t≤3
λ̃(t) =
t/3 3 ≤ t ≤ 4

so
R 4 their number between 2 and 4 is a Poisson random variable with parameter
2
λ̃(t) = 25/6. Hence the probability that there were exactly 3 additional
customers is
(25/6)3 −25/6
P (N4 − N2 = 6|C4 − C3 = 3) = e = 0.18692 . . .
3!
There are many other equivalent solutions.

[Please turn over]

MATH11029 Stochastic Modelling (MSc) 7

(4) In a shop there are two cashiers (A and B) with a single queue for them. Customers
arrive at the queue as a Poisson process with rate λ, and wait for the first available
cashier. If both cashiers are available, they pick one equally likely. Each cashier
finishes with a customer after an exponential waiting time, with parameters µa and
µb for cashier A and B, respectively. Assume that λ < µa + µb .

(a) Formulate a Markov chain model with state space S = {0, a, b, 2, 3, . . . }, where
a and b mean that only cashier A or only B is busy, and the numbers mean the
number of customers in the system. Give all the transition rates. [6 marks]
(b) Write down the detailed balance equations. [6 marks]
(c) Find the stationary probabilities of the process. [6 marks]
(d) When is the chain positive recurrent? [2 marks]
(e) Calculate the mean queue length in the stationary state (not counting people
being served)? [5 marks]

[Please turn over]

MATH11029 Stochastic Modelling (MSc) 8

Solution:

(a) Let Xt be the state of the system at time t. The transition rates between states
are
q0a = q0b = λ/2, qa2 = qb2 = qj,j+1 = λ for j ≥ 2

qa0 = q2b = µa , qb0 = q2a = µb , qj+1,j = µa + µb for j ≥ 2

with diagonal elements

q00 = −λ qaa = −(µa + λ) qb = −(µb + λ) qii = −(µa + µb + λ)

for i ≥ 2, and all other jump rates are zero.

(b) The detailed balance equations are
λ λ
π0 = πa µa , π0 = πb µb
2 2
πa λ = π2 µb , πb λ = π2 µa
πj λ = πj+1 (µa + µb ) for j ≥ 2

λ
(c) Applying the last equation recursively, we get πj = π2 ( µa +µb
)j−2 , which makes
sense only for λ < µa + µb . Also,
µb µa 2µa µb
πa = π2 , πb = π2 , π0 = π2
λ λ λ2
P
From πi = 1, we get
 −1
2µa µb µa + µb µa + µb
π2 = 2
+ +
λ λ µa + µb − λ
again, for λ < µa + µb .
(d) For λ < µa + µb all stationary probabilities are positive, hence the system is
positive recurrent.
(e) In state j with j ≥ 2 the queue length is j − 2, hence the mean queue length is
 j−2 X  j
X X λ λ
(j − 2)πj = (j − 2)π2 = π2 j
j≥2 j≥2
µa + µb j≥0
µa + µb

which can be summed for example as

X X 1 α
jαj = α∂α αj = α∂α =
1−α (1 − α)2
to finally obtain the mean queue length
 −2
λ λ
π2 1−
µa + µb µa + µb

[End of Paper]