MIDDLE EAST TECHNICAL UNIVERSITY

I E 3 6 1 - O P E R A T I O N A L R E S E A R C H III

HOMEWORK
#1
Onur YILMAZ

Instructor:
İSMAİL SERDAR BAKAL

November 2010, Ankara

Question 1

a.

In Question 1, two players, Aylin and Barış are playing Rock, Paper, Scissors with given
strategies and their choices depend only on the last played games and have no relation to the
history of the game, therefore being a discrete time stochastic process this system can be defined
as a Markov chain.

X t : Aylin’s choice of game for time t

Y t : Barış’s choice of game for time t

States can be defined as their choices from Rock(R), Paper(P) or Scissors(S) for Aylin
and Barış, respectively, for discrete times.

S = { X t Y t ∨X t : Aylin' s choice for time t , Y t : ış '

¯ s choice for timet } or equivalently

S = { PP , PR , PS , RP , RR , RS , SP , SR , SS }

Considering given strategies transition probability matrix (P) can be given as a table as
follows:

PP PR PS RP RR RS SP SR SS
PP 0.25 0 0.25 0 0 0 0.25 0 0.25
PR 0 0 0 0 0.25 0.25 0 0.25 0.25
PS 0 0 0 0 0 0 0 0 1
RP 0.5 0 0 0 0 0 0.5 0 0
RR 0 0 0 0.25 0.25 0 0.25 0.25 0
RS 0 0 0 0 0 0 0.5 0 0.5
SP 0.25 0.25 0 0 0 0 0.25 0.25 0
SR 0 0 0 0 0.5 0 0 0.5 0
SS 0 0 0 0 0 0 0 0.5 0.5

b.
 In this game the rules and player strategies do not change throughout the game as
mentioned in the question. From the transition probability matrix and state transition diagram,
which is shown below as Diagram 1-b, it can be investigated whether this process is ergodic or
not. From state diagram it can be shown that all states are communicative with each other. In
addition all states are recurrent, which means not having a transient property which imposes to
being reachable from a state but not being able to reach this state. Observing diagram and
probability matrix, it can be stated that all states are aperiodic and having these three properties
this Markov chain can be classified as ergodic which implies the effect of initial state diminishes
in the long term. For an ergodic Markov chain, for the long term we can use steady state
analysis.

Diagram 1-b: State transition diagram with related probabilities

 Using steady state analysis, in order to find stationary distribution we need to find π =
[ π 1 π 2 π 3 … π n ] equilibrium distribution vector. By using the theorem which states
n
π = πx P where ∑ π i = 1
i=1

and using Excel Solver and given probability matrix, πvector is found as it is:

PP PR PS RP RR RS SP SR SS
π1 π2 π3 π4 π5 π6 π7 π8 π9
π= 0.085261 0.036281 0.021315 0.055329 0.221315 0.00907 0.145125 0.313833 0.112472

In addition, since stationary distribution vector can be found without any error by using
theorem, it also shows that there exists stationary distribution of Markov chain for this process.

c.

In the long term situation of the game can be identified from stationary distribution. For
Aylin, according to the specifications of game; win, loss and draw situations are highlighted
below in π vector.

PP PR PS RP RR RS SP SR SS
π1 π2 π3 π4 π5 π6 π7 π8 π9
π = 0.085261 0.036281 0.021315 0.055329 0.221315 0.00907 0.145125 0.313833 0.112472
Draw Win Loss Loss Draw Win Win Loss Draw

Summing up the related probabilities we could find the ratio in the long turn:

P ( Draw ) = 0.085261 + 0.221315 + 0.112472 = 0,419048

P ( Win ) = 0.036281 + 0.00907 + 0.145125 = 0,190476

P ( Loss ) = 0.021315 + 0.055329 + 0.313833 = 0,390477

With rounding these probabilities, ratio of win:loss:ratio can be stated as 19:39:42.

 Question 2

a.

In the second question we are asked to analyze a reordering system consisting of three
processors. For each processor probability of being successful is given as:

P( P1 is successful ) = 0,8

P( P2 is successful ) = 0,64

P( P3 is successful ) = 0,512

For this process states can be defined as the order of processes at a discrete time.

X t : ordering of processes at time t

Using this state definition, state space can be constructed as following:

S = {P1P2P3, P1P3P2, P2P1P3, P2P3P1, P3P1P2, P3P2P1 }

In this question ordering activity does not take the history of past orderings into account.
It only considers the last order of processes; therefore this system can be identified as a Markov
chain.

Considering properties of the system given in the question, transition probability matrix
can be calculated. For instance, transition of the system from state P1P2P3 to state P1P2P3,
which is the first element of matrix, is calculated in this manner:

P P 1 P 2 P 3 , P 1 P 2 P 3 = P( P1 is successful ) + P( P1 is failed ) * P( P2 is failed ) * P( P3 is failed )

P P 1 P 2 P 3 , P 1 P 2 P 3 = 0,8 + 0,2 * 0,36 * 0,488
P P 1 P 2 P 3 , P 1 P 2 P 3 = 0,835136
With this approach all transitions are calculated and transition probability matrix (P) is given as a
table below:

P1P2P3 P1P3P2 P2P1P3 P2P3P1 P3P1P2 P3P2P1

P1P2P3 0.835136 0.036864 0.128 0 0 0
P1P3P2 0.062464 0.835136 0 0 0.1024 0
P2P1P3 0.288 0 0.675136 0.036864 0 0
P2P3P1 0 0 0.140544 0.675136 0 0.18432
P3P1P2 0 0.3904 0 0 0.547136 0.062464
P3P2P1 0 0 0 0.31232 0.140544 0.547136

b.

In order to find long-run probabilities we need to use steady state analysis for this system.
Since the system’s properties and probabilities of orderings do not change over the time, in
addition being an ergodic Markov chain this system can be analyzed for long run.

Using steady state analysis, in order to find stationary distribution we need to find π =
[ π 1 π 2 π 3 … π n ] equilibrium distribution vector. By using the theorem which states
n
π = πx P where ∑ π i = 1
i=1

and using Excel Solver with given probability matrix (P), πvector is found as it is:

P1P2P3 P1P3P2 P2P1P3 P2P3P1 P3P1P2 P3P1P2

π1 π2 π3 π4 π5 π6
π= 0,433655 0,255927 0,192736 0,050554 0,067129 0,029835

In the long run, probability of having Processor 1 closer to the front than Processer 3 can
be determined from stationary distribution vectorπ. As highlighted in the below, these occur
when in the long run this system is in the state of P1P2P3, P1P3P2 and P2P1P3.
 P1P2P3 P1P3P2 P2P1P3 P2P3P1 P3P1P2 P3P1P2

π1 π2 π3 π4 π5 π6
π= 0,433655 0,255927 0,192736 0,050554 0,067129 0,029835

Summing up the related probabilities will give the total probability of having Processor 1 closer
to the front than Processor 3 in the long-run. As calculated below, this probability is 0,882318.

P ( P1 is closer to front than P3 in the long run ) = π 1 + π 2 + π 3

= 0,433655 + 0,255927 + 0,192736
= 0,882318
 Question 3

In the third question we need to consider a periodic review inventory system with (s=1,
S=4) and a given periodic demand distribution with backlogging restriction. It is assumed that
control of the inventory is done at the end of each period and next period starts with
replenishment if necessary.

States can be defined as the level of inventory at the end of time periods:

X t : number of items in the inventory at the end of time period t

Using this state definition, state space can be constructed as following:

S = {0, 1, 2, 3, 4 }

Using the demand distribution and given periodic inventory system, transition probability
matrix for this process can be given as:

  0 1 2 3 4
0 0 0 0,2 0,3 0,5
1 0,5 0,5 0 0 0
P= 2 0,2 0,3 0,5 0 0
3 0 0,2 0,3 0,5 0
4 0 0 0,2 0,3 0,5

a.

Since this system is an ergodic Markov chain ensuring that all states in the system has the
three properties of being ergodic which are being recurrent, aperiodic and communicative, this
system can be analyzed for steady state probabilities.

Using steady state analysis, in order to find stationary distribution we need to find π =
[ π 1 π 2 π 3 … π n ] equilibrium distribution vector. By using the theorem which states
 n
π = πx P where ∑ π i = 1
i=1

and using Excel Solver with given probability matrix, πvector is found as following:

S= 0 1 2 3 4

π1 π2 π3 π4 π5
π= 0,163613 0,227749 0,248691 0,196335 0,163613

Long-run mean inventory level can be found by

4
E ( inventory level )=∑ π i∗inventory level
i=0

¿ 0 * π 1+ 1 * π 2 + 2 * π 3 + 3 * π 4 + 4 * π 5
= 0 * 0,163613 + 1* 0,227749 + 2 * 0,248691 + 3 * 0,196335 + 4*0,163613
= 1,968588

Considering long-run probabilities and related level of inventories, mean inventory level in the
long run can be stated as 1,968588.

b.

In this inventory system, demand is lost whenever the inventory level is less than the
demand of related period. Considering this inventory system, since demand is never greater than
2, there is no loss when the inventory level is 4,3 and 2. In addition, when inventory level is zero,
replenishment occurs and in the next period inventory level becomes 4, where no demand can be
lost.

Therefore, for investigating lost demand only inventory level of 1 must be considered. In
addition when the inventory level is one, in order to witness a lost demand, demand of related
period must be more than 1, in this concept only 2. In the long run we could identify the fraction
of having inventory level of 1 from the steady-state distribution. As mentioned in the previous
part, probability of having inventory level of 1 in the long run is π 1 = 0,163613. Besides, demand
distribution function gives the fraction of having demand level of 2 for any period, which is 0.2.
Since these events are independent of each other, probability of occurring together can be
calculated by multiplying them.

P ( D t +1 = 2 and X t = 1) = P ( D t +1 = 2 ) * P ( X t = 1)
= P ( D t +1 = 2 ) * π 2
= 0,2 * 0,227749
= 0.04555

As calculated above, having a lost demand for this system occurs with the probability of 4,55 %.

c.

In order to calculate expected long-run average daily profit, we need to consider the long-
run steady state probabilities of inventory levels and demand distribution for different cases.

Daily Profit = Daily Revenues – Daily Costs

We need the expected level of demand for any period. As calculated below it is 0,7 items
per day.

2
Expected demand for a period is E ( D ) = ∑ pk * k
k=0

E ( D ) = 0,5 * 0 + 0,3 * 1 + 0,2 * 2 = 0,7 per day.

For the first case, if there is no lost demand, only selling price and cost of keeping item in
the inventory should be considered for related periods and their probabilities:
 = π 5 * ( 0,7 v - 4k ) + π 4 * ( 0,7 v - 3k ) + π 3 ( 0,7v - 2k)
= 0,163613( 0,7 v - 4k ) + 0,196335 * (0,7 v - 3k ) + 0,248691 * ( 0,7v - 2k)
= 0,426047 * v - 1,740839 * k

For the second case, if we have 1 item in the inventory we sell zero or one item with
related demand distribution or we can have lost demand.

= π 2 * ( 0,3 v - k ) - π 2 * 0,2 * m
= 0,227749 * ( 0,3 v - k ) - 0,227749 * 0,2 * m
= 0,068325 * v - 0,227749 * k - 0,04555 * m

Finally, if we had zero items in the inventory at the end of the period we will have
replenishment. In addition inventory holding cost for items in replenishment are not taken into
account because it is assumed that the checking is done at the end of the period; however new
items are shipped at the beginning of the next period, explaining why there is no inventory
holding cost for these items.

= - π1 * o
= - 0,163613 * o

Combining these three cases long-run average daily profit can be calculated as shown below:

= 0,426047 * v - 1,740839 * k -0,163613 * o + 0,068325 * v - 0,227749 * k - 0,04555 * m

Average Daily Profit = 0,494372 v - 0,163613 o - 1,968588 k - 0,04555 m

d.

In order to find the number of days between successive replenishments, we need to

consider long-run probabilities where inventory level is strictly less than 1, which is π 1 in this
system.

Using mean first passage times formula for m ii = 1 / π i :

 m 11 = 1 / π 1
= 1 / 0,163613
= 6,111984

As calculated above, replenishments are expected to occur between nearly 6.11 days.

Question 4

a.

In the fourth question analyzing of a tennis player’s performance with regard to his
morale is conducted.

For any discrete time, results of his two last games will be used for analyzing this system.

X t : result of the game at time t

States can be defined as the result of last two games using “L” for losing, “W” for
winning the related game.

S = { X t X t −1 ∨X t ∈ {W , L } ∧ X t −1 ∈ { W , L } ∧t ≥ 1 } or equivalently

S = { WL , WW ,≪, LW }

It is also given in the question that X 1 X 0 = WW.

Considering given probabilities given in the question transition probability matrix (P) can
be given as a table as follows:

WL WW LL LW
WL 0 0 0,6 0,4
WW 0,3 0,7 0 0
LL 0 0 0,7 0,3
LW 0,4 0,6 0 0
 Since the probabilities do not change over the time and they do not consider the whole
past history of the games, in other words since they only consider the last state of the system,
performance of this player can be identified as a Markov chain.

b.

As given in the question, Mert has won his first two games, which can also be stated as
his first state is WW according to state definition.

In order to find a future loss followed by another loss, path from WW to LL could be
investigated. For the first three paths from WW to LL are given as following with related
probability calculations:

P = 0.3 * 0.6

P = 0.7 * 0.3 * 0.6

P = 0.7 * 0.7 * 0.3 * 0.6

Intuitively it can be said that this path has the probability of form:
∞

∑ 0,7 i∗0,3∗0,6
i=0

∞
i
= 0,3∗0,6 ∑ 0,7
i=0
 1
= 0,3 * 0,6 * by using geometric series
1−0,7
= 0,6

As calculated above, probability of having the first loss followed by another loss is 0,6.

c.

In order to find a probability mass function for L which is defined as the number of
games played until the first loss, we need to consider available paths from initial state, WW, to
occurring of first loss. In the calculation of L, the first two games which Mert has already won
are not considered. For the rest of the system number of the games until the first loss can be
identified from paths as following:

L=1 P ( L=1 ) = 0,3

L =2 P ( L=2 ) = 0,7 * 03

L =3 P ( L=3 ) = 0,7 * 0,7 * 0,3

L =4 P ( L=4 ) = 0,7 * 0,7 * 0,7 * 0,3

Since it can be derived from these initial situations, probability mass function of L can be
given as:
 P( L ) = P ( X t = WL | X t −1 = WW ) * [ P ( X t = WW | X t −1 = WW ) ] L−1

P ( L ) = (0,3) * (0,7) L−1 where L > 0 and integer

d.

Since this system is an ergodic Markov chain, steady state analysis can be used for
finding steady-state probabilities. Using steady state analysis, in order to find stationary
distributions we need to find π = [ π 1 π 2 π 3 … π n ] equilibrium distribution vector. By using the
theorem which states

n
π = πx P where ∑ π i = 1
i=1

and using Excel Solver with given probability matrix, πvector is found as it is:

S= WL WW LL LW

π1 π2 π3 π4
π= 0,166667 0,333334 0,333334 0,166667

e.

In this part of the question, it is assumed that in the 2000 th game this system reaches
stability where steady-state analysis and its results can be used. In other words these two
consecutive games are assumed that they can be investigated as a state in the long-run.

P ( X 2000 = W | X 2000 = X 2001 ) is need to be calculated for this reason. For 2000 th and 2001st
game this system could be thought in WW or in LL for the long-run. Probability of being in
states WW or LL in the long run can be gathered from steady state distribution vector.
 P ( X 2000 = W | X 2000 = X 2001 ) =

P ( X 2000=W ∧ X 2000 =X 2001 =W ) + P ( X 2000=W ∧ X 2000 =X 2001 =L )

=
P ( X 2000 =X 2001 =L ) + P ( X 2000 =X 2001 =W )

π 2+ 0 0,333334
= = = 0,5
π 2+ π 3 0,333334+0,333334

As calculated above, probability of Mert’s winning 2000th game given that the 2000th and
2001st games have same results is 0,5.

f.

In this question we are asked to calculate the expected number of games until first
occurrence of two consecutive losses. Since the initial state of this system is Win-Win, expected
number of transitions before we first reach Loss-Loss state can be found by using mean first
passage times formula.

As defined in the question E[ D ] = m WW −¿ which is calculated as following:

m WW −¿= 1 + ∑ PWW −k∗m k−¿ by the definition of mean passage time formula
k ≠≪¿ ¿

mWW −¿= 1 + PWW−WW ∗m WW−¿ + PWW−WL∗mWL−¿ + PWW− LW∗m LW −¿ ¿

¿

m WW −¿ = 1 + 0,7 * m WW −¿ + 0,3 * m WL−¿ + 0

0,3 m WW −¿ = 1 +0,3 * m WL−¿
mWW −¿ = 10/3 + m WL−¿ 1st equation

m WL−¿ must be calculated since it is unknown in 1st equation.

m WL−¿ = 1 + ∑ PWL−k ∗m k−¿

k ≠≪¿ ¿

m WL−¿= 1 + PWL−WW ∗mWW −¿ + PWL−WL∗mWL−¿ + PWL−LW ∗m LW −¿ ¿

¿

m WL−¿= 1 + 0 + 0 + 0,4 *m LW −¿ ¿ 2nd equation

¿
 Combining 1st and 2nd equations:

mWW −¿ = 10/3 + 1 + 0,4 * mLW −¿ 3rd equation

m LW −¿ must be calculated since it is unknown in 3rd equation.

m LW −¿ ¿ = 1 + ∑ PLW −k∗mk−¿ ¿
¿ k ≠≪¿ ¿ ¿

m LW −¿ ¿ = 1 + P LW −WW∗mWW −¿ + P LW −WL∗mWL−¿ + P LW −LW ∗mLW −¿ ¿

¿ ¿

m LW −¿ ¿ = 1 + 0,6 * m
WW −¿ + 0,4 * m WL−¿ + 0 4th equation
¿

Combining 1st, 3rd and 4th equation and renaming m values for readability:

m LW −¿ ¿ = Z mWW −¿ = X m WL−¿ = Y
¿
Z = 1 + 0,6 X + 0,4 Y
m LW −¿ ¿ = 1 + 0,6 * m
WW −¿ + 0,4 * m WL−¿
¿ X= 10/3 + Y
m WW −¿ = 10/3 + * m WL−¿ Y = 1 + 0,4 Z
m WL−¿= 1 + 0,4 *m LW −¿ ¿
¿

Solving these equations simultaneously:

Z = 1 + 0,6 ( 10/3 + Y) + 0,4 Y

Z=3+Y
Y = 1 + 1,2 + 0,4 Y
0,6 Y = 2,2
Y = 11/3 and X =7

As calculated above m WW −¿ = 7, implies that starting with initial state WW, expected
number of games until the first occurrence of state LL is 7 games.
 g.

In this part of the question total numbers of the games until Mert will end his career is
asked. According to his press conference he will end his career when he faces three consecutive
losses. Since three conssequitive games are considered in this question there is need for new state
definition which we can investigate mean first passage times for three games.

Defining the states for three games is given as:

X t : result of the game at time t

State space can be defined as the result of last three games using “L” for losing, “W” for
winning the related game.

S = { X t X t −1 X t −2∨X t ∈ { W , L } ∧ X t−1 ∈ { W , L } ∧ X t−2 ∈ { W , L } ∧ t ≥1 }

or equivalently S = { WWW ,WWL ,WLW , WLL , LLL , LLW , LWL , LWW }

Considering given probabilities given in the question transition probability matrix (P) can
be given as a table as follows:

WWW WWL WLW WLL LLL LLW LWL LWW

WWW 0.7 0.3 0 0 0 0 0 0
WWL 0 0 0.4 0.6 0 0 0 0
WLW 0 0 0 0 0 0 0.4 0.6
WLL 0 0 0 0 0.7 0.3 0 0
LLL 0 0 0 0 0.7 0.3 0 0
LLW 0 0 0 0 0 0 0.4 0.6
LWL 0 0 0.4 0.6 0 0 0 0
LWW 0.7 0.3 0 0 0 0 0 0

With this transition matrix mean first passage times can be calculated for states from
WWW to LLL and from WWL to LLL. By formulation and using related probabilities expected
number of games can be found as:

= P¿
 = 0,7∗m WWW − LLL + 0,3∗m WWW −LLL

Since calculating mean first passage time values for this transition probability matrix takes
too long further calculations cannot be given.