Lesson 4: Stationary stochastic processes

Umberto Triacca

Dipartimento di Ingegneria e Scienze dell’Informazione e Matematica

Università dell’Aquila,
umberto.triacca@univaq.it

Umberto Triacca Lesson 4: Stationary stochastic processes

Stationary stochastic processes

Stationarity is a rather intuitive concept, it means that the

statistical properties of the process do not change over time.

Umberto Triacca Lesson 4: Stationary stochastic processes

Stationary stochastic processes

There are two important forms of stationarity:

1 strong stationarity;
2 weak stationarity.

Umberto Triacca Lesson 4: Stationary stochastic processes

Stationary stochastic processes

Strong stationarity concerns the shift-invariance (in time) of its

finite-dimensional distributions.

Weak stationarity only concerns the shift-invariance (in time) of

first and second moments of a process.

Umberto Triacca Lesson 4: Stationary stochastic processes

Strongly stationary stochastic processes

Definition. The process {xt ; t ∈ Z} is strongly stationary if

Ft1 +k,t2 +k,··· ,ts +k (b1 , b2 , · · · , bs ) = Ft1 ,t2 ,··· ,ts (b1 , b2 , · · · , bs )

for any finite set of indices {t1 , t2 , · · · , ts } ⊂ Z with s ∈ Z+ , and

any k ∈ Z.
Thus the process {xt ; t ∈ Z} is strongly stationary if the joint
distibution function of the vector (xt1 +k , xt2 +k , ..., xts +k ) is equal
with the one of (xt1 , xt2 , ..., xts ) for for any finite set of indices
{t1 , t2 , · · · , ts } ⊂ Z with s ∈ Z+ , and any k ∈ Z.

Umberto Triacca Lesson 4: Stationary stochastic processes

Strongly stationary stochastic processes

The meaning of the strongly stationarity is that the

distribution of a number of random variables of the
stochastic process is the same as we shift them along the
time index axis.

Umberto Triacca Lesson 4: Stationary stochastic processes

Strongly stationary stochastic processes

If {xt ; t ∈ Z} is a strongly stationary process, then

x1 , x2 , x3 , ...

have the same distribution function.

(x1 , x3 ), (x5 , x7 ), (x9 , x11 ), .....

have the same joint distribution function and further

(x1 , x3 , x5 ), (x7 , x9 , x11 ), (x13 , x15 , x17 ), ...

must have the same joint distribution function, and so on.

Umberto Triacca Lesson 4: Stationary stochastic processes

Strongly stationary stochastic processes

If the process is {xt ; t ∈ Z} is strongly stationary, then the joint

probability distribution function of (xt1 , xt2 , ..., xts ) is invariant
under translation.

Umberto Triacca Lesson 4: Stationary stochastic processes

iid process

An iid process is a strongly stationary process. This follows almost

immediate from the definition.

Since the random variables xt1 +k , xt2 +k , ..., xts +k are iid, we have
that

Ft1 +k,t2 +k,··· ,ts +k (b1 , b2 , · · · , bs ) = F (b1 )F (b2 ) · · · F (bs )

On the other hand, also the random variables xt1 , xt2 , ..., xts are iid
and hence

Ft1 ,t2 ,··· ,ts (b1 , b2 , · · · , bs ) = F (b1 )F (b2 ) · · · F (bs ).

We can conclude that

Ft1 +k,t2 +k,··· ,ts +k (b1 , b2 , · · · , bs ) = Ft1 ,t2 ,··· ,ts (b1 , b2 , · · · , bs )

Umberto Triacca Lesson 4: Stationary stochastic processes

iid process

Remark. Let {xt ; t ∈ Z} be an iid process. We have that the

conditional distribution of xT +h given values of (x1 , ..., xT ) is

P(xT +h ≤ b|x1 , ..., xT ) = P(xT +h ≤ b)

So the knowledge of the past has no value for predicting the

future. An iid process is unpredictable.

Example. Under efficient capital market hypothesis, the stock

price change is an iid process. This means that the stock price
change is unpredictable from previous stock price changes.

Umberto Triacca Lesson 4: Stationary stochastic processes

Strongly stationary stochastic processes

Consider the discrete stochastic process

{xt ; t ∈ N}

where xt = A, with A ∼ U (3, 7) (A is uniformly distributed on the

interval [3, 7]).

This process is of course strongly stationary.

Why?

Umberto Triacca Lesson 4: Stationary stochastic processes

Strongly stationary stochastic processes

Consider the discrete stochastic process

{xt ; t ∈ N}

where xt = tA, with A ∼ U (3, 7)

This process is not strongly stationary

Why?

Umberto Triacca Lesson 4: Stationary stochastic processes

Weakly stationary stochastic processes

If the second moment of xt is finite for all t, then the mean E (xt ),
the variance var(xt ) = E [(xt − E (xt ))2 ] = E (xt2 ) − (E (xt ))2 and
the covariance cov(xt1 , xt2 ) = E [(xt1 − E (xt1 ))(xt2 − E (xt2 ))] are
finite for all t, t1 and t2 .

Why?

Hint: Use the Cauchy-Schwarz inequality

|cov(x, y )|2 ≤ var(x)var(y )

Umberto Triacca Lesson 4: Stationary stochastic processes

Weakly stationary stochastic processes

Definition The process {xt ; t ∈ Z} is weakly stationary, or

covariance-stationary if
1 the second moment of xt is finite for all t, that is E |xt |2 < ∞
for all t
2 the first moment of xt is independent of t, that is
E (xt ) = µ ∀t
3 the cross moment E (xt1 xt2 ) depends only on t1 − t2 , that is
cov(xt1 , xt2 ) = cov(xt1 +h , xt2 +h ) ∀t1 , t2 , h

Umberto Triacca Lesson 4: Stationary stochastic processes

Weakly stationary stochastic processes

Thus a stochastic process is covariance-stationary if

1 it has the same mean value, µ, at all time points;
2 it has the same variance, γ0 , at all time points; and
3 the covariance between the values at any two time points,
t, t − k, depend only on k, the difference between the two
times, and not on the location of the points along the time
axis.

Umberto Triacca Lesson 4: Stationary stochastic processes

Weakly stationary stochastic processes
An important example of covariance-stochastic process is the
so-called white noise process.
Definition . A stochastic process {ut ; t ∈ Z} in which the random
variables ut , t = 0 ± 1, ±2... are such that
1 E (u ) = 0 ∀t
t
2 Var(u ) = σ 2 < ∞ ∀t
t u
t t−k ) = 0 ∀t, ∀k
3 Cov(u , u

is called white noise with mean 0 and variance σu2 , written

ut ∼ WN(0, σu2 ).
First condition establishes that the expectation is always constant
and equal to zero. Second condition establishes that variance is
constant. Third condition establishes that the variables of the
process are uncorrelated for all lags.
If the random variables ut are independently and identically
distributed with mean 0 and variance σu2 then we will write
ut ∼ IID(0, σu2 )
Umberto Triacca Lesson 4: Stationary stochastic processes
White Noise process

Figure shows a possible realization of an IID(0,1) process.

Figure : A realization of an IID(0,1).

Umberto Triacca Lesson 4: Stationary stochastic processes

Random Walk process

An important example of weakly non-stationary stochastic

processes is the following.
Let
{yt ; t = 0, 1, 2, ...}
be a stochastic processs where y0 = δ < ∞ and yt = yt−1 + ut for
t = 1,2,..., with ut ∼ WN(0, σu2 ).

This process is called random walk.

Umberto Triacca Lesson 4: Stationary stochastic processes

Random Walk process

The mean of yt is given by

E (yt ) = δ

and its variance is

Var(yt ) = tσu2
Thus a random walk is not weakly stationary process.

Umberto Triacca Lesson 4: Stationary stochastic processes

Random Walk process

Figure shows a possible realization of a random walk.

Figure : A realization of a random walk.

Umberto Triacca Lesson 4: Stationary stochastic processes

Relation between strong and weak Stationarity

First note that finite second moments are not assumed in the
definition of strong stationarity, therefore, strong stationarity does
not necessarily imply weak stationarity.

For example, an iid process with standard Cauchy distribution is

strictly stationary but not weak stationary because the second
moment of the process is not finite.

Umberto Triacca Lesson 4: Stationary stochastic processes

Relation between strong and weak Stationarity

If the process {xt ; t ∈ Z} is strongly stationary and has finite

second moment, then {xt ; t ∈ Z} is weakly stationary.
PROOF. If the process {xt ; t ∈ Z} is strongly stationary, then

..., x−1 , x0 , x1 , ...

have the same distribution function and

(xt1 , xt2 ) and (xt1 +h , xt2 +h )

have the same joint distribution function for all t1 , and t2 and h.
Because, by hypothesis, the process {xt ; t ∈ Z} has finite second
moment, this implies that
E (xt ) = µ ∀t
cov(xt1 , xt2 ) = cov(xt1 +h , xt2 +h ) ∀t1 , t2 , h

Umberto Triacca Lesson 4: Stationary stochastic processes

Relation between strong and weak Stationarity

Of course a weakly stationary process is not necessarily strongly

stationary.

weak stationarity ; strong stationarity

Umberto Triacca Lesson 4: Stationary stochastic processes

Relation between strong and weak Stationarity

Here we give an example of a weakly stationary stochastic process

which is not strictly stationary.
Let {xt ; t ∈ Z} be a stochastic process defined by
(
ut if t is even
xt = √1 (u 2 − 1) if t is odd
2 t

where ut ∼ iidN(0, 1).

This process is weakly stationary but it is not strictly stationary.

Umberto Triacca Lesson 4: Stationary stochastic processes

Relation between strong and weak Stationarity

We have
(
E (ut ) = 0 if t is even
E (xt ) = √1 E (u 2 − 1) =
2 t 0 if t is odd

and 
var(ut ) = 1 if t is even
var(xt ) = 1 2
2 var(ut−1 ) = 1 if t is odd
Further, because xt and xt−k are independent random variables, we
have
cov(xt , xt−k ) = 0 ∀k
Thus, the process xt is weakly stationary. In particular,
xt ∼ WN(0, 1).

Umberto Triacca Lesson 4: Stationary stochastic processes

Relation between strong and weak Stationarity
Now, we note that

P(xt ≤ 0) = P(ut ≤ 0) = 0.5 for t even

and
 
1
P (xt ≤ 0) = P √ (ut2 − 1) ≤ 0
2
2


= P ut−1 ≤ 1
= P (|ut−1 | ≤ 1)
= P (−1 ≤ ut−1 ≤ 1)
= 0.6826 for t odd

Hence the random variables of the process are not identically

distributed. This implies that the process is not strongly stationary
Umberto Triacca Lesson 4: Stationary stochastic processes
Relation between strong and weak Stationarity

There is one important case however in which weak stationarity

implies strong stationarity.

If {xt ; t ∈ Z} is a weakly stationary Gaussian stochastic process,

then {xt ; t ∈ Z} is strongly stationary.

Why?

Umberto Triacca Lesson 4: Stationary stochastic processes

Relation between strong and weak Stationarity

Let {xt ; t ∈ Z} be a Gaussian stochastic process. Introducing the

vector b = (b1 , b2 , ..., bs )0 ∈ Rs , the multidimensional density
function of the vector (xt1 , xt2 , ..., xts ) is
 
1 1
ft1 ,t2 ,...,ts (b) = p exp − (b − µ )0Σ −1 (b − µ ) .
(2π)s det (Σ
(Σ) 2

where µ = (E (xt1 ), E (xt2 ), ..., E (ts ))0 and Σ = Cov(xti , xtj ) .

 

We note that a multivariate Gaussian distribution is fully

characterized by its first two moments.

Umberto Triacca Lesson 4: Stationary stochastic processes

Relation between strong and weak Stationarity

Let {xt ; t ∈ Z} be a Gaussian stochastic process. Assume that the

process is weakly stationary. If the process i weakly stationary,
then
1 E (xt ) = µ ∀t
2 Var(xt ) = γ0 < ∞ ∀t
3 Cov(xt1 +k , xt2 +k ) = Cov(xt1 , xt2 ) ∀t1 , t2 , ∀k
and hence
ft1 +k,t2 +k,...,ts +k (b) = ft1 ,t2 ,...,ts (b)
It follows that the joint distibution function of the vector
(xt1 +k , xt2 +k , ..., xts +k ) is equal with the one of (xt1 , xt2 , ..., xts ) for
for any finite set of indices {t1 , t2 , · · · , ts } ⊂ Z with s ∈ Z+ , and
any k ∈ Z. This implies that the process {xt ; t ∈ Z} is strongly
stationary.

Umberto Triacca Lesson 4: Stationary stochastic processes

Relation between strong and weak Stationarity

Conversely, if {xt ; t ∈ Z} is a Gaussian strongly stationary

stochastic process, then it is weakly stationary because it has finite
variance.
Thus we can conclude that in the case of Gaussian stochastic
process, the two definitions of stationarity are equivalent.

Umberto Triacca Lesson 4: Stationary stochastic processes

White Noise process

We note that a white noise process is not necessarily strongly

stationary. Let w be a random variable uniformly distributed in the
interval (0; 2π). We consider the process {Zt ; t = 1, 2, ...} defined
by
Zt = cos(tw ) t = 1, 2, ...
We have that
1 E (Zt ) = 0 ∀t
1
2 Var(Zt ) = 2 ∀t
3 Cov(Zt , Zt−k ) = 0 ∀t, ∀k
Thus Zt ∼ WN(0, .5). However, it can be shown that is not
strongly stationary.

Umberto Triacca Lesson 4: Stationary stochastic processes