Time Series and System

Analysis with Applications

Chapter 2: Autoregressive Moving Average

Models

Outline

Simple linear regression

Multiple linear regression

Simple Linear Regression

Examine the relationship between a quantitative
outcome(y)
with a single quantitative variable(x).
y=the dependence or outcome or response
variable
x=the independence or predictor or explanatory
variable
Simple means only a single explanatory
variable(x)

Simple linear regression model

We have observation data (xi , yi), where i =1,2,3 n.
For, linear regression model:
The dependence of y is a straight line as function of x
plus the an error

yi 0 1 x i

term or residual.

Where
is the intercept,
the error or residual

i
is the slope and

is

Simple linear regression model

We can not find the exact line but we can the estimated
one.
Thus, we have to find the best fitted line that minimizes
the sum of the error or residuals by using least square
method: n

(
y

y
)(
x

x
)

i
i

0 y 1 x
i 1
1

where

2
(
x

x
)
i

i 1

x
i 1

y
i 1

Simple linear regression model

Remember that the line always go through means value
()
For example: we have data:
x
y

10

0.4505 1.0838 2.229 3.9133 4.1524 5.8258 6.5383 7.9961 8.4427 9.4427 10.1067

1 0.9983

0 0.4463

y 0.4463 0.9983 x

Simple linear regression model

The estimated line

y 0.4463 0.9983 x

Simple linear regression model

subtracted from y is denoted as . Similarly,
Thus, the linear regression line become
Yi 1 X i i

For estimation

Yi 1 X i

i ~ NID(0, 2 )

i 1, 2, 3, ..., n
n

Y X
i 1
n

X
i 1

2
i

^ 2

(Y
i 1

Xi)

Residentual Sum of Squares

Number of Residuals

Simple linear regression model

Observation=Prediction +Error
Yi 1 X i i

Y i

Yi for
For example, 95% probability limit
1 value
known

of
and
, are
Y 1.96

if we assume

Simple linear regression model

For example: we have data,
x

By using least square method we can determine

0.5926

0.2815

Simple linear regression model

The plot of the model
With 95% probability limits
for observation:

Y 1.96 0.5926 X 1.04

Y 0.5926 X

Multiple linear regression model

The Multiple regression means estimation of dependent
variables based on multi-independent variables
Assume we have n observation data,

where

i 1,2,..., n

i ~ NID(0, 2 )

Multiple linear regression model

In term of the estimation, we can calculate

by using the
^
^

least square method:

1 , 2 ... k

=(XX)-1 XY

Multiple linear regression model

The standard variation
n

^ 2

2
(
y

...

x
)
i 1 i1 2 i 2
k ik
i 1

Residentual Sum of Squares

Number of Residuals

The model of multi-linear regression is:

y i 1 x1 2 x2 ... k xk

Multiple linear regression model

With 95% probability limits for the true value :

y i 1.96
For the special case k = 2:
x11
x
X 21

xn1

x12
x22

xn 2

X'X

2
i1

xi1 xi 2
i

yi

i1 i

X'Y

2
x
i2

i1 i 2

i
n

i2