1 - Autoregressive Models
Printer-friendly version
A time series is a sequence of measurements of the same variable(s) made over time. Usually the
measurements are made at evenly spaced times - for example, monthly or yearly. Let us first consider
the problem in which we have a y-variable measured as a time series. As an example, we might
have y a measure of global temperature, with measurements observed each year. To emphasize that
we have measured values over time, we use "t" as a subscript rather than the usual "i,"
i.e., ytyt means yy measured in time period tt. An autoregressive model is when a value from a time
series is regressed on previous values from that same time series. for example, ytyt on yt−1yt−1:
yt=β0+β1yt−1+ϵt.yt=β0+β1yt−1+ϵt.
In this regression model, the response variable in the previous time period has become the predictor
and the errors have our usual assumptions about errors in a simple linear regression model.
The order of an autoregression is the number of immediately preceding values in the series that are
used to predict the value at the present time. So, the preceding model is a first-order autoregression,
written as AR(1).
If we want to predict yy this year (ytyt) using measurements of global temperature in the previous
two years (yt−1,yt−2yt−1,yt−2), then the autoregressive model for doing so would be:
yt=β0+β1yt−1+β2yt−2+ϵt.yt=β0+β1yt−1+β2yt−2+ϵt.
This model is a second-order autoregression, written as AR(2), since the value at time tt is predicted
from the values at times t−1t−1 and t−2t−2. More generally, a kthkth-order autoregression, written as
AR(k), is a multiple linear regression in which the value of the series at any time t is a (linear)
function of the values at times t−1,t−2,…,t−kt−1,t−2,…,t−k.
Autocorrelation and Partial Autocorrelation
The coefficient of correlation between two values in a time series is called the autocorrelation
function (ACF) For example the ACF for a time series ytyt is given by:
Corr(yt,yt−k).Corr(yt,yt−k).
This value of k is the time gap being considered and is called the lag. A lag 1 autocorrelation
(i.e., k = 1 in the above) is the correlation between values that are one time period apart. More
generally, a lag k autocorrelation is the correlation between values that are k time periods apart.
The ACF is a way to measure the linear relationship between an observation at time t and the
observations at previous times. If we assume an AR(k) model, then we may wish to only measure the
association between ytyt and yt−kyt−k and filter out the linear influence of the random variables that
lie in between (i.e., yt−1,yt−2,…,yt−(k−1)yt−1,yt−2,…,yt−(k−1)), which requires a transformation on the
time series. Then by calculating the correlation of the transformed time series we obtain the partial
autocorrelation function (PACF).
The PACF is most useful for identifying the order of an autoregressive model. Specifically, sample
partial autocorrelations that are significantly different from 0 indicate lagged terms of yy that are
useful predictors of ytyt. To help differentiate between ACF and PACF, think of them as analogues
to R2R2 and partial R2R2 values as discussed previously.
Graphical approaches to assessing the lag of an autoregressive model include looking at the ACF and
PACF values versus the lag. In a plot of ACF versus the lag, if you see large ACF values and a non-
random pattern, then likely the values are serially correlated. In a plot of PACF versus the lag, the
pattern will usually appear random, but large PACF values at a given lag indicate this value as a
possible choice for the order of an autoregressive model. It is important that the choice of the order
makes sense. For example, suppose you have blood pressure readings for every day over the past two
years. You may find that an AR(1) or AR(2) model is appropriate for modeling blood pressure.
However, the PACF may indicate a large partial autocorrelation value at a lag of 17, but such a large
order for an autoregressive model likely does not make much sense.
Consecutive values appear to follow one another fairly closely, suggesting an autoregression model
could be appropriate. We next look at a plot of partial autocorrelations for the data:
To obtain this in Minitab select Stat > Time Series > Partial Autocorrelation. Here we notice that
there is a significant spike at a lag of 1 and much lower spikes for the subsequent lags. Thus, an
AR(1) model would likely be feasible for this data set.
Approximate bounds can also be constructed (as given by the red lines in the plot above) for this plot
to aid in determining large values. Approximate (1−α)×100%(1−α)×100% significance bounds are
given by ±z1−α/2/√ n ±z1−α/2/n. Values lying outside of either of these bounds are indicative of an
autoregressive process.
We can next create a lag-1 price variable and consider a scatterplot of price versus this lag-1 variable:
There appears to be a moderate linear pattern, suggesting that the first-order autoregression model
yt=β0+β1yt−1+ϵtyt=β0+β1yt−1+ϵt
could be useful.
The plot below gives a plot of the PACF (partial autocorrelation function), which can be interpreted
to mean that a third-order autoregression may be warranted since there are notable partial
autocorrelations for lags 1 and 3.
The next step is to do a multiple linear regression with number of quakes as the response variable and
lag-1, lag-2, and lag-3 quakes as the predictor variables. (In Minitab, we used Stat >> Time Series
>> Lag to create the lag variables.) In the results below we see that the lag-3 predictor is significant
at the 0.05 level (and the lag-1 predictor p-value is also relatively small).
14.2 - Regression with Autoregressive Errors
Next, let us consider the problem in which we have a y-variable and x-variables all measured as a
time series. As an example, we might have yas the monthly highway accidents on an interstate
highway and x as the monthly amount of travel on the interstate, with measurements observed for
120 consecutive months. A multiple (time series) regression model can be written as:
yt=Xtβ+ϵt.yt=Xtβ+ϵt.
The difficulty that often arises in this context is that the errors (ϵtϵt) may be correlated with each
other. In other words, we have autocorrelationor a dependency between the errors.
We may consider situations in which the error at one specific time is linearly related to the error at
the previous time. That is, the errors themselves follow a simple linear regression model that can be
written as
ϵt=ρϵt−1+ωt.ϵt=ρϵt−1+ωt.
Here, |ρ|<1|ρ|<1 is called the autocorrelation parameter and the ωtωt term is a new error term that
follows the usual assumptions that we make about regression errors: ωt∼iidN(0,σ2)ωt∼iidN(0,σ2).
(Here "iid" stands for "independent and identically distributed.) So, this model says that the error at
time t is predictable from a fraction of the error at time t - 1 plus some new perturbation ωtωt.
Our model for the ϵtϵt errors of the original Y versus X regression is an autoregressive model for the
errors, specifically AR(1) in this case. One reason why the errors might have an autoregressive
structure is that the Y and X variables at time t may be (and most likely are) related to
the Y and X measurements at time t – 1. These relationships are being absorbed into the error term of
our multiple linear regression model that only relates Y and X measurements made at concurrent
times. Notice that the autoregressive model for the errors is a violation of the assumption that we
have independent errors and this creates theoretical difficulties for ordinary least squares estimates of
the beta coefficients. There are several different methods for estimating the regression parameters of
the Y versus X relationship when we have errors with an autoregressive structure and we will
introduce a few of these methods later.
The error terms ϵtϵt still have mean 0 and constant variance:
E(ϵt)=0Var(ϵt)=σ21−ρ2.E(ϵt)=0Var(ϵt)=σ21−ρ2.
However, the covariance (a measure of the relationship between two variables) between adjacent
error terms is:
Cov(ϵt,ϵt−1)=ρ(σ21−ρ2),Cov(ϵt,ϵt−1)=ρ(σ21−ρ2),
which implies the coefficient of correlation (a unitless measure of the relationship between two
variables) between adjacent error terms is:
We can use partial autocorrelation function (PACF) plots to help us assess appropriate lags for the
errors in a regression model with autoregressive errors. Specifically, we first fit a multiple linear
regression model to our time series data and store the residuals. Then we can look at a plot of the
PACF for the residuals versus the lag. Large sample partial autocorrelations that are significantly
different from 0 indicate lagged terms of ϵϵ that may be useful predictors of ϵtϵt.
Least Squares Method : We can take any other year
as the origin, and for that year X would be 0. Considerable
saving of both time and effort is possible if the origin is
taken in the middle of the whole time span covered by the
entire series. The origin would than be located at the
mean of the X values. Sum of the X values would then
equal 0. The two normal equations would then be
simplified to
∑Y = Na ...(i)
or a =
and ∑XY = b∑X or2
b = ...(ii)
Two cases of short cut method are given below. In the first
case there are odd number of years while in the second
case the number of observations are even.
Illustration : Fit a straight line trend on the following
data :
Year 1996 1997 1998 1999 2000 2001 2002 2003 2004
Y 4 7 7 8 9 11 13 14 17
Solution : Since we have 9 observations, therefore, the
origin is taken at 2000 for which X is assumed to be 0.
------------------------------
Year Y X XY X 2
------------------------------
1996 4 – 4 – 16 16
1997 7 – 3 – 21 9
1998 7 – 2 – 14 4
1999 8 – 1 – 8 1
2000 9 0 0 0
2001 11 1 11 1
2002 13 2 26 4
2003 14 3 42 9
2004 17 4 68 16
-----------------------------
Total 90 0 88 60
------------------------------
Thus n = 9, SY = 90, SX = 0, SXY = 88, and SX2 = 60
Substituting these values in the two normal equations, we
get
90 = 9a or a = 90/9 or a = 10
88 = 60 or b = 88/60 or b = 1.47
Trend equation is : Yc = 10 + 1.47 X
Inserting the various values of X, we obtain the trend
values as below :
----------------------------------------------
2003 6.7 – 7 – 46.9 49
2004 5.3 – 5 – 26.5 25
2005 4.3 – 3 – 12.9 9
2006 6.1 – 1 – 6.1 1
2007 5.6 1 5.6 1
2008 7.9 3 23.7 9
2009 5.8 5 29.0 25
2010 6.1 7 42.7 49
----------------------------------------------
Total 47.8 0 8.6 168
----------------------------------------------
From the above computations, we get the following
values.
n = 8, ∑Y = 47.8, ∑X = 0, ∑XY = 8.6, ∑X = 168 2
(iii) ∑X Y
2
= a∑X +
2
b∑X +
3
c∑X
4
(iii) ∑X Y
2
= a∑X + c∑X2 4
----------------------------------------------------------------------
-------------
Year Y X X
2
X 3
X 4
XY XY
2
Yc
------------------------------------------------------
-----------------------------
2000 100 – 2 4 – 8 16 –
200 400 97.744
2001 107 – 1 1 – 1 1 –
107 107 110.426
2002
128 0 0 0 0 0 0 126.680
2003
140 +1 1 +1 1 +140 140 146.5
06
2004 181 +2 4 +8 16 +
362 724 169.904
2005
192 +3 9 +27 81 +576 1728 196.8
74
----------------------------------------------------------------------
----------------
N = 6 ∑Y = 848 ∑X = 3 ∑X = 19 ∑X = 27 ∑X = 115 ∑XY =
2 3 4
------------------------------------------------------
--------------------------------
848 = 6a + 3b + 19c ...(i)
771 = 3a +19b +27c ...(ii)
3,099 = 19a + 27b +115c ...(iii)
Solving Eqns. (i) and (ii), get
35b + 35c = 695 ...(iv)
Multiplying Eqn. (ii) by 19 and Eqn. (iii) by 3. Subtracting
(iii) from (ii), we get
5352 = 280b + 168 c ...(v)
Solving Eqns. (iv) and (v), we get
c = 1.786
Substituting the value of c in Eqn. (iv), we get
b = 18.04 [35 b +(35 × 1.786) = 695]
Putting the value of b and c in Eqn. (i), we get
a = 126.68 [848 = 6a + (3 × 18.04) + (19 × 1.786))
Thus a = 126.68, b =18.04 and c = 1.786
Substituting the values in the equation
Yc = 126.68 + 18.04X + 1.786X 2
following
Years 1 2 3 4 5 6 7
Values 35 38 40 42 36 39 45
– 84c = – 4
c = 4/84 = 0.05
By substituting the value of c in equation (i) we get the
value of a
7a + 28 × 4/48 = 275
7a = 275 – 1.33
a = 273.67/7 = 39.09
We may get the value of b with the help of equation (ii)
28b = 28
b = 1
The required equation would be:
Yc = 39.09 + 1X + 0.05 X2
= 39.09 + X + 0.05 X2
https://sol.du.ac.in/mod/book/view.php?id=1317&chapterid=1071