Y=Xβ +ε
where
Yi = β 0 + β 1Xi1 + β 2X12 + . β X1(p-1) + . . .
p-1 ε i,
X is the n x p matrix with first column all 1’s and the values of X1,
X2 , . . .Xp-1 (assumed to be of rank p)
The most commonly used criterion to estimate the parameters in the model
is the principle of least squares, which involves minimizing
n
Q= ∑ε
1
i
2
= Σ[Yi - β 0 - β 1Xi1 - β 2Xi2 - . β Xi(p-1)]2 = (Y = X β )′ ( Y = X β )
p-1
X′Xb = X′Y
b = (X′X)-1X′Y
Note: If we assume that the errors are normally distributed then the least
squares estimator b is also the maximum likelihood estimator of β (to be
discussed later).
Residuals ei are the difference between observed and fitted values and are
given by
ei = Yi – Yˆi ,
or in vector form by
The resulting minimum value of Q, called the sum of squared errors SSE, is
given by
Yˆi
2
SSE =∑ ei = (Y- Xb)′(Y –Xb) = Σ(Yi – )2
Page 24
The fitted values are given by
Yˆ = Xb = X(X′X)-1X′Y = HY
High values of the diagonal elements hij indicate that the observation Yi has
a high influence on the fitted value.
.0.445
0.329
Diag=
0.265
0.961
The diagonal elements hii measure the influence (leverage) of the individual
observations. Both h22 and h44 have very high leverage. For example,
observed values: 20
y
10
1 2 3 4 5 6 7 8 9 10
x
Page 25
From the graph it is easy to see why the observation Y4 has a large influence
(leverage) on its fitted value (and on the fitted regression line as well).
predictions.
2. Use the fitted regression line, getting fitted values Yˆ = βˆ 0 + βˆ1 x in the
simple linear regression model.. Then compute SSE = ∑(Yi −Yˆi ) 2 .
Compare the sum of squared errors of fitted and observed values for the two
methods. Then
equals the proportionate reduction in the sum of squared errors using the
fitted regression line vs. using the sample mean Y . R2.is usually expressed
as a percentage reduction. It is also interpreted as the amount of variability
in the observations that can be explained (or accounted for) by the
predictors. Note that the sample variance of the observations is
S y2 = SSTO/ (n-1)
and the variance of the residuals using the regression equation is given by
The F-test is used to test the hypothesis that all of the parameters β 1, β 2.,
…, β p-1 are simultaneously zero. Use the p-value of the test to make a
decision on this (this is probably practically not an issue!).
Confidence Intervals.
Recall (from page 23) that b = (X′X)-1X′Y is the estimated vector of (vector)
of parameters β . It can be shown that the variance –covariance matrix of b
is given by
which is estimated by
The square root of the diagonal elements s(bi) of this matrix are the standard
errors of the estimated regression parameters b1, b2., …, bp-1. Confidence
intervals for the bi ‘s are then given by
Similarly, one can construct confidence intervals for the mean response
µnew=E(Ynew) corresponding to a population mean indexed by for values of
x1, x2., …, xp-1. The mean response is estimated by Ŷnew = X h' b , where
'
X new is the (row) vector of values of x1, x2., …, xp-1. It can be shown that
the standard error of the estimated response is given by
If there are (P-1) predictors x1, x2., …, xP-1. one can conceivably fit 2P-1
different models to the data. For example, there are P-1 models with one
predictor x1, P(P-1) models with 2 predictors, etc. Some criteria used for
comparing models include the following (p as a subscript below refers to the
number of predictors in a model):
SSEp or R p2 . Note first that SSEp and R p2 are equivalent measures, in that
SSE
R p2 =1= p
SSTO
The goal in using either of these statistics is to choose a model where their
2
values are ‘small’. One can plot, e.g., R p against p and choose a model, or
models, where it is asmptoting (not changing).
Ra2, p ,
is the same measure as R p2 but with an adjustment for sample size. It
is given by
( n −1) SSE p MSE p
Ra2, p = 1−
( n − p ) SSTO
= 1− S y2
2
where S y = SSTO/(n-1) is the sample variance of the observations. Thus,
Ra2, p looks at how the ratio of sample variances for the model with p
Page 28
Cp . This criterion is concerned with the total mean squared error of the n
fitted values for each subset selection model. It is a bit complicated to
describe here. Suffice to say, most statisticians now prefer to use the BIC
criterion.
We will look at an example using the : SDSS Quasar Sloan Digital Sky
Survey team (CASt dataset SDSS_quasar.dat). Here are 8 of the first 10
observations in the dataset (which contains 46420 observations in all). The
variables are as follows: