n
Y
f (xi )
(1)
i=1
n
Y
i=1
f (xi |)
(2)
Note that the same is used in each term of the product, or in each marginal density. A different
value of would lead to a different properties for the random sample.
1.2. Statistics. Let X1 , X2 , ..., Xn be a random sample of size n from a population and let
T (x1 , x2 , ..., xn ) be a real valued or vector valued function whose domain includes the sample space
of (X1 , X2 , ..., Xn ). Then the random variable or random vector Y = (X1 , X2 , ..., Xn ) is called a
statistic. A statistic is a map from the sample space of (X1 , X2 , ..., Xn ) call it X, to some space of
values, usually R1 or Rn . T is what we compute when we observe the random variable X take on
some specific values in a sample. The probability distribution of a statistic Y = T(X) is called the
sampling distribution of Y. Notice that T() is a function of sample values only, it does not depend
on any underlying parameters, .
1.3. Some Commonly Used Statistics.
1.3.1. Sample mean. The sample mean is the arithmetic average of the values in a random sample.
It is usually denoted
n
X
1 , X2 , , Xn ) = X1 + X2 + ... + Xn = 1
X(X
Xi
n
n i=1
(3)
BASIC STATISTICS
S 2 (X1 , X2 , , Xn ) =
1 X
2
(Xi X)
n 1 i=1
(4)
The observed value of S 2 in any sample is demoted by the lower case letter, i.e., s2 .
1.3.3. Sample standard deviation. The sample standard deviation is the statistic defined by
S = S2
(5)
(6)
1X
I(Xi < x)
F (X1 , X2 , , Xn )(x) =
n i=1
(7)
where F (X1 , X2 , , Xn )(x) means we are evaluating the statistic F (X1 , X2 , , Xn ) at the particular
value x. The random sample X1 , X2 , ..., Xn is assumed to come from a probability defined on R1
and I(A) is the indicator of the event A. This statistic takes values in the set of all distribution
functions on R1 . It estimates the function valued parameter F defined by its evaluation at x R1
F (P )(x) = P [X < x]
2. D ISTRIBUTION
OF
(8)
S AMPLE S TATISTICS
b: (n 1)s =
n
P
i=1
i=1
(xi x
)2 =
n
P
i=1
x1 +x2 +...+xn
.
n
x2i n
x2
Part a says that the sample mean is the value about which the sum of squared deviations is
minimized. Part b is a simple identity that will prove immensely useful in dealing with statistical
data.
Proof. First consider part a of theorem 1. Add and subtract x from the expression on the lefthand
side in part a and then expand as follows
n
X
i=1
(xi x
+x
a)2 =
n
X
i=1
(xi x
)2 + 2
(xi x
)(
x a) = x
n
X
i=1
2
n
X
i=1
xi a
(xi x)(
x a) +
n
X
i=1
xi x
n
X
i=1
= n
x an
x n
x2 + n
xa
=0
x
+x
n
X
i=1
n
X
i=1
(
x a)2
(9)
(10)
BASIC STATISTICS
(xi a)2 =
n
n
X
X
(
x a)2
(xi x)2 +
(11)
i=1
i=1
(xi x
)2 =
=
n
X
i=1
n
X
i=1
n
X
i=1
x2i 2
x
n
X
xi +
n
X
x2
(12)
i=1
i=1
x2i 2n
x2 + n
x2
x2i n
x2
b: V ar
n
P
i=1
g (Xi ) = n (V arg(X1 ))
Proof. First consider part a of theorem 2. Write the expected value of the sum as the sum of the
expected values and then note that Eg(X1 ) = Eg(X2 ) = ...Eg(Xi ) = ...Eg(Xn ) because the Xi are
all from the same distribution.
E
n
X
g(Xi )
i=1
n
X
E(g(Xi )) = n(Eg(X1 ))
(13)
i=1
First consider part b of theorem 2. Write the definition of the variance for a variable z as E(z
E(z))2 and then combine terms in the summation sign.
V ar
n
X
i=1
g(Xi )
=E
=E
=E
"
n
X
"
n
X
i=1
i=1
" n
X
i=1
g(Xi )
g(Xi )
n
X
n
X
g(Xi )
i=1
E (g(Xi ))
i=1
#2
(g(Xi ) E (g(Xi ))
!#2
#2
(14)
BASIC STATISTICS
V ar
n
X
g(Xi )
i=1
(16)
(17)
Now we can use the fact that the X1 and Xj in the sample X1 , X2 , , Xn are independent to
assert that each of the covariances in the sum in 15 is zero. We can then rewrite 15 as
V ar
n
X
i=1
g(Xi )
n
X
V ar g(Xi )
i=1
n
X
V ar g(X1 )
i=1
= n V ar g(X1 )
(18)
2.3. Theorem 3 on expected values of sample statistics.
Theorem 3. Let X1 , X2 , Xn be a random sample from a population with mean and variance 2 < .
Then
=
a: E X
=
b: V arX
c: ES 2 = 2
2
n
BASIC STATISTICS
Proof of part a. In theorem 2 let g(X) = g(Xi ) = Xni . This implies that Eg(Xi ) =
!
!
n
n
X
X
1
1
1
=E
Xi = E
Xi = (nEX1 ) =
EX
n i=1
n
n
i=1
Proof of part b.
In theorem 2 let g(X) = g(Xi ) =
= V ar
V arX
Xi
n .
(20)
Proof of part c.
As in part b of theorem 1, write S 2 as a function of the sum of square of Xi minus n times the mean of
Xi squared and then simplify
#!
" n
X
1
2
2
2
X nX
ES = E
n 1 i=1 i
1
2
nEX12 nE X
n1
2
1
2
2
2
n( + ) n
= 2
+
=
n1
n
The last line follows from the definition of the variance of a random variable, i.e.,
=
(21)
2
V ar X = X
= EX 2 (EX)2
= EX 2 2X
2
E X 2 = X
+ 2X
2.4. Unbiased Statistics. We say that a statistic T(X)is an unbiased statistic for the parameter of
is an unbiased statistic
the underlying probability distribution if E T(X) = . Given this definition, X
for ,and S 2 is an unbiased statistic for 2 in a random sample.
3. M ETHODS
OF
E STIMATION
Let Y1 , Y2 , Yn denote a random sample from a parent population characterized by the parameters 1 , 2 , k . It is assumed that the random variable Y has an associated density function
f ( ; 1 , 2 , k ).
3.1. Method of Moments.
3.1.1. Definition of Moments. If Y is a random variable, the rth moment of Y, usually denoted by r ,
is defined as
r = E(Y r )
Z
y r f (y; 1 , 2 , k ) dy
=
(22)
E(Y r ) = r = gr (1 , 2 , k )
(23)
if the expectation exists. Note that 1 = E(Y ) = Y , the mean of Y. Moments are sometimes
written as functions of .
BASIC STATISTICS
3.1.2. Definition of Central Moments. If Y is a random variable, the rth central moment of Y about a
is defined as E[(Y a)r ]. If a = r , we have the rth central moment of Y about Y , denoted by r ,
which is
r = E[(Y Y )r ]
Z
(y y )r f (y; 1 , 2 , k ) dy
=
(24)
Note that 1 = E[(Y Y )] = 0 and 2 = E[(Y Y )2 ] = V ar[Y ]. Also note that all odd
numbered moments of Y around its mean are zero for symmetrical distributions, provided such
moments exist.
3.1.3. Sample Moments about the Origin. The rth sample moment about the origin is defined as
n
r = x
rn =
1X r
y
n i=1 i
(25)
3.1.4. Estimation Using the Method of Moments. In general r will be a known function of the parameters 1 , 2 , k of the distribution of Y, that is r = gr (1 , 2 , k ). Now let y1 , y2 , , yn be a
random sample from the density f (; 1 , 2 , k ). Form the K equations
n
1 =g1 (1 , 2 , k ) =
1 =
1X
yi
n i=1
2 =g2 (1 , 2 , k ) =
2 =
1X 2
y
n i=1 i
(26)
..
.
n
K =gK (1 , 2 , k ) =
K =
1X K
y
n i=1 i
The estimators of 1 , 2 , k , based on the method of moments, are obtained by solving the
system of equations for the K parameter estimates 1 , 2 , K .
This principle of estimation is based upon the convention of picking the estimators of i in such a
manner that the corresponding population (theoretical) moments are equal to the sample moments.
These estimators are consistent under fairly general regularity conditions, but are not generally
efficient. Method of moments estimators may also not be unique.
3.1.5. Example using density function f (y) = (p + 1) y p . Consider a density function given by
f (y) = (p + 1) y p 0 y 1
= 0 otherwise
(27)
Let Y1 , Y2 , Yn denote a random sample from the given population. Express the first moment
of Y as a function of the parameters.
BASIC STATISTICS
E(Y ) =
y f (y) dy
y (p + 1) y p dy
y p+1 (p + 1) dy
(28)
1
y p+2 (p + 1)
=
(p + 2)
0
p+1
p+2
Then set this expression of the parameters equal to the first sample moment and solve for p.
1 = E(Y ) =
p+1
p+2
p+1
1X
yi = y
=
p+2
n i=1
p + 1 = (p + 2) y = p
y + 2
y
(29)
p p
y = 2 y 1
p(1 y) = 2 y 1
p =
2 y 1
1 y
3.1.6. Example using the Normal Distribution. Let Y1 , Y2 , Yn denote a random sample from a normal distribution with mean and variance 2 . Let (1 , 2 ) = (, 2 ). The moment generating
function for a normal random variable is given by
MX (t) = et+
t2 2
2
(30)
The moments of X can be obtained from MX (t) by differentiating with respect to t. For example
the first raw moment is
E(X) =
d t +
e
dt
t2 2
2
t=0
= ( + t 2 ) e
=
The second raw moment is
2 2
t+ t 2
(31)
t=0
BASIC STATISTICS
d2 t + t2 2
2
e
dt2
t=0
t2 2
d
=
+ t 2 e t+ 2
dt
t=0
2 2
t2 2
t
2
+ 2 et+ 2
e t+ 2
= + t 2
E(x2 ) =
= +
So we have =
(32)
t=0
(33)
Now set the first population moment equal to its sample analogue to obtain
n
1X
yi = y
n i=1
(34)
= y
Now set the second population moment equal to its sample analogue
n
2 + 2 =
1X 2
y
n i=1 i
n
1X 2
y 2
n i=1 i
v
u n
u1 X
y 2 2
=t
n i=1 i
2 =
=t
y 2 y2
n i=1 i
v
u n
uX (yi y)2
=t
n
i=1
(35)
(36)
This is, of course, different from the sample standard deviation defined in equations 4 and 5.
3.1.7. Example using the Gamma Distribution. Let X1 , X2 , Xn denote a random sample from a
gamma distribution with parameters and . The density function is given by
f (x; , ) =
x
1
x1 e
()
=0
otherwise
0x<
(37)
BASIC STATISTICS
x
1
x1 e dx
()
0
Z
x
1
=
x(1+)1 e dx
() 0
E(X) =
(38)
x
1
x(1+)1 e dx
1+ (1 + )
(39)
The integrand of equation 39 is a gamma density with parameters and 1 This integrand
will integrate to one so that we obtain the expression in front of the integral sign as the E(X).
E(X) =
1+ (1 + )
()
(40)
(1 + )
=
()
The gamma function has the property that (t) = (t 1)(t 1) or (v + 1) = v(v). Replacing
(1 + ) with () in equation 40, we obtain
E(X) =
=
(1 + )
()
()
()
(41)
=
We can find the second moment by finding E(X 2 ). To do this we multiply the gamma density
in equation 38 by x2 instead of x. Carrying out the computation we obtain
E(X ) =
x2
x1 e
()
1
=
()
dx
(42)
(2+)1
dx
10
BASIC STATISTICS
E(X 2 ) =
=
2+ (2 + )
()
2+
x
1
x(2+)1 e dx
(2 + )
2+
(2 + )
()
2 ( + 1) (1 + )
()
2 ( + 1) ()
()
(43)
= 2 ( + 1)
Now set the first population moment equal to the sample analogue to obtain
n
1X
xi = x
n i=1
(44)
x
=
Now set the second population moment equal to its sample analogue
n
1 X 2
x
n i=1 i
Pn
2
i=1 xi
2 =
n ( + 1)
Pn
x2
i
i=1
2 =
x
+
1
n x
2 ( + 1 ) =
2 =
Pn
x2
i=1 i
2
nx
nx
+
2
n x
2 + n x
=
n x
=
=
=
Pn
i=1
n
X
i=1
Pn
i=1
n
X
(45)
x2i
i=1
x2i n x
2
x2i n x
2
nx
(xi x
)
nx
3.1.8. Example with unknown distribution but known first and second moments. Let Y1 , Y2 , Yn denote
a random sample from an unknown distribution with parameters and . We know the following
about the distribution of Y.
BASIC STATISTICS
11
Y = + u
E(u) = 0
(46)
E(u2 ) = V ar(u) = 2
Consider estimators for and 2 . In a given sample
yi = + u
i
(47)
ui = yi
The sample moments are then as follows
First sample moment =
Second sample moment =
Substituting from expression 47 we obtain
First sample moment =
Pn
Pn
Pn
i=1
ui
Pn
i=1
u2i
(48)
i=1
yi
n
2
i
i=1
(49)
n
If we set the first sample moment equal to the first population moment we obtain
Pn
i
i=1
= 0
n
n
X
yi = n
(50)
i=1
Pn
yi
=
n
If we set the second sample moment equal to the second population moment we obtain
Pn
i=1
2
yi
n
i=1
= 2
2 =
2
i
i=1
Pn
(51)
12
BASIC STATISTICS
(52)
where E(i ) = 0 and Var(i ) = for all i. This is equivalent to stating that the population from
which yi is drawn has a mean of and a variance of 2 .
The least squares estimator of is obtained by minimizing the sum of squares errors, SSE, defined by
SSE =
n
X
i=1
e2i =
n
X
i=1
2
(yi )
(53)
The idea is to pick the value of to estimate which minimizes SSE. Pictorially we select the
value of which minimizes the sum of squares of the vertical deviations in figure 1.
The solution is obtained by finding the value of that minimizes equation 53.
BASIC STATISTICS
13
n
X
SSE
(yi )(1)
= 0
=2
i=1
(54)
n
1 X
=
yi = y
n i1
This method chooses values of the parameters of the underlying distribution, , such that the
distance between the elements of the random sample and predicted values are minimized.
3.2.2. Example with two parameters. Consider the model
yt = 1 + 2 xt + t
= 1 + 2 xt + et
(55)
et = yt 1 2 xt
where E(t ) = 0 and Var(t ) = 2 for all t. This is equivalent to stating that the population from
which yt is drawn has a mean of 1 + 2 xt and a variance of 2 .
Now if these estimated errors are squared and summed we obtain
SSE =
n
X
e2t =
t=1
n
X
t=1
(yt 1 2 xt ) 2
(56)
This sum of squares of the vertical distances between yt and the predicted yt on the sample regression line is abbreviated SSE. Different values for the parameters give different values of SSE.
The idea is to pick values of 1 and 2 that minimize SSE. This can be done using calculus. Specifically
X
X
SSE
=2
(yt 1 2 xt )(1) = 2
et = 0
1
t
X
SSE
=2
(yt 1 2 xt ) (xt )
2
(57)
= 2
X
(yt xt 1 xt 2 x2t ) = 2t et xt = 0
t
et = 0
(58)
et xt = 0.
These equations are often referred to as the normal equations. Note that the normal equations
imply that the sample mean of the residuals is equal to zero and that the sample covariance between
the residuals and x is zero since the mean of et is zero. The easiest method of solution is to solve
the first normal equation for 1 and then substitute into the second. Solving the first equation gives
14
BASIC STATISTICS
X
t
(yt 1 2 Xt ) = 0
yt = n 1 + 2
xt
(59)
yt
t xt
= y 2 x
n
n
This implies that the regression line goes through the point (
y, x). The slope of the sample
regression line is obtained by substituting 1 into the second normal equation and solving for 2 .
This will give
1 =
X
t
(yt xt 1 xt 2 x2t ) = 0
X
t
yt xt = 1
xt + 2
= (
y 2 x
)
x2t
xt + 2
= n y x n2 x
2 + 2
X
t
x2t
= (
y 2 x
) n x
+ 2
= n y x + 2
x2t
(60)
x2t
x2t n
x2
P
yt xt n
xy
2 = Pt 2
2
x
n
x
t t
P
(yt y) (xt x
)
= tP
2 x
2
(x
)
t
t
3.2.3. Method of moments estimation for example in 3.2.2. Let Y1 , Y2 , Yn denote a random sample
from an unknown distribution with parameters 1 , 2 , and . We know the following about the
distribution of Y.
Y = 1 + 2 X + u
E(u) = 0
E(u2 ) = V ar(u) = 2
(61)
E(u x) = 0
(62)
BASIC STATISTICS
15
Pn
i=1
n
Pn
i=1
n
Pn
i=1
u
i
u
2i
(63)
xi ui
1 2 xi
y
i
i=1
Pn
Pn
Pn
i=1
2
yi 1 2 xi
n
1 2 xi
y
x
i
i
i=1
n
(64a)
(64b)
(64c)
If we set the first sample moment equal to the first population moment we obtain
Pn
i=1
yi 1 2 xi
n
n
X
i=1
yi 2
n
X
= 0
yi = n1 + 2
i=1
i=1
n
X
xi = n1
i=1
y 2 x
= 1
n
X
xi
(65)
16
BASIC STATISTICS
n
X
xi yi = 1
n
X
n
X
xi + 2
i=1
i=1
i=1
= (
y 2 x
)
= y
n
X
i=1
x2i
n
X
xi + 2
n
X
xi + 2
= n
yx
+ 2
i=1
n
X
i=1
x2t n x
2
n
X
i=1
n
X
x2i
i=1
i=1
= n
yx
2 xn
x + 2
n
X
x2i
i=1
i=1
xi 2 x
n
X
n
X
x2i
i=1
x2t n x
2
(66)
!
xi yi n
yx
Pn
xi yi n
y x
2 = Pi=1
n
2 nx
2
x
i=1 i
Pn
(x x
)(yi y)
Pn i
= i=1
(x
)2
i=1 i x
We obtain an estimate for 2 from equation 64b
Pn
i=1
2
yi 1 2 xi
n
= 2
2 =
2
1 2 xi
y
i
i=1
Pn
(67)
BASIC STATISTICS
17
3.3. Method of maximum likelihood estimation (MLE). Least squares is independent of a specification of a density function for the parent population. Now assume that
yi f ( ; = (1 , . . . K )) , i.
(68)
3.3.1. Motivation for the MLE method. If a random variable Y has a probability density function f(;
) characterized by the parameters = (1 , . . . , k ), then the maximum likelihood estimators (MLE)
of (1 , . . . , k ) are the values of these parameters which would have most likely generated the given
sample.
3.3.2. Theoretical development of the MLE method. The joint density of a random sample y1 , y2 ,. . . ,yn
is given by L = g(y1 , . . . , yn ; ) = f (y1 ; ) f (y2 ; ) f (y3 ; ) f (yn ; ) . Given that we have
a random sample, the joint density is just the product of the marginal density functions. This is
referred to as the likelihood function. The MLE of the i are the i which maximize the likelihood
function.
The necessary conditions for an optimum are:
L
= 0,
i
i = 1, 2, ..., k
(69)
This gives k equations in k unknowns to solve for the k parameters 1 , . . . , k . In many instances
it will be convenient to maximize = ln L rather than L given that the log of a product is the sum
of the logs.
3.3.3. Example 1. Let the random variable Xi be distributed as a normal N(, 2 ) so that its density
is given by
f (xi ; , 2 ) =
1
2 2
1
2
( xi )
(70)
L =
n
Y
i=1
2 2
n
n
2 2
ln L = =
x2
xn
( x1 ) e 1
) e 1
)
2 (
2 (
1
2
1
22
Pn
2
i=1 (xi )
n
n
1 X
(xi )2
ln(2 2 )
2
2 2 i=1
The MLE of and 2 are obtained by taking the partial derivatives of equation 71
(71)
18
BASIC STATISTICS
Pn
n
1 X
i=1 xi
(xi ) = 0
=
= 2
=x
i=1
n
n
=
2
2
2
2
2
n
1
2 X
2
(xi
) = 0
(1)(
2 )
2
i=1
n
X
1
n
2
(xi
)
=
2
2
(2
2 )2 i=1
n =
n
1 X
2
(xi
)
2 i=1
2 =
n
1 X
2
(xi
)
n i=1
2 =
n
1 X
2
(xi x
)
n i=1
n 1
n
(72)
s2
The MLE of 2 is equal to the sample variance and not S2 ; hence, the MLE is not unbiased as can
be seen from equation 21. The MLE of is the sample mean.
3.3.4. Example 2 - Poisson. The random variable Xi is distributed as a Poisson if the density of Xi is
given by
f (xi ; ) =
( e xi
xi is a non-negative integer
xi !
otherwise
(73)
mean (X) =
V ar (X) =
The likelihood function is given by
L =
e x1
x1 !
Pn
en
= Qn
i=1
i=1
e x2
x2 !
e xn
xn !
xi
(74)
xi !
ln L = = n +
n
X
i=1
xi ln ln
n
Y
i=1
xi !
BASIC STATISTICS
19
n
X
1
xi = 0
= n +
i=1
Pn
= i=1 xi = x
(75)
f (y) =
(p + 1)y p
n
Y
0 y 1
(76)
otherwise
(p + 1) yip
i=1
ln L = =
n
X
ln [(p + 1) yip ]
i=1
n
X
(ln(p + 1) +
(77)
ln yip )
i=1
n
X
(ln(p + 1) + p ln yi )
i=1
1
=
+ ln yi = 0
p
p+1
i=1
n
X
n
X
1
( ln yi )
=
p + 1
i=1
n
X
n
( ln yi )
=
p + 1
i=1
i=1
(78)
n
i=1 ln yi
p + 1 = Pn
n
1
i=1 ln yi
p = Pn
0 p 1
(79)
20
BASIC STATISTICS
L = ni=1 pyi ( 1 p )1 yi
= p
Pn
i=1
n
X
ln L = =
Pn
yi
n
X
( 1 p )n
yi
yi ln p +
i=1
i=1
(80)
yi
i=1
ln (1 p)
To obtain the MLE estimator differentiate 80 with respect to p where we assume that 0 < p < 1.
=
p
n
X
i=1
Pn
i=1
yi p
n
X
n
X
yi
i=1
Pn
i=1
yi
(n
Pn
i=1
1 p
yi )
= 0
Pn
(n
i = 1 yi )
=
1 p
yi = n p p
n
X
yi
(81)
i=1
yi = n p
i=1
Pn
i=1
yi
= p
3.3.7. Example 5. Let Y1 , Y2 , Yn denote a random sample from an unknown distribution with
parameters 1 , 2 , and . We know the following about the distribution of Yi .
Yi = 1 + 2 Xi + ui
E(ui ) = 0
E(u2i ) = V ar(ui ) = 2
ui and uj are independent for all i 6=j
(82)
BASIC STATISTICS
L =
n
Y
i=1
2 2
n
2 2
n
e 22
ln L = =
=
21
1
2
y2 1 2 x2
yn 1 2 xn
( y1 1 2 x1 ) e 1
) e 1
)
2 (
2 (
Pn
i=1 (yi
1 2 x i )2
(83)
n
n
1 X
(yi 1 2 xi )2
ln(2 2 )
2
2 2 i=1
n
n
n
1 X
(yi 1 2 xi )2
ln(2)
ln( 2 )
2
2
2 2 i=1
The MLE of 1 , 2 , and 2 are obtained by taking the partial derivatives of equation 83
1 X
(yi 1 2 xi ) = 0
= 2
1
i
1 X
(yi 1 2 xi ) (xi ) = 0
= 2
1
n
X
n
1
2
(yi 1 2 xi ) = 0
=
+
2
2 2
(2 2 )2 i=1
(84a)
(84b)
(84c)
X
i
(yi 1 2 xi ) = 0
y i = n 1 + 2
xi
(85)
1 =
yi
xi
= y 2 x
We can find 2 by substituting 1 into equation 84b and then solving for 2 . This will give
22
BASIC STATISTICS
X
i
(yi xi 1 xi 2 x2i ) = 0
yi xi = 1
xi + 2
X
xi + 2
= (
y 2 x) n x
+ 2
= n y x n2 x2 + 2
X
i
x2i
= n y x + 2
x2i
= (
y 2 x)
x2i
(86)
x2i
x2i n
x2
P
yi xi n
xy
2 = Pi 2
2
x
n
x
i t
P
) (xi x)
( yi y
P
=
2
)2
i (xt x
n
X
n
1
2
(yi 1 2 xi ) = 0
=
+
2
2 2
(2 2 )2 i=1
n
X
1
n
2
(yi 1 2 xi )
=
2 2
(2 2 )2 i=1
n =
n
1 X
2
(yi 1 2 xi )
2 i=1
2 =
n
1 X
2
(yi 1 2 xi )
n i=1
2 =
n
2
1 X
yi 1 2 xi
n i=1
(87)
BASIC STATISTICS
n
X
23
(88)
ai y i
i=1
n
X
(89)
ai y i
i=1
=E
E()
=
=
n
X
i=1
n
X
n
X
ai E(yi )
ai
i=1
n
X
=
=>
ai y i
i=1
(90)
ai
i=1
n
X
ai = 1
i=1
n
X
"
n
X
i=1
ai y i
(91)
a2i 2
i=1
because the covariance between yi and yj (i 6= j) is equal to zero due to the fact that the ys are
drawn from a random sample.
The P
problem of obtaining a BLUE of becomes that of minimizing
n
straint i = 1 ai = 1 . This is done by setting up a Lagrangian
Pn
i=1
24
BASIC STATISTICS
L(a, ) =
n
X
i=1
n
X
ai 1)
a2i (
(92)
i=1
L
= 2a1 = 0
a1
.
.
.
(93)
L
= 2an = 0
an
n
X
L
ai + 1 = 0
=
i=1
The first n equations imply that a1 = a2 = a3 = . . . an so that the last equation implies that
n
X
i=1
ai 1 = 0
nai 1 = 0
nai = 1
1
ai =
n
n
n
X
1 X
=
ai y i =
yi = y
n i=1
i=1
(94)
BASIC STATISTICS
25
OF
E STIMATORS
4.1. Introduction to sample properties of estimators. In section 3 we discussed alternative methods of estimating the unknown parameters in a model. In order to compare the estimating techniques we will discuss some criteria which are frequently used in such a comparison. Let denote
an unknown parameter and let and be alternative estimators. Now define the bias, variance and
mean squared error of as
= E ()
Bias ()
=E
V ar ()
2
E ()
2
M SE ( ) = E
2
+ Bias
= V ar ()
(95)
2
= E E + E
=E
2
E + E
2
h
i
2
= E E
+ 2 E E E + E
(96)
2
2
= E E ()
+ E ()
since E E
= 0
2
= V ar + Bias()
4.2. Specific properties of estimators.
4.2.1. Unbiasedness. is said to be an unbiased estimator of if E = .
In figure 2, is an unbiased estimator of , while is a biased estimator.
4.2.2. Minimum variance. is said to be a minimum variance estimator of if
V ar V ar
(97)
where is any other estimator of . This criterion has its disadvantages as can be seen by noting
that = constant has zero variance and yet completely ignores any sample information that we may
26
BASIC STATISTICS
fHL
fHL
fHL
fHL
fHL
fHL
(98)
where is any other estimator of . This criterion takes into account both the variance and bias
of the estimator under consideration. Figure 4 shows three alternative estimators of .
BASIC STATISTICS
27
fHL
fHL
fHL
fHL
4.2.4. Best linear unbiased estimators. is the best linear unbiased estimator (BLUE) of if
=
n
X
ai yi linear
i=1
= unbiased
E()
(99)
V ar()
V ar()
where is any other linear unbiased estimator of .
the efficient estimators will also be minimum variance
For the class of unbiased estimators of ,
estimators.
4.2.5. Example. Let X1 , X2 , . . ., Xn denote a random sample drawn from a population having a
population mean equal to and a population variance equal to 2 . The sample mean (estimator of
) is calculated by the formula
28
BASIC STATISTICS
=
X
n
X
Xi
n
i=1
(100)
2 =
S2 =
n
X
2
(Xi X)
n
i=1
n
X
2
(Xi X)
n1
i=1
n 1
n
S2
n 1
n
E S2
n 1
n
(101)
V ar X
2
n
(102)
E S
2 2
= V ar S
2 4
=
n 1
(103)
The variance of S2 was derived in the lecture on sample moments. The variance of
2 is easily
2
computed given the variance of S . Specifically,
BASIC STATISTICS
n 1 2
V ar
= V ar
s
n
2
n 1
=
V ar S 2
n
2
n 1
2 4
=
n
n 1
2 (n 1) 4
=
n2
2
We can compute the MSE of
using equations 95, 101, and 104 as follows
2
M SE
2
n 1
2 (n 1) 4
2
2
+
=E
=
n2
n
2
2 (n 1) 4
n 1
n 1
4
=
4 + 4
+
2
n2
n
n
2 (n 1)
(n 1)2
2 n (n 1)
n2
4
=
+
+ 2
n2
n2
n2
n
2 n 2 + n2 2 n + 1 2 n2 + 2 n + n2
= 4
n2
2n 1
= 4
n2
2
29
(104)
2
M SE
2 = 4
n2
n 1
So
2 is a biased estimator of S2 but has lower mean square error.
(105)
(106)