STATISTICAL INFERENCE

PART I
EXPONENTIAL FAMILY & POINT
ESTIMATION

EXPONENTIAL CLASS OF PDFS

X is a continuous (discrete) rv with pdf
f(x;), . If the pdf can be written in the
following form
k

f ( x; ) h ( x )c( ) exp( w j ( ) t j ( x ))
j1

then, the pdf is a member of exponential

class of pdfs of the continuous (discrete)
type. (Here, k is the number of parameters)
2

REGULAR CASE OF THE

EXPONENTIAL CLASS OF PDFS

We have a regular case of the exponential

class of pdfs of the continuous type if
a) Range of X does not depend on .
b) c() 0, w1(),, wk() are real valued functions
of for .
c) h(x) 0, t1(x),, tk(x) are real valued functions of
x.
If the range of X depends on , then it is called
irregular exponential class or range-dependent
exponential class.

EXAMPLES
X~Bin(n,p), where n is known. Is this pdf a
member of exponential class of pdfs? Why?
n x
f ( x; p) p (1 p) n x ;
x

x 0,1,..., n; 0 p 1

n
p
(1 p) n exp( x ln(
))
1 p
x
n
h ( x )
x
t1 ( x ) x

for
for

x 0,..., n; c(p) (1 p) n
x 0,..., n;

p
w1 (p) ln(
)
1 p

for
for

0 p 1
0 p 1

Binomial family is a member of exponential family of distributions.

EXAMPLES
X~Cauchy(1,). Is this pdf a member of
exponential class of pdfs? Why?
f ( x; ) ( (1 [ x ]2 )) 1

1
exp{ ln(1 x 2 2x 2 )}

1
h ( x ) ; c( ) 1; ln(1 x 2 2x 2 ) t1 ( x ) w1 ( )

Cauchy is not a member of exponential family.

STATISTICAL INFERENCE
Determining certain unknown properties of
a probability distribution on the basis of a
sample (usually, a r.s.) obtained from that
distribution

Point Estimation: 5

(
)
Interval Estimation: 3 8
Hypothesis Testing: H 0 : 5
H1 : 5
6

STATISTICAL INFERENCE
Parameter Space ( or ): The set of all
possible values of an unknown parameter,
; .
A pdf with unknown parameter: f(x; ), .
Estimation: Where in , is likely to be?
{ f(x; ), }

The family of pdfs

7

STATISTICAL INFERENCE
Statistic: A function of rvs (usually a
sample rvs in an estimation) which does not
contain any unknown parameters.
2
X , S , etc
Estimator of an unknown parameter :
A statistic used for estimating .
: estimator U X 1 , X 2 ,L , X n
X : Estimator
An observed value

8
x : Estimate : A particular value of an estimator

POINT ESTIMATION
: a parameter of interest; unknown
Goal: Find good estimator(s) for or its
function g().

SOME METHODS OF POINT

ESTIMATION
Method of Moments Estimation,
Maximum Likelihood Estimation,
Least Squares Estimation
10

METHOD OF MOMENTS
ESTIMATION (MME)
Let X1, X2,,Xn be a r.s. from a population
with pmf or pdf f(x;1, 2,, k). The MMEs
are found by equating the first k population
moments to corresponding sample moments
and solving the resulting system of
equations.
Population Moments

k E X k

Sample Moments
1n k
Mk Xi
n i1
11

METHOD OF MOMENTS
ESTIMATION (MME)
1 M 1
1 n
E X Xi
n i1

3 M 3
2 M 2
n
n
1
1
2
2
3
E X X i E X X i3
n i1
n i1

so on

Continue this until there are enough equations

to solve for the unknown parameters.

12

EXAMPLES
Let X~Exp().
For a r.s of size n, find the MME of .
For the following sample (assuming it is
from Exp()), find the estimate of :
11.37, 3, 0.15, 4.27, 2.56, 0.59.

13

EXAMPLES
Let X~N(,). For a r.s of size n, find the
MMEs of and .
For the following sample (assuming it is
from N(,)), find the estimates of and
:
4.93, 6.82, 3.12, 7.57, 3.04, 4.98, 4.62,
4.84, 2.95, 4.22

14

DRAWBACKS OF MMES
Although sometimes parameters are
positive valued, MMEs can be negative.
If moments do not exist, we cannot find
MMEs.

15

MAXIMUM LIKELIHOOD
ESTIMATION (MLE)
Let X1, X2,,Xn be a r.s. from a population
with pmf or pdf f(x;1, 2,, k), the
likelihood function is defined by
L 1 , 2 ,L , k x1 , x2 ,L , xn f x1 , x2 ,L , xn ;1 , 2 ,L , k
f xi 1 , 2 ,L , k
n

i 1
n

f xi ;1 , 2 ,L , k
i 1

16

MAXIMUM LIKELIHOOD
ESTIMATION (MLE)
For each sample point (x1,,xn), let
1 x1 ,..., xn ,L ,k x1 ,..., xn
be a parameter value at which
L(1,, k| x1,,xn) attains its maximum as a
function of (1,, k), with (x1,,xn) held fixed. A
maximum likelihood estimator (MLE) of
parameters (1,, k) based on a sample (X1,,Xn) is
1 x1 ,..., xn ,L ,k x1 ,..., xn
The MLE is the parameter point for which the
observed sample is most likely.

17

EXAMPLES
Let X~Bin(n,p), where both n and p are unknown. One
observation on X is available, and it is known that n is
either 2 or 3 and p=1/2 or 1/3. Our objective is to
estimate the pair (n,p).
x

(2,1/2)

(2,1/3)

(3,1/2)

(3,1/3)

Max. Prob.

1/4

4/9

1/8

8/27

4/9

1/2

4/9

3/8

12/27

1/2

1/4

1/9

3/8

6/27

3/8

1/8

1/27

1/8

(2,1/ 3) if x 0
(2,1/ 2) if x 1

n
,
p
x

(3,1/ 2) if x 2
(3,1/ 2) if x 3

18

MAXIMUM LIKELIHOOD
ESTIMATION (MLE)
It is usually convenient to work with the logarithm
of the likelihood function.
Suppose that f(x;1, 2,, k) is a positive,
differentiable function of 1, 2,, k. If a
supremum 1 ,2 ,L ,k exists, it must satisfy the
likelihood equations
ln L 1 , 2 ,L , k ; x1 , x2 ,L , xk
0, j 1, 2,..., k
j

MLE occurring at boundary of cannot be

obtained by differentiation. So, use inspection.

19

MLE
Moreover, you need to check that you are
in fact maximizing the log-likelihood (or
likelihood) by checking that the second
derivative is negative.

20

EXAMPLES
1. X~Exp(), >0. For a r.s of size n, find the
MLE of .

21

EXAMPLES
2. X~N(,2). For a r.s. of size n, find the
MLEs of and 2.

22

EXAMPLES
3. X~Uniform(0,), >0. For a r.s of size n,
find the MLE of .

23

INVARIANCE PROPERTY OF THE

MLE
If is the MLE of , then for any function
(), the MLE of () is .
Example: X~N(,2). For a r.s. of size n, the
MLE of is X . By the invariance property
2
2
of MLE, the MLE of is X .

24

ADVANTAGES OF MLE
Often yields good estimates, especially for
large sample size.
Invariance property of MLEs
Asymptotic distribution of MLE is Normal.
Most widely used estimation technique.
Usually they are consistent estimators. [will
define consistency later]

25

DISADVANTAGES OF MLE
Requires that the pdf or pmf is known except
the value of parameters.
MLE may not exist or may not be unique.
MLE may not be obtained explicitly (numerical
or search methods may be required). It may
be sensitive to the choice of starting values
when using numerical estimation.
MLEs can be heavily biased for small
samples.
26

S
x
y

x
0
1
2
x
y
x
ii
i
22
Sxii(x)xnSxy(xiii)yxyin
LEAST SQUARES ERROR

Minimize the square of error

Mostly used in regression
As an example consider simple linear
model: yi 0 1 xi n i , i=1,2,...,n
n
LSE says minimize i2 ( yi 0 i xi )2
i 1

i 1

27

Problems
1. Let Xi (i=1,,n) be a random sample from
gamma distribution s.t. Xi~Gamma(2,). The
p.d.f. of X1 is given by:
1
x1 /
f ( x1 )
x
e
; x1 0, 0
2 1
(2)
a)Is this a member of exponential class?

b) Find the MLE of .

c) Find the MME of .
28

Problems
2. Let X 1 , X 2 ,..., X n be a random sample from inverse
gamma distribution with the following probability
density function.
f (x | , )

e / x

( ) x

for 0, 0, x 0

a) Does the distribution of X 1 , X 2 ,..., X n belong to the

exponential family of distributions?
b) Find the kth moment, i.e. E(Xk), for any k=1,2,
c) Find MMEs of and .
29

Problems
3. Sometimes the regression function is known to be linear
and go through the origin, i.e. Y=0 for X=0. For example,
let Y be the volume of beer sales in the supermarkets,
and X be the number of beer bootless stocked in these
supermarkets. If X=0, then obviously Y=0. In such cases,
the model can be given as in Equation (1).
for i=1,2,,n and
(1)
Here both 1 and are unknown, and Xs are fixed values.

Yi 1X i i

i ~ N(0, )

30

Problems
a) Find the least squares estimator of 1 for
the model given in Equation (1).
b) Find the maximum likelihood estimator of
1 for the model given in Equation (1).
c) Find the expected value

d) Find the variance of 1 .

of 1

31

Problems
e) In an experiment involving 12 similar
branch offices of a bank, new customers
were offered gifts for opening accounts.
The value of the gift was proportional to
the minimum cash deposited in that
branch. We would like to study whether
such gifts were helpful in bringing new
accounts. The following table provides the
data.
32

Branch

Problems
Size of Minimum Deposit Number of New Accounts

(dollars)
1
20
114
2
115
560
3
180
550
4
120
520
5
110
475
6
90
625
7
130
666
8
88
589
9
56
446
10
170
690
11
120
393
12
160
762
Calculate the estimate of 1 for this data. Write down the estimated
regression equation given in Equation (1) for this data. Interpret your
findings.

33