Statistics 550 Notes 8

Reading: Section 1.6.1-1.6.4

I. Correction on Minimal Sufficiency
The statement of Theorem 2 in Notes 7 was wrong. The
correct statement is
Theorem 2 (Lehmann and Scheffe, 1950): Suppose S ( X ) is
a sufficient statistic for . Also suppose that if for two
sample points x and y , the ratio f ( x | ) / f ( y | ) is
constant as a function of , then S ( x ) S ( y ) . Then
S ( X ) is a minimal sufficient statistic for .
Proof: Let T ( X ) be any statistic that is sufficient for . By
the factorization theorem, there exist functions g and h
such that f ( x | ) g (T ( x) | ) hx( ) . Let x and y be any
two sample points with T ( x ) T ( y ) . Then
f ( x | ) g (T ( x ) | ) h( x ) h( x )

.
f ( y | ) g (T ( y ) | )h( y ) h( y )
Since this ratio does not depend on , the assumptions of
the theorem imply that S ( x ) S ( y ) . Thus, S ( X ) is at
least as coarse a partition of the sample space as T ( X ) , and
consequently S ( X ) is minimal sufficient.
Example 1: Consider the ratio

x
n
x
i1 i (1 ) i1 i
n

f (x | )

.
n
n
f ( y | ) i1 yi (1 ) n i1 yi
If this ratio is constant as a function of , then we must
n
n
have i 1 xi i 1 yi . Since we have shown that
n

T ( X ) X i is a sufficient statistic, it follows from the

i 1

above sentence and Theorem 2 that T ( X ) X i is a

i 1

minimal sufficient statistic.

II. Exponential Families
The binomial and normal models exhibited the interesting
feature that there is a natural minimal sufficient statistic
whose dimension is independent of the sample size. The
exponential family models are a general class of models
that exhibit this feature.
The class of exponential family models includes many of
the mostly widely used statistical models (e.g., binomial,
normal, gamma, Poisson, multinomial). Exponential
family models have an underlying structure with elegant
properties that we will discuss.
One-parameter exponential families: The family of
distributions of a model {P : } is said to be a oneparameter exponential family if there exist real-valued
2

functions ( ), B( ), T ( x ), h( x ) such that the pdf or pmf

may be written as
p ( x | ) h( x ) exp{ ( )T ( x ) B( )}
Comments:
(1) For an exponential family, the support of the
distribution (i.e., { x : p( x | ) 0} ) cannot depend on .
Thus, X 1 ,K , X n iid Uniform (0, ) is not an exponential
family model.
(2) For an exponential family model, T ( x ) is a sufficient
statistic by the factorization theorem.
(3) , B, T are not unique. For example, can be
multiplied by a constant c and T can be divided by the same
constant c.
Examples of one-parameter exponential family models:
(1) Poisson family.
Let X ~ Poisson( ), 0 . Then for x {0,1, 2,...} ,
x e 1
p( x | )
exp{x log } .
x!
x!
This is a one-parameter exponential family with
( ) log , B( ) , T ( x) x, h( x) 1/ x ! .
(2) Binomial family.
Let X ~ Binomial(n, ), 0 1 . Then for
x {0,1, 2,..., n} ,

 n
p( x | ) x (1 ) n x
x
n

exp[ x log
n log(1 )]
x
1

This is a one-parameter exponential family with
n

( ) log
,
B
(

n
log(1

),
T
(
x
)

x
,
h
(
x
)

1
x
The family of distributions obtained by taking iid samples
from one-parameter exponential families are themselves
one-parameter exponential families.
Specifically, suppose X ~ P and {P : } is an
exponential family, then for X1 , K , X n iid with common
distribution P ,

h
(
x
)
exp

)
T ( xi ) nB( )

i
i

i 1

p( x1 , K , xn | )

A sufficient statistic is

n
i 1

T ( xi ) and it is one dimensional

whatever the sample size n is.

For X 1 ,K , X n iid Poisson ( ), the sufficient statistic

n
i 1

T ( xi ) has a Poisson ( n ) distribution and hence has an

exponential family model. It is generally true that the

sufficient statistic of an exponential family model follows
an exponential family.
Theorem 1.6.1: Let {P : } be a one-parameter
exponential family of discrete distributions:
4

p( x | ) h( x ) exp{ ( )T ( x ) B ( )}
Then the family of the distributions of the statistic T ( X ) is
a one-parameter exponential family of discrete distributions
whose pdf may be written
h *(t ) exp{ ( )t B ( )}
for suitable h*.

Proof: By definition,
P [T ( x) t ] p ( x | )
{ x:T ( x ) t }

h( x) exp[ ( )T ( x) B( )]

{ x:T ( x ) t }

exp[ ( )t B( )]{
*
h
If we let (t )

{ x:T ( x ) t }

h( x)}

{ x:T ( x ) t }

h( x) , the result follows.

A similar theorem holds for continuous exponential

families.
A useful reparameterization of the exponential family
model is to index ( ) as the parameter to yield
p ( x | ) h( x ) exp[T ( x) A( )] ,
where A( ) log L h( x) exp[T ( x )]dx in the continuous
case and the integral is replaced by a sum in the discrete
space. If , then A( ) must be finite. Let
{ :| A( ) | } . The model given by with ranging
over is called the canonical one-parameter exponential
family generated by T and h. is called the natural
5

parameter space and T is called the natural sufficient

statistic. The canonical one-parameter exponential family
contains the one-parameter exponential family with
parameter space and can be thought of as the
biggest possible parameter space for the exponential
family.
Example 1: Let X ~ Poisson( ), 0 . Then for
x {0,1, 2,...} ,

x e 1
p( x | )
exp{x log }
x!
x!
Letting log , we have
1
p ( x | ) exp{ x exp( )}, x={0,1,2,...} .
x!
We have

1
A( ) log e x
x 0 x !

(e ) x
log
x!
x 0

log exp(e ) e
Thus, { :| A( ) | } .
Note that if 1 , then would still be a one-parameter
exponential family but it would be a strict subset of the
canonical one-parameter exponential family generated by T
and h with natural parameter space { :| A( ) | }
.

A useful result about exponential families is the following

computational shortcut for moments of the natural
sufficient statistic:
Theorem 1.6.2: If X is distributed according to and is an
interior point of , then the moment-generating function
of T ( X ) exists and is given by
M ( s ) E[exp( sT ( X ))]exp[ A(s ) A( )]
for s in some neighborhood of 0.
Moreover,
E [T ( X )] A '( ), Var [T ( X )] A ''( ) .
Proof: This is the proof for the continuous case.
M ( s ) E (exp( sT ( X ))) L

h( x) exp[(s )T ( x) A( )]dx
{exp[ A( s ) A( )]} L h( x) exp[( s )T ( x) A( s )]dx
exp[ A( s ) A( )]

because the last factor, being the integral of a density, is

one. The rest of the theorem follows from the moment
generating property of M ( s ) (see Section A.12 of Bickel
and Doksum).
Comment on proof: In order for the moment generating
function (MGF) properties to hold, the MGF must exist (be
less than infinity) for s in some neighborhood of 0. The
proof that the MGF exists for s in some neighborhood of 0
relies on the fact that is an interval or , which is
established in Section 1.6.4.

Example 1 continued: Let X ~ Poisson( ), 0 .

The natural sufficient statistic is T ( X ) X and log ,
A( ) e . Thus, using Theorem 1.6.2,

E [ X ]

d
e
e

log

d log

d2
Var [ X ] 2 e
e

log

d
log
Example 2: Suppose X 1 ,K , X n is a sample from a
population with pdf
x
x2
p ( x | ) 2 exp( 2 ), x 0, 0

2
This is known as the Rayleigh distribution. It is used to
model the density of time until failure for certain types of
equipment. The data comes from an exponential family:
n
xi2
n xi
p ( x1 ,K , xn | ) 2 exp( 2 )
i 1 2
i 1

1
xi exp( 2
2
i 1
n

x
i 1

2
i

n log 2 )

Here

1
1
2
,

, B( ) n log 2 , A( ) n log(2 ) .
2
2
2
n

Therefore, the natural sufficient statistic

X
i 1

2
i

has mean

A '( ) n / 2n 2 and variance A ''( ) n / 2 4n 4 .

8

Proving that a one parameter family is not an exponential

family
A one parameter exponential family is a family
p ( x | ) h( x ) exp{ ( )T ( x ) B( )} , .
Consider a one parameter family { p ( x | ), } . If the
support of p( x | ) is different for different , then the
family is not an exponential family because p ( x | ) 0 if
and only if h( x ) 0 .
Suppose that the support of p( x | ) is the same for all
. We can write the pdf or pmf of the family as
p ( x | ) h( x ) exp{g ( x, )} .
In order for this to be an exponential family, we need to be
able to write
g ( x , ) ( )T ( x ) B( )
for some functions , B and T .
Suppose holds. Then for any two sample points x1 and x2
,
g ( x1 , ) g ( x2 , ) ( )[T ( x1 ) T ( x2 )] and
for any four sample points x1 , x2 , x3 , x4 ,
g ( x1 , ) g ( x2 , ) T ( x1 ) T ( x2 )

g ( x3 , ) g ( x4 , ) T ( x3 ) T ( x4 )
is constant as a function of .

Thus, a necessary condition for a one-parameter

exponential family is that for any four sample points,
x1 , x2 , x3 , x4 ,
g ( x1 , ) g ( x2 , )
g ( x3 , ) g ( x4 , )
must be constant as a function of .
Proof that the Cauchy family is not an exponential family:
The Cauchy family is
1
p( x | )
(1 ( x ) 2 )

1
exp log
2

(1

(
x

)
exp{ log log[1 ( x ) 2 ]}, , x
Thus, for the Cauchy family,
g ( x , ) log log[1 ( x ) 2 ] .
For any four sample points x1 , x2 , x3 , x4 ,
g ( x1 , ) g ( x2 , ) log[1 ( x1 ) 2 ] log[1 ( x2 ) 2 ]

g ( x3 , ) g ( x4 , ) log[1 ( x3 ) 2 ] log[1 ( x4 ) 2 ]
This is not constant as a function of so the Cauchy
family is not an exponential family.
II. Multiparameter exponential families
One-parameter exponential families have a natural onedimensional sufficient statistic regardless of the sample

10

size. A k-parameter exponential family has a k-dimensional

sufficient statistic regardless of the sample size.
The family of distributions of a model {P : } is said to
be a k-parameter exponential family if there exist realvalued functions 1 ,K ,k , B of such that the pdf or pmf
may be written as
k

p ( x | ) h( x ) exp{ j ( )T j ( x ) B ( )}
j 1

By the factorization theorem, T ( X ) (T1 ( X ),K , Tk ( X )) is

a sufficient statistic.
Example 1: Suppose X 1 ,K , X n is iid N ( , 2 ) . Then
n
( xi ) 2
1
2
p( x | , )
exp{
}
2
2
2
i 1
x n 2
1
1
n

2
i 1 i

exp 2 2 i 1 xi 2 2 2
2

which corresponds to a two-parameter exponential family

n
n
2
with T ( X ) ( i 1 X i , i 1 X i ) .

Example 2: Multinomial. Suppose we observe n

independent trials where each trial can end up in one of k
possible categories {1,...,k} with probabilities
{ p1 , K , pk 1 , pk 1 p1 L pk 1} . Let

11

y1 ( x ),K , yk ( x ) be the number of outcomes in categories

1,...,k in the n trials. Then,
p( x | )

n!
p1 y1 ( x ) L pk yk ( x )
y1 ( x ) L yk ( x )
y1 ( x )

yk 1 ( x )

p1

p
n!

L k 1
pk n

y1 ( x ) L yk ( x ) pk
p k

n!

exp[ y1 ( x) log( p1 / pk ) L y k 1 ( x) log( pk 1 / pk ) n log pk ]

y1 ( x ) L yk ( x )

n!
y1 ( x ) L yk ( x )
k 1

exp[ y1 ( x ) log( p1 / pk ) L y k 1 ( x) log( pk 1 / pk ) n log(1 exp(log

i 1

The multinomial is a (k-1) parameter exponential family

with (log( p1 / pk , K , log( pk 1 / pk )) ,
k 1

T ( x ) y1 ( x ),K , yk 1 ( x ) and A( ) n log(1 exp(i )) .

i 1

Moments of Sufficient Statistics: As with the oneparameter exponential family, it is convenient to index the
family by (1 ,K ,k ) . The analogue of Theorem 1.6.2
that calculates the moments of the sufficient statistics is
Corollary 1.6.1:
A

A
E0 T ( X )
(0 ), K ,
(0 )
k
1

2 A
Var0 T ( X )
(0 )
a b

12

pi
))]
pk

Example 2 continued: For the multinomial distribution,

k 1

E[ y j ( x )]
n log 1 e i
j
i 1

nei
k 1

1 e

i 1

pi
pk

k 1

1
i 1

pi
pk

npi
1
pi
pk
pk
n

k 1

nei e j
i
Cov0 [ yi ( x ), y j ( x )]
n log 1 e
npi p j , i j
k 1
j k
i 2
i 1

(1 e )
i 1

Var0 [ yi ( x )]

i
n
log
1

npi (1 pi ) .
j 2
i 1

k 1

Curved Exponential Families:

A curved exponential family is a family
k

p ( x | ) h( x ) exp{ j ( )T j ( x ) B( )}
j 1

for which dim( ) k .

An exponential family for which dim( ) k is a full

exponential family.
Example of a curved exponential family:
X 1 , K , X n ~ N ( , 2 ) .

13

( xi ) 2
1
exp{
}
2
2
2

p( x | )
i 1

1
1
exp

2
2

2
i
i 1

i 1 i
2

n 2
2
2

1
1
n
1 n
n

exp

2 i 1 i
i 1 i
2

2
2

This is an exponential family with (1/(2 2 ),1/ ) .

The parameter space is a curve:

14