ELEC6111: Detection and Estimation Theory

Minimax Hypothesis Testing

In deriving Bayes decision rule, we assumed that we know both the a priori
probabilities
0
t and
1
t as well as the likelihoods ) | (
0
H y p and ) | (
1
H y p . This
means that we have both the knowledge of the mechanism generating the state of the
nature and the mechanism affecting our observations (our measurements about the
state of nature). It is, however, possible that we may not have access to all the
information. For example, we may not know the a priori probabilities. In such a case,
the Bayes decision rule is not a good rule since it can only be derived for a given a
priori probability.
An alternative to the Byes hypothesis testing, in this case, is the Minimax
Hypothesis testing. The minimax decision rule minimizes the maximum possible risk,
i.e., it minimizes,
{ } ) ( ), ( max
1 0
o o R R

over all
o
.
Lets look at
) , (
0
o t r
, i.e., the overall risk for a decision rule
o
when the a priori
probability is
] 1 , 0 [
0
e t
. It can be written as:
) ( ) 1 ( ) ( ) , (
1 0 0 0 0
o t o t o t R R r + =



ELEC6111: Detection and Estimation Theory
Minimax Hypothesis Testing
Note that, for a given decision rule o , as
0
t varies from 0 to 1, ) , (
0
o t r goes
linearly from ) , 0 ( ) (
1
o o r R = to ) , 1 ( ) (
0
o o r R = . Therefore, for a given decision rule
o the maximum value of ) , (
0
o t r as
0
t varies over the interval ] 1 , 0 [ occurs either at
0
0
= t or 1
0
= t and it is )} ( ), ( max{
1 0
o o R R .
So minimizing )} ( ), ( max{
1 0
o o R R is equivalent to minimizing ) , ( max
0
1 0
0
o t
t
r
s s
.
Thus, the minimax decision rule is:
) , ( max min
0
1 0
0
o t
t o
r
s s

Let
0
t
o denote the optimum (Bayes) decision rule for the a priori probability
0
t .
Denote the corresponding minimum Bayes risk by ) (
0
t V , i.e., ) , ( ) (
0 0
o t t r V = .
It is easy to show that ) (
0
t V is a continuous concave function of
0
t for
] 1 , [
0
o e t and has the end points
11
) 0 ( C V = and
00
) 1 ( C V = .

ELEC6111: Detection and Estimation Theory
Minimax Hypothesis Testing
The following figure shows a typical graph of ) (
0
t V and
) , (
0
o t r . Lets draw a tangent to ) (
0
t V parallel to ) , (
0
o t r .
Denote this line by ) , (
0
0 t
o t
'
r .


ELEC6111: Detection and Estimation Theory
Minimax Hypothesis Testing
Since ) , (
0
0 t
o t
'
r lies entirely below the line ) , (
0
o t r , it has a lower maximum
compared to ) , (
0
o t r . Also note that since it touches ) (
0
t V at
0 0
t t ' = then
0
t
o
'
is the
minimum risk (Bayes) rule for a priori probability
0
t ' . Since for any ] 1 , 0 [
0
e t we can
draw a tangent to ) (
0
t V and find the minimum risk rule as a Bayes rule, it is clear that
the minimax decision rule is the Bayes rule for the value of
0
t that maximizes ) (
0
t V .
Denoting this point by
L
t , we note that point,
) ( ) ( )} ( ), ( max{
1 0 1 0
L L L L
R R R R
t t t t
o o o o = =


ELEC6111: Detection and Estimation Theory
Minimax Hypothesis Testing
Proposition: The Minimax Test

Let
L
t be the a priori probability that maximizes ) (
0
t V and such that either 0 =
L
t , or 1 =
L
t , or
), ( ) (
1 0
L L
R R
t t
o o =
then
L
t
o is a minimax rule.


ELEC6111: Detection and Estimation Theory
Minimax Hypothesis Testing
Proof

Let ), ( ) (
1 0
L L
R R
t t
o o = then for any
0
t we have,
), , ( ) , ( ) , ( min max
0 0
1 0
0
L L
r r r
L
o t o t o t
t
o t
= =
s s

So, we have
), , ( max min ) , ( max ) , ( min max
0
1 0
0
1 0
0
1 0
0 0 0
o t o t o t
t o
t
t o t
r r r
L
s s s s s s
> =
Also, for each o we have
). , ( min max ) , ( max
0
1 0
0
1 0
0 0
o t o t
o t t
r r
s s s s
>
This implies that,
). , ( min max ) , ( max min
0
1 0
0
1 0
0 0
o t o t
o t t o
r r
s s s s
>
Combining the two inequalities, we get,
. ) , ( min max ) , ( max min
0
1 0
0
1 0
0 0
o t o t
o t t o
r r
s s s s
=
Therefore,
. ) , ( max min ) , (
0
1 0
0
o t o t
t
o
t
r r
L
L
s s
=
That is,
L
t
o is the minimax rule.
ELEC6111: Detection and Estimation Theory
Minimax Hypothesis Testing
Discussion

By definition: ) , ( ) (
0
0 0 t
o t t r V = . So, for every ] 1 , 0 [
0
e ' t , we have ) ( ) , (
0
0
0
t
t
o t V r >
'
and
). ( ) , (
0
0
0
t
t
o t ' =
'
V r Since ) , (
0
0
t
o t
'
r , as a function of
0
t is a straight line, it has to be tangent to
) (
0
t V at .
0 0
t t ' = If ) (
0
t V is differentiable at
0
t ' , we have,
). ( ) ( / ) , ( ) (
0 0 0
1 0 0 0 0 t t t
o o t o t t
' ' '
= = ' ' R R d dr V
Now consider the case that ) (
0
t V has an interior maximum but is not differentiable at that point. In
this case we define two decision rules
0 0
lim
t t t t
o o
L L |

= and . lim
0 0
t t t t
o o
L L +
+
=
The critical regions for these two decision rules are,
), | ( ) ( ) | ( ) )( 1 ( | {
0 10 00 1 01 11 1
H y p C C H y p C C y
L L
s I e = I

t t
and
), | ( ) ( ) | ( ) )( 1 ( | {
0 10 00 1 01 11 1
H y p C C H y p C C y
L L
< I e = I
+
t t
Take a number ] 1 , 0 [ e q and devise a decision rule
~
L
t
o that uses the decision rule

L
t
o with probability
qand uses
+
L
t
o with probability q 1 . It means that it decides
1
H if
+
I e
1
y , decides
0
H if
c
y ) (
1

I e and
decides
1
H with probability q if y is on the boundary of

I
1
.




ELEC6111: Detection and Estimation Theory
Minimax Hypothesis Testing
Discussion
Note that the Bayes risk is not a function of q , so ) ( ) , (
~
L L
V r
L
t o t t = ' but the conditional risks
depend on q,
). ( ) 1 ( ) ( ) (
~
+
+ =
L L
L j j j
R q qR R
t t
t o o o
To achieve ) ( ) (
~
1
~
0
L L
R R
t t
o o = , we need to choose,
.
) ( ) ( ) ( ) (
) ( ) (
1 0 1 0
1 0
+ +
+ +
+

=
L L L L
L L
R R R R
R R
q
t t t t
t t
o o o o
o o


Note that ) ( ) ( ) (
0 0

= '
L L
R R V
L t t
o o t , so we have:
.
) ( ) (
) (
+
+
' '
'
=
L L
L
V V
V
q
t t
t


This is called a randomized decision rule.
ELEC6111: Detection and Estimation Theory
Example: Measurement with Gaussian Error


Consider the measurement with Gaussian error with unifom costs

The function ) (
0
t V can be written as,
)), ( 1 )( 1 ( ) ( ) (
1
0
0
0 0
o
t
t
o
t
t t
'
+
'
= Q Q V
With
.
2
)
1
log(
1 0
0
0
0 1
2

t
t

o
t
+
+

= '
We can find the rule making conditional risks ) (
0
o R and ) (
1
o R equal by letting,
)) ( 1 ( ) (
1 0
o
t
o
t '
=
'
Q Q
and solving for t ' .
ELEC6111: Detection and Estimation Theory
Example: Measurement with Gaussian Error
We can solve this by inspection and get:
.
2
1 0

t
+
= '
L

So, the minimax decision rule is:

+ <
+ >
=
. 2 / ) ( 0
2 / ) ( 1
) (
1 0
1 0


o
t
y if
y if
y
L


Conditional risks for measurement with Gaussian error

ELEC6111: Detection and Estimation Theory
Neyman-Pearson Hypothesis Testing
In Bayes hypothesis testing as well as minimax, we are concerned with the average risk, i.e., the
conditional risk averaged over the two hypotheses. Neyman-Pearson test, on the other hand, recognizes the
asymmetry between the two hypotheses. It tries to minimize one of the two conditional risks with the other
conditional risk fixed (or bounded).
In testing the two hypotheses
0
H and
1
H , the following situations may arise:
-
0
H is true but
1
H is decided. This is called a type I error or a false alarm. This comes from radar
application where
0
H represents no target and
1
H is the case of target present. The probability
of this event is called false alarm probability or false alarm rate and is denoted as ) (o
F
P
-
1
H is true but
0
H is decided. This is called a type II error or a miss. The probability of this event is
called miss probability and is denoted as ) (o
M
P
-
0
H is true and
0
H is decided. Probability of this event is ) ( 1 o
F
P .
-
1
H is true and
1
H is decided. This case represents a detection. The detection probability is
) ( 1 ) ( o o
M D
P P = .

In testing
0
H versus
1
H , one has to tradeoff between the probabilities of two types of errors. Neyman-
Pearson criterion makes this tradeoff by bounding the probability of false alarm and minimizing miss
probability subject to this constraint, i.e., the Neyman-Pearson test is,
) ( max o
o
D
P subject to , ) ( o o s
F
P
where o is the bound on false alarm rate. It is called the level of the test.
ELEC6111: Detection and Estimation Theory
Neyman-Pearson Hypothesis Testing
For obtaining a general solution to the Neyman-Pearson test, we need to define a randomized decision
rule. We define the randomized test,

<
=
>
=
L
L
L
y L if
y L if q
y L if
y
L
t
t
t
o
t
) ( 0
) (
) ( 1
) (
~

where
L
t is the threshold corresponding to
L
t .
While in a non-randomized rule, ) ( y o gives the decision, in a randomized rule, ) (
~
y
L
t
o gives the probability of
decision.
Then we have,
}
I
= = , ) | ( ) ( )} ( { ) (
0
~ ~
0
~
dy H y p y Y E P
F
o o o
where {.}
0
E is expectation under hypothesis
0
H . Also,
}
I
= = . ) | ( ) ( )} ( { ) (
1
~ ~
1
~
dy H y p y Y E P
D
o o o
ELEC6111: Detection and Estimation Theory
Neyman-Pearson Lemma
Consider a hypothesis pair
0
H and
1
H :
0 0
~ : P Y H
Versus
1 1
~ : P Y H
where
j
P has density ) | ( ) (
j j
H y p y p = for 1 , 0 = j . For 0 > o , the following statements are true:
1. Optimality: Let
~
o be any decision rule satisfying . ) (
~
o o s
F
P Let
~
o ' be any decision rule of the form
) (
), | ( ) | ( 0
) | ( ) | ( ) (
) | ( ) | ( 1
0 1
0 1
0 1
~
A
H y p H y p if
H y p H y p if y
H y p H y p if

<
=
>
= '
q
q
q
o
where 0 > q and 1 ) ( 0 s s y are such that . ) (
~
o o = '
F
P Then ). ( ) (
~ ~
o o
D D
P P > '
This means that any size- decision rule of form (A) is Neyman-Pearson rule.
2. Existence: For any ) 1 , 0 ( e o there is a decision rule, NP
~
o , of form (A) with
0
) ( = y for which
. ) (
~
o o = NP
F
P
3. Uniqueness: Suppose that o ' ' is any Neyman-Pearson rule of size- for
0
H versus
1
H . Then o ' ' must be of
the form (A).
ELEC6111: Detection and Estimation Theory
Neyman-Pearson Lemma (Proof)
1. Not that, by definition, we always have 0 )] | ( ) | ( )][ ( ) ( [
0 1
~ ~
> ' H y p H y p y y o o (why?)
So, we have,
. 0 )] | ( ) | ( )][ ( ) ( [
0 1
~ ~
> '
}
I
dy H y p H y p y y o o
Expanding the above expression, we get,
. ) | ( ) ( ) | ( ) ( ) | ( ) ( ) | ( ) (
0
~
0
~
1
~
1
~
} } } }
I I I I
(

' > ' dy H y p y dy H y p y dy H y p y dy H y p y o o q o o

Applying the expressions for the detection probability and false alarm rate, we have:
. 0 )] ( [ )] ( ) ( [ ) ( ) (
~ ~ ~ ~ ~
> = ' > ' o o q o o q o o
F F F D D
P P P P P

2. Let
0
q be the smallest number such that (look at the Figure in next slide):
o q s > )] | ( ) | ( [
0 0 1 0
H Y p H Y p P .
Then if o q < > )] | ( ) | ( [
0 0 1 0
H Y p H Y p P , choose,
)] | ( ) | ( [
)] | ( ) | ( [
0 0 1 0
0 0 1 0
0
H Y p H Y p P
H Y p H Y p P
q
q o

=
>
=
Otherwise, choose
0
arbitrarily. Consider a Neyman-Pearson decision rule, NP
~
o , with
0
q q = and
0
) ( = y . For this
decision rule, the false alarm arte is,
o q q o o = = + > = = )] | ( ) | ( [ )] | ( ) | ( [ } { ) (
0 0 1 0 0 0 0 1 0
~
0
~
H Y p H Y p P H Y p H Y p P E P NP NP
F
.


ELEC6111: Detection and Estimation Theory
Neyman-Pearson Lemma (Proof)
.




2. See the text.

ELEC6111: Detection and Estimation Theory
Neyman-Pearson Lemma (Example): Measurement with Gaussian Error
For this problem, we have,
), ( 1 ) ( ) ( ) ) ( [ )] | ( ) | ( [
0 0 0 1 0
o
q q
|
o
q q
q q q
'
=
'
= ' > = > = > Q Y P y L P H Y p H Y p P
where .
2
) log(
1 0
0 1
2


q
o q

+

= '

Any value of o can be achieved by choosing,
. ) 1 ( ) (
0
1
0
1
0
o o| o o q + = + = '

Q
Since 0 ) (
0
= ' =q Y P , the choice of
0
is arbitrary and we can choose 1
0
= . So, we have

' <
' >
=
. 0
1
) (
0
0
~
q
q
o
y if
y if
y NP

ELEC6111: Detection and Estimation Theory
Neyman-Pearson Lemma (Example): Measurement with Gaussian Error
The detection probability for NP
~
o is
( ) d Q Q Q Q Q Y P Y E P NP NP
D
= |
.
|

\
|
=
'
= ' > = =

) ( ) ( ) ( ) ( )} ( { ) (
1 0 1 1 1 0
0 1
~
1
~
o
o

o
o
q
q o o