Lippincott Williams & Wilkins is collaborating with JSTOR to digitize, preserve and extend access to
Epidemiology.
http://www.jstor.org
forMultiple Comparisons
Rothman
Kenneth].
for making
Adjustments
readily. Unfortunately,
null. The
theoretical
serves
"chance"
follows
laws that may
be studied
holds
that nature
A policy
of not making
observations.
regular
through
is preferable
it will
because
lead to fewer errors of interpretation
for multiple
when
the data under
comparisons
on nature.
are not random numbers
scientists
to explore
but actual observations
should not be so reluctant
Furthermore,
turn out to be wrong
that they penalize
themselves
that may
by missing
important
possibly
findings.
(Epidemiology
which
research,
adjustments
evaluation
leads
1990;1:43-46)
Keywords:
null
comparisons,
multiple
hypothesis,
significance
Scientists
exacerbated
the
when
unusual
result
main
or
focus
is one
In many
amines.
of many
error.
to measurement
ascribed
result
In other
to the
ex
a study
that
unexpected
to
pertain
incidental
relations
an
instances
not
does
can
be
an
situations
researcher
to
whether
ignore
an
unanticipated
pretation.
In
its most
problem
testing.
common
is closely
Under
that
hypothesis
statistical
two
factors
significance
are
unre
Editor,
Epidemiology
?1990
Epidemiology
statistics.
dent
associations
are
for
examined
statistical
signifi
In practice,
is often
much
larger:
for
example,
Gardner
(1) examined
5000
the multiple-comparison
guise,
linked with
testing,
if they
about
peculiar
as
to
opposed
are
correct.
conducting
a single
Of
course,
a multitude
comparison,
there
of
that
is nothing
comparisons,
increases
the
problem
comparisons.
Resources
Inc.
43
ROTHMAN
incorrect
an
factors,
error
that
no
assert
that
occur
can
relation
when
an
two
between
or
test.
diagnostic
In screening,
the
predic
information
on
data
those
associations.
asso
An
may
comparisons
have
no
bearing
on
the
one
in ques
boils
finding.
This
to
this:
alone
chance
statement
does
not
can
carry
cause
any
the
unusual
obvious
adjustment
sta
Chance
2.
No
ings.
there
presumption 1, as already demonstrated,
would be no need for corrective statistical action, and
therefore this presumption is fundamental to the theory
of adjustments for multiple comparisons. Presumption 2
is inherent in the understanding of chance that under
Not
Only
Can
Cause
and
chewing
occurrence
the
of brain
cancer,
would
requires
an
extension
of
the
concept
of
the
between
any
pair
of
variables
under
ob
only
variation.
random
is not
of the ordinary null hypothesis
necessary for any statistical analysis, since it is always
for each
possible to rely on a separate null hypothesis
This
Without
is appropriate.
comparisons
1: Chance
Presumption
no
1.
for multiple
in
association
pair
of
extension
variables.
Yet,
the
generalization
to
a universal
44
Epidemiology
1Number
MULTIPLECOMPARISONS
variables
that
may
not
be
to
related
one
no
another,
causal
entertain
To
the
universal
null
hy
multiple
are most
recom
enthusiastically
is
the tenability of a universal null hypothesis
mended,
most farfetched. In a body of data replete with associa
tions, itmay be that some are explained by what we call
"chance," but there is no empirical justification for a
that all the associations are unpredictable
hypothesis
manifestations
of random processes. The null hypothesis
relating a specific pair of variables may be only a statis
tical contrivance,
but at least it can have a scientific
counterpart that might be true. A universal null hypoth
esis implies not only that variable number six is unre
lated to variable number 13 for the data in hand, but
also that observed phenomena exhibit a general discon
nectivity that contradicts everything we know.
The
is
untenability of this universal null hypothesis
over
in
the
of
proce
always
presentation
nearly
skipped
dures to deal with multiple comparisons. Teachers of
statistics
even
sometimes
lapse
into
a tacit
of
acceptance
sory perception
easy
to theorize
from biases
the
results
of
studies
on
extrasen
nonrandom
associations
might
arise
even
Epidemiology
1Number
1 45
it is always
Instead,
on
association
its own
to consider
reasonable
for
the
each
it conveys.
information
or observation
of each association
that
to attention.
to
Earmark
for Further
Investigation
Something
Caused
by Chance
Chance is a term often used as if its meaning were well
it is taken to denote a mysteri
understood. Commonly
ous
force
that
introduces
variation
random
into
observ
able phenomena,
and, indeed, I have used the term in
sense up to this point. Nevertheless,
this conventional
it
is important to scrutinize the concept. The Oxford En
glish Dictionary gives 13 definitions for the noun chance
(4). The first is "the falling out or happening of events."
The sixth definition comes closest to the statistical and
scientific usage: "absence of design or assignable cause;
often itself spoken of as the cause or determiner of
appear to happen without
events, which
tion
of
law,
common
as an
in statistics
explains
Despite
to "chance"
science
randomness
term
the
associations,
The
nothing.
the interven
providence."
and
for observed
"explanation"
chance
or
causation
ordinary
reference
asso
usually
not
does
enhance
the
vacuous
that
explanation
explanations
obscure.
these
Nevertheless,
ex
other
and meaningful,
and
planations may be discoverable
should not necessarily be ignored.
The inherent unpredictability
of chance phenomena
would seem to preclude meaningful
research on such
Randomness,
phenomena.
routinely
omitted.
hypothesis
ductive.
causal
perhaps
as
classified
(5). What
explanations
we
rolls,
may be unexpected
and usually could have
coin
the
result
and
tosses,
outcome.
We
because
the
causal
to as a "chance"
refer
random-number
describe
that
have
or unusual, but it is
been prevented. Dice
to known physical
have according
occurrences
"chance"
encounter
caused
a theoreti
is only
however,
the
be
generators
explanations
as
are
too
for
chance
intricate,
the outcome
it may
happen
that
slight
differences
in the
initial
con
ROTHMAN
ditions
We
vations
we
that
term
to
chance
counted
currence
cannot
that
variability
might
for with
of
cancer
lung
as a chance
entirely
so, we
In doing
predict.
connote
once
may
in obser
the variability
phenomenon,
can
the
be
ac
the oc
been
have
we
use
viewed
now
explain
to be
taken
state,
since
knowl
advancing
intensive
it defeats
findings,
In
recent
some
shields
comparisons
tiple
associations
observed
paper
multiple
the
comparisons,
unless
Thus,
the investigator
of multiplicity,
extreme
(and
by the seemingly
is taken
may be mistakenly
impressed
thus seemingly
rare) result.
sistencies
arise.
as
esis
at
true,
least
as
tention
starting
however,
nature
that
an
with
at
follows
extreme
every
regular
observation
to
opportunity
laws,
the
scientist
it
constantly
Therefore
no
is
inherent
to
the
trial-and-error
process
of
on
information
future,
is a concept
is commonly
that
The
accepted.
to any
unacceptable
cause
It lacks
refutes.
any
apparent
for
Science
empiricist.
of comparisons,
heuristic
value.
comprises
alarm.
to
the
1. Gardner
grasp
than
in
confronted
rather
the
References
scientist,
should
sometime
when,
formation
to explain natural
scientist presumes
or association
understand
point.
studies
be
who
investigator
multitude
By claiming that to be impressed by the extreme result is
a mistake,
the writer accepts the universal null hypoth
an
Imagine
John Wiley
4. The Oxford
1971.
sci
5. Kolata
and Sons,
English
G. What
1985.
dictionary.
does
it mean
Oxford:
to be
Oxford
University
Press,
Science
1986;
random?
231; 1068-70.
In: Newman
6. Poincar? JH: Chance.
JR, ed. The world of mathe
matics. Vol 2. New York: Simon and Schuster,
1956:1380-94.
the means of several groups. N Engl JMed
7. Godfrey K. Comparing
1985;313:1450-6.
8. Burke M. The scientific
Science
method.
1986;231:659.
9. Light RJ, Pillemer DB. Summing
up. The science of reviewing
research. Cambridge: Harvard University
Press, 1984.
46
Epidemiology
1Number