P e r sp ec ti v e s

Perspective
Comparative Effectiveness: Asking The Right
Questions, Choosing The Right Method
Methodological choice should be driven by policy goals.
by Steven M. Teutsch, Marc L. Berger, and Milton C. Weinstein

ABSTRACT: The Medicare Prescription Drug, Improvement, and Modernization Act (MMA)
of 2003 has placed renewed focus on assessing the comparative effectiveness of various
therapeutic options. Unfortunately, all of the evidence needed to fully assess these options
is rarely available to drug formulary decisionmakers. Comparative randomized trials fre-
quently fail to find differences when there indeed are some, while decision-modeling ap-
proaches are more likely to identify differences where there are none. We consider the con-
sequences of these strategies. This paper proposes a framework for using different
methods to assess available evidence. We contend that choosing the appropriate method
can occur only when there are clear policy goals.

T
h e e s c a l at i n g c o s t of medical
care in general, and prescription drugs
in particular, has fostered the demand
for mechanisms to assure that money is being
spent wisely. The Medicare Prescription
Drug, Improvement, and Modernization Act
(MMA) of 2003, the advent of the Academy of
Managed Care Pharmacy’s Format for Formu-
lary Submissions, and Oregon Medicaid’s evi-
dence-based formulary decision process are
manifestations of the increasing demand for
good comparative information on effective-
ness. We discuss the information that is
needed for effective decision making and how
decisionmakers’ needs should inform priori-
ties and the choice of scientific method.
Methods For Assessing Evidence
While evidence-based decision making as
currently practiced may seem like an approach
that should be universally acclaimed, it has im-
portant limitations. Perhaps of greatest impor-
tance is the relative scarcity of randomized
controlled trials (RCTs) to answer critical
questions about comparative effectiveness.
n Randomized controlled trials. When
assessing efficacy, RCTs are considered to be
the gold standard. Even though typically at
least two pivotal RCTs are performed to ob-
tain Food and Drug Administration (FDA)
clearance to market a new drug, these rarely
provide all of the information practitioners
need; they usually compare the new drug to
placebo and frequently use surrogate or inter-
mediate measures of efficacy, such as blood
pressure or low-density lipoprotein (LDL)
cholesterol rather than outcomes such as car-
diovascular mortality. The comparison against
placebo is based on regulatory requirements
and the desire to minimize the uncertainty
surrounding efficacy assessments. Post-

Steven Teutsch (steven_teutsch@merck.com) is executive director, outcomes research and management, at Merck
and Company Inc., in West Point, Pennsylvania. Marc Berger is vice president in that department. Milton Weinstein
is the Henry J. Kaiser Professor of Health Policy and Management in the Department of Health Policy and
Management, Harvard School of Public Health, in Boston, Massachusetts.

128 Ja n u a r y/ Fe b r u a r y 2 0 0 5
DOI 10.1377/hlthaff.24.1.128 ©2005 Project HOPE–The People-to-People Health Foundation, Inc.
 P erspectiv e: E vidence

launch, comparative efficacy studies of alter- evidence-based decision making. The reality is
native treatments employing intermediate that it will never be feasible to perform enough
measures of efficacy are more common, since RCTs to assess the relative effectiveness of all
their completion requires longer time frames important treatment strategies for different
and larger sample sizes and since they may ad- target populations. Moreover, often RCTs can-
dress competitive marketplace issues. Larger, not provide other critical information. For ex-
longer-term RCTs using true health outcomes ample, harms of treatment are frequently
(such as mortality) have been increasingly per- assessed by long-term comprehensive surveil-
formed during the past decade. The impor- lance, and information on impact among
tance of these trials has been dramatically il- groups at different risks can be assessed from
lustrated by their impact on clinical practice observational studies.
and evidence-based guide- Value of certainty. A salient
lines; examples include the characteristic of RCTs is their
“Adhering to a policy
Antihypertension and Lipid- ability to reduce the uncer-
Lowering Treatment to Pre- that technology will tainty surrounding estimates
ve nt Heart Attack Tr ial be adopted only of efficacy. Thus, RCTs are
(ALLHAT), the Women’s when there is most needed where decision-
Health Initiative, and the unambiguous makers require a high level of
Heart Protection Study. evidence would delay certainty and provide the
Limitations. Even so, groups greatest value where the bur-
the adoption of new
developing guidelines that den of illness (including cost)
demand evidence of the high- technologies by as well as potential benefits,
est quality often find that a many years.” risks, and costs of interven-
sufficient number of studies tions are high. In other cir-
of important clinical ques- cumstances, observational
tions are not available. This situation will studies, systematic reviews, and decision mod-
never disappear, as the full range of benefits els may provide enough certainty for decision
and risks associated with therapeutic deci- making, and results may be available more rap-
sions across the range of potential clinical ap- idly if new studies are needed.1 The level of cer-
plications is not known until long after the tainty required for particular efficacy judg-
technologies have been widely adopted. In ments must be determined by decisionmakers
addition, high-quality RCTs are frequently and stakeholders. Methodologists can then as-
limited by their focus on carefully selected, ad- sess the strategies that can most efficiently
herent populations, which maximizes the op- provide the requisite information.
portunity to demonstrate benefits, and they The subjectivity factor. Subjectivity is intro-
potentially underestimate harms when used duced into all decision making, even when it is
by patients and in settings more typically en- based on evidence. This reflects in part differ-
countered in real-world practice. ences among various stakeholders as to the
Moreover, the majority of studies are de- level of certainty that should be required for
signed with intermediate measures as the trial particular decisions—differences based on
endpoints and do not provide information on variations in preferences and values. These le-
the critical health outcomes that really interest gitimate differences of opinion can be ad-
decisionmakers. Thus, extrapolation from one dressed only by assuring that both delibera-
population to another, from efficacy studies to tions and decision processes are transparent,
effectiveness (that is, results that would be ex- issues are fully vetted, and conflicts of interest
pected in more typical practice settings), or are addressed.2 In doing so, the assumptions
from one technology to another is usually re- underlying projections of risks, benefits, and
quired. This introduces additional uncertainty, costs should be clearly identified, so that all
which belies the aura of rigor associated with stakeholders can provide appropriate input.

H E A L T H A F F A I R S ~ Vo l u m e 2 4 , N u m b e r 1 129
P e r sp ec ti v e s
When best available evidence suffices. Although
this is not ideal, important decisions—such as
coverage decisions—must be based on the best
available information. Delaying decisions until
definitive information is available is usually
not an option. Indeed, Mark McClellan, ad-
ministrator of the Centers for Medicare and
Medicaid Services (CMS), has said that the
government needs to find alternative ap-
proaches to obtaining information in a timely
fashion, including use of systematic reviews
and data from real-world clinical practice.3
n Observational studies. Although RCTs
are not available for many key effectiveness
questions, a wealth of useful information does
exist in observational studies and other
sources. It can be assembled through rigorous
literature synthesis, including meta-analysis.
The full spectrum of information can then be
incorporated into a formal analytic framework,
such as a decision tree or Markov model, which
can be used to assess the benefits, harms, and
costs of alternative interventions.4 Thus, ana-
lysts can extrapolate from one population, time
frame, or technology to another. They can take
advantage of expert opinion and make reason-
able assumptions to link diverse information.
Organizing and combining this information al-
lows the net benefit of alternatives to become
apparent. The risks associated with this ap-
proach arise from assumptions that are not well
founded and from combining information that
turns out to be inaccurate or inappropriate.
Transparency and conflicts of interest. The issues
of transparency and conflicts of interest may
be more acute for decision models because of
the many obvious and less-than-obvious
choices that are made. For example, extrapola-
tion from one therapeutic agent to another be-
cause of similar effects on surrogate markers
(for example, LDL cholesterol, based on as-
sumptions of analogous mechanisms of action)
may ignore effects on other important out-
comes. The harms associated with cerivastatin
(Baycol) led to its recall in 2001, even though it
was in the same class (statins) as other highly
effective drugs with proven outcomes.5
Comparison with RCTs. When RCTs are avail-
able, the specificity of an evidence-based stan-
130
dard is very high, and false-positive conclu-
sions are uncommon. However, this highly
specific standard carries a price of reduced
sensitivity. Adhering to a policy that technol-
ogy will be adopted only when there is unam-
biguous evidence that it has an appropriate
benefit, risk, and cost profile would delay the
adoption of new technologies by many years
and, while it might stimulate some additional
studies, would likely discourage continued
medical innovation and public health benefit.
Use of systematic evidence reviews can fill the
information gap and support more rapid adop-
tion and dissemination of medical innovations,
but with additional risk.
A strict requirement for good-quality RCTs
minimizes the risk of a “Type I error” or “error
of commission.” If multiple high-quality RCTs
find that “drug A” is safer and more effective
than “drug B,” one can have high confidence
that this is so. Because detailed data are often
not available on drug A and drug B for specific
subpopulations, however, advantages of drug
B for those patients may not be identified by
RCT-level evidence. Thus the probability of a
“Type II error,” or “error of omission,” may be
large. Real differences among alternatives may
not be found because there are insufficient
studies available. Conversely, by admitting
more varied information, observational studies
and decision modeling can often elucidate real
differences even though head-to-head RCTs are
not available or have limitations. This informa-
tion comes at a price, however. While the risk
of a Type II error may be small, there is a very
real possibility of making Type I errors, that is,
finding differences when none exists.
Toward A Coherent Policy
Approaching a coherent policy regarding
how to address the lack of critical evidence to
assess comparative effectiveness requires a
more nuanced approach and should be tai-
lored to the situation. For someone with a seri-
ous health condition such as cancer, for which
long-term comparative treatment trials are not
available, patients are generally willing to
choose treatment based on imperfect informa-
tion. Conversely, for a preventive service in the
Ja n u a r y/ Fe b r u a r y 2 0 0 5

asymptomatic population—such as screening
for the breast cancer (BRCA) gene, followed
by treatment with tamoxifen—one would ex-
pect a high degree of certainty that benefits
outweigh harms before subjecting the popula-
tion to the service. In such a case, a strict re-
quirement of good-quality RCT evidence may
make sense.
Whether more evidence should be ob-
tained at any stage ought to depend on
whether the value of the information obtained,
in terms of getting people on or off treatment,
outweighs the cost and time of conducting the
necessary studies. It will be important to rec-
ognize the trade-offs among different method-
ological approaches to informed decision mak-
ing, so that choices made will be as closely
aligned with intended goals as possible. This
will require an open dialogue with all stake-
holders, as others have described.6
When decisions are made on imperfect in-
formation, processes need to be in place to re-
evaluate those decisions as new information
becomes available, and decisionmakers need to
consider the costs and benefits of changing
those policies once adopted. The recent
decision by the CMS on narrow coverage of
positron emission tomography (PET) scans for
Alzheimer’s disease represents a provisional
decision, which is based on available data and
which will be revisited and revised after a clin-
ical trial has been completed.
n Comparative effectiveness of drugs.
Over the past few years, interest in the com-
parative effectiveness of drugs has increased.
The Oregon Medicaid program has contracted
with the Oregon Health and Science Univer-
sity (OHSU) Evidence-based Practice Center
(EPC) to conduct evidence reviews for use in
Medicaid formulary decisions. The EPC uses
analytic frameworks and evidence-based
methods adapted from the U.S. Preventive Ser-
vices Task Force (USPSTF).7 Additional states
will be using these reviews for their Medicaid
formularies. MMA directs the CMS to work
with the Agency for Healthcare Research and
Quality (AHRQ) on conducting studies of
comparative effectiveness.
Because outcomes trials often take many
H E A L T H A F F A I R S ~ Vo l u m e 2 4 , N u m b e r 1
P erspectiv e: E vidence
years to complete and existing resources allow
for only a limited number of head-to-head
comparative outcomes studies, there has been
increasing interest in using observational data
from cohort studies, claims data, and a hoped-
for electronic medical record to complement
RCTs. This call is tempered by the under-
standing that even well-done observational
studies can lead to erroneous conclusions. The
experience with combination hormone ther-
apy is a recent example, in which observa-
tional studies suggested substantial cardiovas-
cular benefits. Only with the recent Women’s
Health Initiative, a large randomized trial, was
the overall net increase in harms apparent.
One policy goal should be to ensure that ad-
equate data are developed to assess the true
benefit-risk-cost profiles of new therapies.
Given the growing number of examples where
“true outcome” RCTs have yielded results un-
favorable to the manufacturers that were the
study sponsors, it is an open question whether
such studies will routinely be performed in the
future. To encourage this, incentives could be
provided to manufacturers, such as giving for-
mulary preference to drugs with endpoint out-
come data or extensive drug experience, or
both. There could be more stringent require-
ments for information regarding effectiveness
or safety, or both, from new entrants into an
established class. Alternatively, these studies
will need to be funded through the National
Institutes of Health (NIH), AHRQ, or other
governmental sources.
n Criteria for government priority list.
The government will need a systematic ap-
proach to prioritizing what studies need to be
done. MMA calls for AHRQ to develop a prior-
ity listing of where additional research is
needed; AHRQ has conducted hearings and is
considering various criteria.
We propose the following criteria: (1)
What is the value of gaining additional infor-
mation? (2) What do we really need to know
to make a good policy decision regarding the
use of one technology or another in the treat-
ment of a particular health condition? Aligned
with this is the corollary question: (3) How
certain do we need to be about what we know?
131
P e r sp ec ti v e s
How these questions are answered can permit
researchers to decide upon the appropriate
methods to assess comparative effectiveness.
n Choosing the “right” method. It is im-
portant that methodological choices be driven
by policy goals. Strict reliance on good-quality
RCTs may be most appropriate where there
are treatment options with excellent benefit-
risk-cost profiles or where there exist high lev-
els of confidence in the ability to identify real
differences among therapies. These differences
are important. For example, for potentially
hazardous interventions among patients at
low risk of poor outcomes, demanding long-
term outcomes data is an incentive that re-
wards innovators who conduct the required
RCTs to demonstrate the true benefit-risk-
cost profile of a new treatment option.
Alternatively, comparative effectiveness
studies that use observational data and model-
ing may be preferable when addressing thera-
pies for which there are no available treat-
ments with acceptable benefit-risk-cost
profiles or where less certainty in efficacy esti-
mates is required. This may commonly apply
to the development of novel cancer chemo-
therapeutic agents for patients for whom no
therapeutic options exist.
When a decision must be made regardless
of the quality of data available or when it can
be made only when high-quality data are avail-
able, “choosing the right method” to develop
the evidence (and the appropriate level of cer-
tainty) is relatively simple. Of course, most
technologies and interventions lie between
these extremes, and in these instances the im-
portance of stakeholders’ preferences and val-
ues in making decisions increases.
We propose that a taxonomy of types of de-
cisions should be developed. The taxonomy
would define what level of evidentiary cer-
tainty and generalizability should be required
for a category of decision and how that level
was determined. It would make explicit the
influence of stakeholder values and prefer-
ences that established the parameters of each
category. Such a taxonomy across the spec-
trum of decisions would promote greater con-
sistency than would probably emerge from in-
132
dividual decisions over time. This process
would increase the likelihood of good decision
making, raise public confidence in the decision
process, and provide guidance for the use or
conduct of the appropriate types of studies.
T
h e r e i s n o s i n g l e “right” answer on
which approach along the continuum
from observational data to strict evi-
dence-based decisions is correct, but rigid ad-
herence to one approach or another will
clearly lead to suboptimal decision making.
The proper choice of method requires that
stakeholders clearly assess the purposes,
harms, and benefits of alternative approaches
and establish criteria against which different
technologies should be evaluated.
NOTES
1. Observational studies, unlike RCTs, do not in-
volve any intervention to study participants;
they measure health outcomes as they naturally
occur in real-world populations. Systematic evi-
dence reviews are structured analyses of avail-
able evidence from a comprehensive literature
search with a detailed evaluation of the quality of
studies found and a summary of their findings.
Decision models are formal analytic frameworks,
which may incorporate costs, the value of out-
comes, and the probabilities for particular bene-
fits and harms to assess the overall benefits and
costs of treatment alternatives.
2. N. Daniels and J. Sabin, “Limits to Health Care:
Fair Procedures, Democratic Deliberation, and
the Legitimacy Problem for Insurers,” Philosophy
and Public Affairs 26, no. 4 (1997): 303–350.
3. J.D. Kleinke, “Think Globally, Protect Locally: A
Conversation with Mark McClellan,” Health Af-
fairs 23, no. 3 (2004): 177–185.
4. S.J. Goldie and P.S. Corso, “Decision Analysis,” in
Prevention Effectiveness: A Guide to Decision Analysis
and Economic Evaluation, 2d ed., ed. A.C. Haddix,
S.M. Teutsch, and P.S. Corso (New York: Oxford
University Press, 2003), 103–126.
5. See Food and Drug Administration, “Baycol Infor-
mation,” August 2001, www.fda.gov/cder/drug/
infopage/baycol/default.htm (23 November 2004).
6. Daniels and Sabin, “Limits to Health Care.”
7. Methods, reviews, recommendations, and re-
ports of the USPSTF are available at Agency for
Healthcare Research and Quality, “Preventive
Services,” www.preventiveservices.ahrq.gov (21
October 2004).
Ja n u a r y/ Fe b r u a r y 2 0 0 5