I SpecialArticle

Guidelines for reading literature reviews

AndrewD . O x m an, MD
Gordon H . Guy att, MD
O ne strategy for dealing with th e b urgeoning
m edical literature is to rely on reviews of th e
literature. Alth ough th is strategy is efficient,
readers m ay b e m isled if th e review does not
m eet scientific standards. T h erefore, guidelines
th at will h elp readers assess th e scientific q uali-
ty of th e review are proposed. T h e guidelines
focus on th e definition of th e q uestion, th e
com preh ensiveness of th e search strategy , th e
m eth ods of ch oosing and assessing th e prim ary
studies, and th e m eth ods of com b ining th e
results and reach ing appropriate conclusions.
Application of th e guidelines will allow clin-
icians to spend th eir valuab le reading tim e on
h igh - q uality m aterial and to j udge th e validity
of an auth or' s conclusions.
Une facon efficace de se tenir au courant de la
litterature m ddicale touj ours plus ab ondante
clest de se rab attre sur les revues gindrales. Mais
si celles- ci ne se conform ent pas aux norm es
scientifiq ues, elles risq uent d' induire en erreur.
1 1 est propose ici des lignes directrices afin
d' aider le lecteur d apprdcier la q ualitd scientifi-
q ue d' une revue gdndrale. E lles s" attach ent d
ddterm iner si la q uestion y est b ien dnoncde, la
rech erch e b ib liograph iq ue est com plete, les tra-
vaux retenus sont b ien ch oisis et b ien analy sds,
et les divers resultats sont m is en regard de
facon d cerner des conclusions valab les. E n
suivant ces lignes directrices le clinicien uti-
lisera son tem ps precieux . b on escient.
From th e departm ents of C linical E pidem iology and B iostatistics
and of Medicine, McMaster University , H am ilton, O nt.
D r. Guy att is a career scientist of th e O ntario Ministry of
H ealth .
R eprint req uests to: D r. Gordon H . Guy att, McMaster Universi-
ty H ealth Sciences C entre, 3 H 7 - 1 2 0 0 Main St. W , H am ilton,
O nt. L8N3 Z5
C linicians wh o are attem pting to k eep
ab reast of developm ents m ust find way s to
deal with th e ex ponentially ex panding liter-
ature. E fficient strategies for finding and storing
relevant studiesl- 6 and for discarding invalid or
inapplicab le studies7 - ' 2 are availab le. H owever,
processing th e literature for an answer to a clinical
q uestion rem ains tim e consum ing, and it is not
feasib le for clinicians to read all th e prim ary
literature for each of th e m y riad clinical issues th at
confront th em daily .
O ne solution to th is prob lem is th e literature
review or overview in wh ich th e prim ary research
relevant to a clinical q uestion is ex am ined and
sum m ariz ed. H owever, reviews, as well as prim ary
studies, m ust b e read selectively and critically . J ust
as flawed m eth ods in a study of diagnosis or
th erapy m ay invalidate th e results, an unscientific
literature review m ay com e to incorrect conclu-
sions. Auth ors of reviews do collect and analy se
data from prim ary research , alth ough th is is som e-
tim es done sub j ectively and sub consciously . T h e
fundam ental difference b etween a review and a
prim ary study is th e unit of analy sis, not th e
scientific principles th at apply .
Five conflicting recom m endations for m anag-
ing m ild h y pertension, q uoted from th e literature,
are sh own b elow.
* T h e availab le data . . . lead th is reviewer to
conclude th at treatm ent of m ild h y pertension [ 9 0 to 1 0 4
m m H g] to ach ieve diastolic pressures b elow 9 0 m m H g
is th e appropriate pub lic h ealth policy b ased on current
evidence. ' 3
* Most patients with diastolic b lood pressure in
th e 9 0 to 1 0 4 m m H g range sh ould b e treated unless
contraindications to drug th erapy ex ist. . . . In certain
patients, vigorous dietary and b eh avioral m odifications
m ay b e attem pted b efore instituting or as an adj unct to
ph arm acologic th erapy . ' 4
* Non- drug m easures are often effective for m ild
h y pertension. T h e initial ch oice b etween th iaz ides and
b eta- adrenoceptor b lock ing drugs often depends on th e
ph y sician' s personal preference. . . . W ith care, th e risk s
C MAJ , V O L. 1 3 8, APR IL 1 5, 1 9 88 6 9 7
-
For
prescrib ing
inform ation see
page
7 1 3
of antih y pertensive th erapy are considerab ly less th an
th e b enefits. 1 5
* T h e b enefits of drug treatm ent for patients with
m ild h y pertension [ diastolic b lood pressure b etween 9 0
and 1 0 5 m m H g] rem ain unproven. Non- drug th erapy
h as also b een insufficiently investigated. 1 6
* At present, th erefore, with th e diuretic- b ased
treatm ents principally studied in th e previous trials,
treatm ent of m ild- to- m oderate h y pertension [ diastolic
b lood pressure b elow 1 1 5 m m H g] is of directly dem on-
strated value only if th e strok e rate is h igh enough
( perh aps
due to age or cereb rovascular disease) for
h alving it to j ustify th e costs and troub le of th erapy . . . .
Lipid- sparing antih y pertensives m igh t h ave m ore im por-
tant effects on MI [ m y ocardial infarction] th an on strok e.
B ut, in th e trials reviewed, th e siz e of th e MI reduction
rem ains uncertain [ R ory C ollins: unpub lish ed ob serva-
tions, 1 9 87 ] .
If one doesn' t h ave som e guidelines for assess-
ing th e reviews from wh ich th ese recom m enda-
tions are tak en, deciding wh ich review to b elieve is
lik e deciding wh ich tooth paste to use. It is a
q uestion of taste rath er th an a q uestion of science.
O ne does not h ave to look far to find oth er
ex am ples of im portant clinical q uestions for wh ich
recent reviews h ave com e to different conclusions:
Sh ould clinicians avoid adm inistering cortico-
steroids b ecause of concern ab out clinically im por-
tant osteoporosis? 1 7 ' 1 8 W h at are th e b enefits to
critically ill patients of cath eteriz ing th e righ t side
of th e h eart? 1 9 ' 2 0 Sh ould m ild h y pok alem ia b e
treated aggressively ? 2 1 ' 2 2
C learly , th e ex pertise of th e auth or is not a
sufficient criterion of a review' s credib ility , since
ex perts reviewing th e sam e topic often com e to
different conclusions. Nor is th e prestige of th e
j ournal or tex tb ook in wh ich th e review is pub -
lish ed a sufficient criterion. R ecent survey s of th e
m edical literature h ave found th at th e scientific
q uality of m ost pub lish ed reviews, including th ose
in th e m ost h igh ly regarded j ournals, is poor. 2 3 - 2 7
In th is article we present a reader' s guide to
assessing research reviews. Sim ilar guidelines h ave
b een suggested b efore, particularly in th e psy ch ol-
ogy and social science literature. 2 8- 3 0 W e focus on
h ow readers of th e m edical literature can decide
wh eth er a review is worth reading and wh eth er its
conclusions are to b e b elieved. O ur guidelines m ay
also b e of use to th ose planning to write a research
review.
Guidelines
W e h ave fram ed our guidelines as a series of
q uestions ( T ab le I) . B efore we address each item in
detail som e general com m ents are warranted. First,
th e q uestions are intended to b e used to assess
overviews of prim ary studies on pragm atic q ues-
tions. Second, th e term " prim ary studies" refers to
research reports th at contain original inform ation
on wh ich th e review is b ased. T h ird, th e intention
of th e guidelines is to encourage efficient use of th e
m edical literature and a h ealth y scepticism , not to
prom ote nih ilism . R eaders wh o apply th ese guide-
lines will find th at m ost pub lish ed reviews h ave
m aj or scientific flaws. 2 3 - 2 7 Indeed, survey s on th e
scientific adeq uacy of m edical research reports
h ave found th at m ost prim ary studies also h ave
m aj or scientific flaws. 2 5
T h ere is a need for im provem ent in th e
design, im plem entation and reporting of b oth re-
views and prim ary studies. None th e less, vast
am ounts of valuab le inform ation ex ist, and to
m ak e inform ed decisions clinicians m ust use th e
research availab le. Alth ough m ost pub lish ed re-
views do not provide strong support for th eir
conclusions, critical readers can discern useful
inform ation and m ak e th eir own inferences, wh ich
m ay or m ay not b e th e sam e as th ose of th e
auth ors.
W ere th e q uestions and m eth ods clearly stated?
W h en ex am ining a review article readers m ust
decide wh eth er th e review addresses a q uestion
th at is relevant to th eir clinical practice or interests.
T h ey th erefore req uire a clear statem ent of th e
q uestions b eing addressed.
, ab le - uideh ines
' -
: - issessi
; i- ' e t5' 8, 1 ij ( S( (
T ab 1 e
' - Fr" t e -
F x
C
i
r- t-
t
e . ,
. ,
- . I.
6 9 8 C MAJ , V O L. 1 3 8,
APR IL 1 5,
1 9 88
I5. i. L.
Any causal q uestion h as th ree k ey elem ents:
th e population, th e ex posure or intervention and
th e outcom e. E x am ples of th ese elem ents in five
k ey areas of clinical inq uiry are presented in T ab le
II. A clear statem ent of th e q uestion req uires
ex plicit specification of all th ree elem ents if th e
reader is to q uick ly decide wh eth er th e review is
relevant. If th ere is no clear statem ent of th e
q uestions b eing addressed at th e b eginning of th e
review th e reader m igh t as well stop. Fuz z y
q uestions tend to lead to fuz z y answers.
Many reviews address several q uestions; for
ex am ple, an article or a ch apter in a tex tb ook ab out
acq uired im m une deficiency sy ndrom e m ay review
wh at is k nown ab out th e cause, diagnosis, progno-
sis, treatm ent and prevention of th e disease. Such
reviews m ay b e ex trem ely h elpful for readers
seek ing a b road overview. H owever, th ey tend to
provide little, if any , support for m ost of th e
inferences th ey m ak e. T y pically , an inference is
presented as a fact followed b y one or m ore
citations. In th is case th e reader h as no b asis upon
wh ich to j udge th e strength or validity of th e
inferences with out reading th e articles th at are
cited. R eaders seek ing answers to specific clinical
q uestions sh ould not rely on reviews th at address
b road topics and encom pass m any q uestions.
In addition, an ex plicit statem ent of th e m eth -
ods used for th e research review is necessary for
th e reader to m ak e an inform ed assessm ent of th e
scientific rigour of th e review and th e strength of
th e support for th e review' s inferences. Unfortu-
nately , th is inform ation is often lack ing. In general,
wh en a review does not state h ow som eth ing was
done - for ex am ple, h ow it was decided wh ich
prim ary studies would b e included
-
it is reason-
ab le to assum e th at it was not done rigorously and
th at a th reat to th e validity of th e review ex ists.
R eaders look ing for answers to specific clinical
q uestions sh ould seek reviews th at clearly report
th e m eth ods used. W ith out k nowing th e auth ors'
m eth ods th e reader cannot distinguish statem ents
b ased on evidence from th ose b ased on th e opin-
ions of th e auth ors.
W ere com preh ensive search m eth ods used to
locate relevant studies?
It is surprisingly difficult to locate all th e
pub lish ed research in a particular area, even wh en
th e area is relatively circum scrib ed. 3 ' - 3 3 For ex am -
ple, D ick ersin and associates3 3 found th at a ME D -
LINE search y ielded only 2 9 % of th e relevant trials
on th e prevention and treatm ent of perinatal h y -
perb ilirub inem ia.
T h is prob lem is ex acerb ated b y th e fact th at
som e of th e relevant m aterial m ay not even b e
pub lish ed. Furth erm ore, th e unpub lish ed studies
m ay b e sy stem atically different from th ose th at
h ave appeared in peer- reviewed j ournals, not in
th at th eir m eth ods are flawed b ut in th at th eir
results are " negative" . R esearch h as suggested th at
of two articles th at use th e sam e m eth ods to
investigate a q uestion th e study y ielding positive
results is m ore lik ely to b e pub lish ed th an th e one
y ielding negative results. 3 3 - 3 7 R esearch conducted
b y an agency th at h as an- investm ent in th e
treatm ent b eing studied ( such as a ph arm aceutical
com pany with a new drug) m ay not even b e
sub m itted for pub lication if its results are nega-
tive. It th us b eh oves an auth or to try to determ ine
th e ex tent of th e " pub lication b ias" in th e area
b eing reviewed.
Auth ors' search strategies vary widely , and
ex perts are no m ore lik ely th an nonex perts to b e
sy stem atic in th eir search . 3 8 T h e m ore selective or
h aph az ard th e auth ors' search for papers th e m ore
lik ely it is th at th ere will b e b ias in th e review. For
ex am ple, auth ors are lik ely to attend to papers th at
support th eir preconceptions.
T h e reader needs assurance th at all th e perti-
nent and im portant literature h as b een included in
th e review. T h e m ore com preh ensive th e auth ors'
search th e m ore lik ely it is th at all th e im portant
articles h ave b een found. T h e reader sh ould look
for an ex plicit statem ent of th e search strategies
used. Ideally , such strategies include th e use of one
or m ore b ib liograph ic datab ases ( including a speci-
fication of th e k ey words and oth er aspects of th e
search strategies3 9 ) , a search for reports th at cite th e
im portant papers found th rough a datab ase such as
th e Science C itation Index , perusal of th e refer-
ences of all th e relevant papers found and personal
com m unication with investigators or organiz ations
active in th e area b eing reviewed ( to m ak e sure
im portant pub lish ed papers h ave not b een m issed
and particularly to look for m eth odologically
adeq uate studies th at h ave not b een pub lish ed) .
W ere ex plicit m eth ods used to deternine wh ich
articles to include in th e review?
A com preh ensive literature search will y ield
m any articles th at m ay not b e directly relevant to
th e q uestion under investigation or th at m ay b e so
m eth odologically weak th at th ey do not contrib ute
valid inform ation. T h e auth ors m ust th erefore
select th ose th at are appropriate for inclusion in
th e review. W h en, as is often th e case, th is process
is unsy stem atic, opportunities for b ias develop.
T h us, it is com m on to find two reviews of th e sam e
q uestion in wh ich different prim ary studies are
included and for th e ch oice of studies to contrib ute
to different conclusions. For ex am ple, in two m eth -
odologically soph isticated and carefully conducted
reviews on wh eth er corticosteroids are associated
with peptic ulcer th e two team s of auth ors used
different criteria for ch oosing wh ich studies would
b e included in th e review. 4 0 , 4 1 T h is difference was
th e m ain reason for th e rem ark ab le result of th e
two reviews: diam etrically opposed conclusions
ab out wh eth er or not th e association ex ists.
T h e auth ors sh ould specify h ow th e articles
were ch osen b y referring to th e th ree b asic ele-
C MAJ , V O L. 1 3 8, APR IL 1 5, 1 9 88 6 9 9
m ents of prim ary studies: th e population, th e
ex posure or intervention and th e outcom e. For
ex am ple, in assessing th e effect of ch olinom im etic
agents in patients with dem entia th e auth ors could
specify th e criteria as follows.
* Population: patients with senile dem entia
in wh om causes oth er th an Alz h eim er' s disease
were ex cluded.
* Intervention: oral adm inistration of ch oli-
nom im etic agents.
* O utcom e: indicated b y m easurem ents of
b oth m em ory and functional status.
O th er m eth odologic criteria m ay b e used to
select prim ary papers for review. In th is ex am ple
th e auth ors m ay consider only studies in wh ich
patients were selected at random to receive th e
treatm ent drug or a placeb o and in wh ich b oth th e
investigator and th e patient were b lind to alloca-
tion.
W as th e validity of th eprim ary studies assessed?
Auth ors will com e to correct conclusions only
if th ey accurately assess th e validity of th e prim ary
studies on wh ich th e review is b ased. If all th e
studies h ave b asic flaws th eir conclusions m ay b e
q uestionab le even if th eir results are com parab le.
For ex am ple, if th e literature on ex tracranial-
intracranial b y pass surgery for th reatened strok e
were reviewed b efore th e results of a recent
random iz ed controlled trial4 2 were pub lish ed, a
large num b er of studies with positive results b ut of
sub optim al design and th us open to b ias would
h ave b een found. T h e appropriate conclusion
would h ave b een th at th e procedure' s effectiveness
was still open to q uestion, despite th e volum e of
studies with positive results; indeed, th e sub se-
q uent trial sh owed no b enefit of surgical over
m edical th erapy .
Meth odologic guidelines for studies of etiolo-
gy , 1 0 ' 4 3 diagnosis, 8 prognosis9 and th erapy 1 1 4 4 are
availab le. In a study of th erapy one is interested in
wh eth er th e allocation to treatm ent was random ,
wh eth er th e sub j ects and investigators were b lind
to th e allocation, and wh eth er all th e relevant
outcom es were m onitored. Im portant aspects of
th e design and conduct of each prim ary study
sh ould b e critiq ued and th e standard used in th ese
critiq ues m ade ex plicit. C ritiq ues sh ould b e report-
ed in sufficient detail to allow readers to j udge th e
m eth odologic q uality of th e prim ary studies. Al-
th ough a study - b y - study critiq ue can b e tedious,
presentation of th e m eth odologic assessm ent in a
tab le m ay allow a rapid assessm ent of validity .
R eaders sh ould b e wary of any review th at focuses
on th e results of studies with out th orough ly dis-
cussing th e m eth ods th at were used to arrive at th e
results.
W h en inform ation ab out th e m eth ods or re-
sults h as b een om itted from a pub lish ed report th e
auth ors of a review can contact th e writers of th e
report to ob tain th e m issing inform ation. A review
is strength ened if th e auth ors h ave discussed th e
im plications of m issing inform ation and h ave
attem pted to collect th e relevant data.
W as th e assessm ent of th eprim ary studies
reproducib le and free from b ias?
E x pert assessm ent of prim ary research studies
generally results in a level of disagreem ent th at is
b oth ex traordinary and distressing. For ex am ple,
correlations m easuring agreem ent ab out th e de-
cision to pub lish or not pub lish prim ary research
studies are alm ost alway s less th an 0 . 5 and average
ab out 0 . 3 , 2 84 5, 4 6 a level not m uch h igh er th an one
would ex pect to ach ieve b y ch ance.
Not only do assessm ents lack reproducib ility ,
b ut also th ey are often b iased. In one study Peters
and C eci4 7 resub m itted previously pub lish ed arti-
cles from respected institutions after th ey sub stitut-
ed th e nam es of th e auth ors and th e institutions
with fictitious nam es. Mah oney 3 5 sub m itted an
article to different referees, vary ing th e results
with out altering th e m eth ods. T h ese studies found
th at th e articles th at cam e from respected institu-
tions and reported positive results were m ore
readily accepted. Furth erm ore, in Peters and C eci' s
study m any of th e articles were rej ected b ecause of
" serious m eth odological flaws" , and in Mah oney ' s
study th e article was j udged as h aving weak er
m eth ods wh en it describ ed negative results.
It is even possib le for auth ors to disagree on
th e results of a study . Num erous conflicting re-
views h ave b een reported in wh ich an auth or wh o
favoured a particular treatm ent classified th e pri-
m ary study as positive, wh ereas an auth or wh o did
not favour th e treatm ent classified th e study as
negative. For ex am ple, Miller4 8 found five reviews
th at com pared drug th erapy plus psy ch oth erapy
with drug th erapy alone for psy ch iatric patients. O f
th e 1 1 studies cited in two or m ore of th e reviews
th e results of 6 were interpreted as positive in at
least one review and as negative in at least one
oth er.
Prob lem s with reproducib ility and b ias can
affect two stages of th e review process: th e de-
cision ab out wh ich papers to include and j udge-
m ent of th e q uality of th e papers included. Such
prob lem s can b e m inim iz ed if ex plicit criteria are
used. H owever, m any of th e criteria will req uire
considerab le j udgem ent of th e auth or of a review.
In an ex am ple we used earlier one of th e criteria
for inclusion in a review of treatm ent with ch olino-
m im etic agents for Alz h eim er' s disease was a
definition of th e population as patients with senile
dem entia in wh om causes oth er th an Alz h eim er' s
disease were ex cluded. Is a statem ent in th e tex t
such as " standard m eth ods for diagnosing Alz -
h eim er' s disease were used" adeq uate or does one
req uire details of h ow oth er causes of dem entia
were ruled out?
E x plicit criteria offer little advantage if th ey
cannot b e reproduced b y oth er auth ors. Ideally , all
7 0 0 C MAJ , V O L. 1 3 8, APR IL 1 5, 1 9 88
th e potential prim ary studies sh ould b e assessed
for inclusion b y at least two auth ors, each b lind to
th e oth er' s decision, and th e ex tent of agreem ent
sh ould b e recorded. R eproducib ility sh ould b e
q uantified with a statistical m easure th at q uanti-
tates agreem ent ab ove and b ey ond th at wh ich
would h ave occurred b y ch ance, such as an intra-
class correlation coefficient4 9 or a x statistic. 50 A
sim ilar process sh ould b e used to assess th e
reproducib ility of th e criteria used to determ ine th e
validity of th e prim ary studies.
E ven if th e criteria for study inclusion or
validity can b e reproduced th ere is no guarantee
th at b ias h as not intruded. For ex am ple, if th e
auth ors b elieve th at a new treatm ent work s th ey
m ay apply inclusion criteria b y wh ich studies with
negative results are sy stem atically ex cluded; th e
validity of such studies th at are included m ay b e
j udged m ore h arsh ly . W h at can b e done to prevent
th is sort of b ias?
In random iz ed controlled trials b ias is avoided
if b oth th e patients and th e clinicians are b lind to
wh eth er th e patients are tak ing th e active drug or a
placeb o. In an assessm ent of prim ary studies th e
m aj or possib le sources of b ias are related to th e
auth ors, th eir institution and th e results. H owever,
one can assess th e content and q uality of a study
th rough its m eth ods with out k nowing th is infor-
m ation; th e relevant sections of th e paper can
sim ply b e " wh ited out" so th at th e reviewers are
b lind to th e auth ors' institutions and results. D e-
cisions ab out study inclusion and validity ideally
sh ould b e m ade under th ese conditions. T h is
added precaution will strength en th e review.
W as variation in th e findings of th e relevant
studies analy sed?
Auth ors of reviews are certain to encounter
variab ility in th e results of studies addressing th e
q uestion of interest. Indeed, if all th e results of
prim ary research were th e sam e a review article
would prob ab ly not b e necessary . It is th e auth ors'
task to try to ex plain th is variab ility .
Possib le sources of variab ility are th e study
design, ch ance and differences in th e th ree b asic
study com ponents ( th e population, th e ex posure or
intervention and th e outcom e) . 5" If random iz ed
controlled trials, b efore- and- after studies and
studies with h istorical controls are all included in a
review, and if th e random iz ed controlled trials
consistently sh ow results th at differ sy stem atically
from th ose of th e oth er studies, th e study design
prob ab ly ex plains th e differences. For ex am ple,
Sack s and colleagues52 found th at random iz ed
controlled trials consistently sh ow sm aller effects
th an studies th at use h istorical controls.
A second ex planation for differences in study
results is ch ance. E ven if two investigations use
com parab le m eth ods and th e true siz e of th e
effects is identical th e play of ch ance will lead to
apparent differences in th e siz e. If th e sam ples are
sm all, ch ance alone m ay lead to apparently large
differences in th e siz e of th e effects. Som e trials of
acety lsalicy lic acid ( ASA) in patients with transient
isch em ic attack s h ave sh own a trend in favour of a
placeb o, wh ereas oth ers h ave sh own reductions in
risk of up to 50 % with ASA. 53 H owever, th e
confidence intervals, wh ich represent th e upper
and lower lim its of th e siz e of th e effects consistent
with th e ob served results, overlap. T h us, alth ough
th e apparently discrepant results m igh t suggest
h y poth eses for testing in sub seq uent studies, th ey
are all consistent with a reduction in risk of
b etween 1 5% and 3 0 % with ASA.
In oth er instances differences in study results
m ay b e so large th at th ey cannot b e ex plained b y
ch ance. T h e auth ors m ust th erefore look to differ-
ences in th e population, ex posure or intervention
and outcom e. In our ex am ple of ch olinom im etic
agents in patients with Alz h eim er' s disease th e
studies with negative results m ay h ave included a
larger num b er of severely affected patients th an
th e studies with positive results. O ne m igh t th en
assum e th at th e intervention work s only in m ildly
affected patients. H owever, th e intervention m ay
h ave differed
-
th at is, h igh er doses or different
agents m ay h ave b een given in th e studies with
positive results. Finally , th e tests used to determ ine
m em ory and functional status m ay h ave b een
different; som e tests are m ore responsive to
ch anges in patient status. H orwitz 51 h as docum ent-
ed m any way s in wh ich differences in th e m eth ods
of random iz ed controlled trials can lead to differ-
ing results.
R eaders of a review sh ould b e alert to wh eth er
th ese five ex planations for differing study results
h ave b een considered and sh ould b e sceptical
wh en differences are attrib uted to one ex planation
with out adeq uate consideration of th e oth ers.
W ere th e findings of th eprim ary studies com b ined
appropriately ?
Meta- analy sis ( th e use of several statistical
tech niq ues to com b ine th e results of different
studies) is b ecom ing increasingly popular, especial-
ly as a m eth od of com b ining results from random -
iz ed controlled trials. H owever, it rem ains contro-
versial, and clinical readers cannot b e ex pected to
j udge th e m erits of a particular statistical tech niq ue
used b y th e auth ors of a m eta- analy sis. Neverth e-
less, th ere are issues th at clinical readers can
address.
T h e crudest form of m eta- analy sis, in wh ich
th e num b er of studies with positive results is
com pared with th e num b er of th ose with negative
results, is not satisfactory . T h is " vote count" ig-
nores th e siz e of th e treatm ent effects and th e
sam ple siz es of each study . T h e m ost satisfactory
m eta- analy sis y ields two pieces of inform ation: th e
m agnitude of th e overall treatm ent effect and th e
lik elih ood th at th is effect would h ave occurred b y
ch ance if th e true effect were z ero. T h e form er m ay
C MAJ , V O L. 1 3 8, APR IL 1 5, 1 9 88 7 0 1
b e ex pressed as a percentage risk reduction, th e
latter as a p value.
T h e prim ary advantage of m eta- analy sis is
th at th e results of different studies can b e com -
b ined accurately and reliab ly to determ ine th e b est
estim ate of th e average m agnitude of th e effects of
th e ex posure or intervention of interest. B efore th e
results are com b ined, h owever, one sh ould consid-
er wh eth er it is appropriate to aggregate across th e
studies. Study designs, or th e th ree b asic study
elem ents, m ay differ sufficiently th at a statistical
com b ination of th e results does not m ak e sense.
Meta- analy sis can b e used to analy se th e variation
in study results to generate or test h y poth eses
ab out th e source of th e differences. H owever, it is
on strongest ground wh en th e m eth ods of th e
prim ary studies are sim ilar and th e differences in
th e study results can b e ex plained b y ch ance.
R eviews in wh ich th e results are not statisti-
cally com b ined sh ould state ex plicitly th e b asis for
th e conclusions and sh ould attem pt to ex plain th e
conflicting results. R eaders sh ould b eware of re-
views th at conclude th at th ere is no effect with out
h aving considered th e studies' power to detect a
clinically im portant effect. W h en several studies do
not sh ow a significant difference th ere is a tenden-
cy for reviewers wh o h ave not used m eta- analy sis
to conclude th at th ere is no effect even wh en
statistical aggegation dem onstrates oth erwise.
C ooper and R osenth al54 dem onstrated th is ex peri-
m entally b y assigning reviewers at random to
eith er use or not use m eta- analy sis to com b ine th e
results of several studies, including som e th at did
not sh ow significant results. Anoth er investigator
m ade th e sam e ob servation wh en h e polled re-
search ers wh o h ad conducted trials of tam ox ifen
citrate as adj uvant th erapy for b reast cancer ( R ory
C ollins: personal com m unication, 1 9 87 ) . Most of
th e research ers concluded from th e availab le infor-
m ation th at tam ox ifen did not produce a longer
disease- free interval; h owever, statistical aggrega-
tion of all th e availab le results dem onstrated a
clinically im portant, statistically significant effect.
It is im portant to rem em b er th at all th e oth er
guidelines we h ave discussed still apply wh eth er
or not th e auth ors of a review h ave used m eta-
analy sis.
fab ie W . Guidelines tot
assessaring 4 - h -
ausal rterence
T 4 * t f{ eIatior- : - t i;- .
* i it la1 . ' ) et ee. r
k Ae r1 ( _ ;t7 3 , i;;
* 1 , . . : . i at o) { ) tFro m rT z
Fise C i: sponse re. . . latr. .
- - ee
. elating d' nterT ia
differer- , . tla v t; e at C L
S
St l;;i- ic! ir< ven] tloi
ais. : e. i. rop. esit C {D ! SE r o t - s: st^SE ti V Z;
W ere th e reviewers' condusions supported b y th e
data cited?
W h eth er or not auth ors h ave used m eta- anal-
y sis, th e results of individual prim ary studies
sh ould b e reported in sufficient detail th at readers
are ab le to critically assess th e b asis for th e
auth ors' conclusions. T h e m eth od of presenting
individual study sum m aries will depend on th e
q uestion addressed. For q uestions of treatm ent
effectiveness and prevention th e siz e of th e effects
and its confidence interval give th e k ey inform a-
tion. R eviews of diagnostic tests m ay provide
sensitivities, specificities and lik elih ood ratios ( and
th eir confidence intervals) . 8 Survival curves m ay
efficiently depict th e m ain results of studies of
prognosis.
W ith q uestions of etiology and causation for
wh ich random iz ed controlled trials are not avail-
ab le th e auth ors can evaluate th e evidence with
criteria for causal inference. V ariations of th ese
criteria h ave b een presented b y several investiga-
tors, 1 0 ' 4 4 ' 55' 56 b ut com m on ingredients include th e
siz e and consistency of th e association b etween th e
causal agent and th e outcom e and th e necessity for
dem onstrating th e appropriate tem poral relation.
O ur version of th ese criteria is presented in T ab le
III. T h e auth ors' com m ents on each of th ese criteria
sh ould, of course, refer directly b ack to th e data in
th e prim ary studies cited.
C onclusion
A literature review is a scientific endeavour,
and, as with oth er scientific endeavours, standards
are availab le for conducting th e review in such a
way th at valid conclusions are reach ed. J ust as
readers of th e clinical literature wh o are unab le to
critically appraise th e m eth ods of prim ary studies
m ay arrive at incorrect conclusions, readers wh o
are unab le to assess th e
scienti' fic
q uality of a
review are apt to b e m isled. W e h ave offered eigh t
guidelines for readers interested in answering a
clinical q uestion relevant to th eir every day prac-
tice. Application of th ese guidelines will allow
readers to q uick ly discard review articles th at are
irrelevant or scientifically unsound, to detect po-
tential sources of b ias and to b e confident of
conclusions m ade from a sy stem atic evaluation of
th e availab le research .
W e th ank D rs. Geoff Norm an, D avid Streiner, D avid L.
Sack ett and B rian H utch ison, and Professor Mik e Gent
for th eir h elp in developing th e guidelines.
T h is work was supported in part b y th e O ntario
Ministry of H ealth .
R eferences
1 . H ay nes R B , McK ib b on K A, Fitz gerald D et al: H owto k eep
ulp with th e m edical literature: I. W h y try to k eep up and
h owto get started. Ann Intern Med 1 9 86 ; 1 0 5: 1 4 9 - 1 53
7 0 2 C MAJ , V O L. 1 3 8, APR IL 1 5, 1 9 88
2 . Idem : H ow to k eep up with th e m edical literature: II.
D eciding wh ich j ournals to read regularly . Ib id: 3 0 9 - 3 1 2
3 . Idem : H ow to k eep up with th e m edical literature: III.
E x panding th e num b er of j ournals y ou read regularly . Ib id:
4 7 4 - 4 7 8
4 . Idem : H ow to k eep up with th e m edical literature: IV .
Using th e literature to solve clinical prob lem s. Ib id: 6 3 6 -
6 4 0
5. Idem : H ow to k eep up with th e m edical literature: V .
Access b y personal com puter to th e m edical literature. Ib id:
81 0 - 82 4
6 . Idem : H owto k eep up with th e m edical literature: V I. H ow
to store and retrieve articles worth k eeping. Ib id: 9 7 8- 9 84
7 . D epartm ent of C linical E pidem iology and B iostatistics,
McMaster University H ealth Sciences C entre: H ow to read
clinical j ournals: I. W h y to read th em and h ow to start
reading th em critically . C an Med Assoc J 1 9 81 ; 1 2 4 : 555-
558
8. Idem : H ow to read clinical j ournals: II. T o learn ab out a
diagnostic test. Ib id: 7 0 3 - 7 1 0
9 . Idem : H owto read clinical j ournals: III. T o learn th e clinical
course and prognosis of disease. Ib id: 86 9 - 87 2
1 0 . Idem : H ow to read clinical j ournals: IV . T o determ ine
etiology or causation. Ib id: 9 85- 9 9 0
1 1 . Idem : H ow to read clinical j ournals: V . T o distinguish
useful from useless or even h arm ful th erapy . Ib id: 1 1 56 -
1 1 6 2
1 2 . Idem : H ow to read clinical j ournals: V I. T o learn ab out th e
q uality of clinical care. C an Med Assoc J 1 9 84 ; 1 3 0 : 3 7 7 -
3 82
1 3 . Lab arth e D R : Mild h y pertension: th e q uestion of treatm ent.
Ann R evPub lic H ealth 1 9 86 ; 7 : 1 9 3 - 2 1 5
1 4 . H ab er E , Slater E E : H igh b lood pressure. In R ub enstein M,
Federm an D D ( eds) : Scientific Am erican Medicine, 9 th ed,
Sci Am , NewY ork , 1 9 86 : sect 1 , V II1 - V II2 9
1 5. R isk s of antih y pertensive th erapy [ E ] . Lancet 1 9 86 ; 2 : 1 0 7 5-
1 0 7 6
1 6 . Sack s H S, C h alm ers T C , B erk AA et al: Sh ould m ild
h y pertension b e treated? An attem pted m eta- analy sis of th e
clinical trials. Mt SinaiJ Med 1 9 85; 52 : 2 6 5- 2 7 0
1 7 . Guy att GH , W eb b er C E , Mewa AA et al: D eterm ining
causation
-
a case study : adrenocorticosteroids and os-
teoporosis. J C h ronic D is 1 9 84 ; 3 7 : 3 4 3 - 3 52
1 8. B ay link D J : Glucocorticoid- induced osteoporosis. N E ngl J
Med 1 9 83 ; 3 0 9 : 3 0 6 - 3 0 8
1 9 . Swan H J C , Ganz W : H em ody nam ic m easurem ents in
clinical practice: a decade in review. J Am C oll C ardiol
1 9 83 ; 1 : 1 0 3 - 1 1 3
2 0 . R ob in E D : T h e cult of th e Swan- Ganz cath eter: overuse
and ab use of pulm onary flow cath eters. Ann Intern Med
1 9 85; 1 0 3 : 4 4 5- 4 4 9
2 1 . H arrington J T , Isner J M, K assirer J P: O ur national ob session
with potassium . Am J Med 1 9 82 ; 7 3 : 1 55- 1 59
2 2 . K aplan NM: O ur appropriate concern ab out h y pok alem ia.
Am J Med 1 9 84 ; 7 7 : 1 - 4
2 3 . Mulrow C D : T h e m edical review article: state of th e
science. Ann Intern Med 1 9 87 ; 1 0 6 : 4 85- 4 88
2 4 . Sack s H S, B errier J , R eitm an D et al: Meta- analy ses of
random iz ed controlled trials. NE ngliJ Med 1 9 87 ; 3 1 6 : 4 50 -
4 55
2 5. W illiam son J W , Goldsch m idt PG, C olton T : T h e q uality of
m edical literature. An analy sis of validation assessm ents. In
B ailar J C III, Mosteller F ( eds) : Medical Uses of Statistics,
NE J M B k s, W alth am , Mass, 1 9 86 : 3 7 0 - 3 9 1
2 6 . H alvorsen K T : C om b ining results from independent inves-
tigations: m eta- analy sis in m edical research . Ib id: 3 9 2 - 4 1 6
2 7 . O x m an AD : A Meth odological Fram ework for R esearch
O verviews, MSc th esis, McMaster U, H am ilton, O nt, 1 9 87 :
2 3 - 3 1 , 9 8- 1 0 5
2 8. Ligh t R J , Pillem er D B : Sum m ing Up: th e Science of
R eviewing R esearch , H arvard U Pr, C am b ridge, Mass, 1 9 84
2 9 . J ack son GB : Meth ods for integrative reviews. R evE duc R es
1 9 80 ; 50 : 4 3 8- 4 6 0
3 0 . C ooper H M: T h e Integrative R esearch R eview' : a Sy stem atic
Approach , Sage, B everly H ills, C alif, 1 9 84
3 1 . Glass GV , McGaw B , Sm ith ML: Meta- Analy sis in Social
R esearch , Sage, B everly H ills, C alif, 1 9 81
3 2 . Poy nard T , C onn H O : T h e retrieval of random iz ed clinical
trials in liver disease from th e m edical literature. C ontrolled
C lin T rials 1 9 85; 6 : 2 7 1 - 2 7 9
3 3 . D ick ersin K , H ewitt P, Mutch L et al: Perusing th e
literature: com parison of ME D LINE search ing with a peri-
natal trials datab ase. Ib id: 3 0 6 - 3 1 7
3 4 . Sim es R J : Pub lication b ias. T h e case for an international
registry of clinical trials. J C lin O ncol 1 9 86 ; 4 : 1 52 9 - 1 54 1
3 5. Mah oney MJ : Pub lication prej udices: an ex perim ental study
of confirm atory b ias in th e peer review sy stem . C ognit T h er
R es 1 9 7 7 ; 1 : 1 6 1 - 1 7 5
3 6 . D evine E C , C ook T D : E ffects of psy ch o- educational inter-
vention on length of h ospital stay : a m eta- analy tic review
of 3 4 studies. In Ligh t R J ( ed) : E valuation Studies R eview
Annual, 8th ed, Sage, B everly H ills, C alif, 1 9 83 : 4 1 7 - 4 3 2
3 7 . Sim es R J : C onfronting pub lication b ias: a coh ort design for
m eta- analy sis. Stat Med 1 9 87 ; 6 : 1 1 - 2 9
3 8. C ooper H M: Literature search ing strategies of integrative
research reviewers: a first survey . K nowledge 1 9 86 ; 8: 3 7 2 -
3 83
3 9 . H uth E J : Needed: review articles with m ore scientific rigor.
Ann Intern Med 1 9 87 ; 1 0 6 : 4 7 0 - 4 7 1
4 0 . Messer J , R eitm an D , Sack s H S et al: Association of
adrenocorticosteroid th erapy and peptic- ulcer disease. N
E nglJ Med 1 9 83 ; 3 0 9 : 2 1 - 2 4
4 1 . C onn H O , B litz er B L: Nonassociation of adrenocortico-
steroid th erapy and peptic ulcer. N E ngl
J
Med 1 9 7 6 ; 2 9 4 :
4 7 3 - 4 7 9
4 2 . E C / IC B y pass Study Group: Failure of ex tracranial- intra-
cranial arterial b y pass to reduce th e risk of isch em ic strok e.
R esults of an international random iz ed trial. N E ngl J Med
1 9 85; 3 1 3 : 1 1 9 1 - 1 2 0 0
4 3 . H ill AB : Principles of Medical Statistics, 9 th ed, Lancet,
London, 1 9 7 1 : 3 1 2 - 3 2 0
4 4 . C h alm ers T C , Sm ith H , B lack b urn B et al: A m eth od for
assessing th e q uality of a random iz ed controlled trial.
C ontrolled C lin T rials 1 9 81 ; 2 : 3 1 - 4 9
4 5. B ailar J C III, Patterson K : J ournal peer review: th e need for
a research agenda. In B ailar J C III, Mosteller F ( eds) :
Medical Uses of Statistics, NE J M B k s, W alth am , Mass,
1 9 86 : 3 4 9 - 3 6 9
4 6 . Marsh H W , B all S: Interj udgm ental reliab ility of reviews for
th e J ournal of E ducational Psy ch ology . J E duc Psy ch ol
1 9 81 ; 7 3 : 87 2 - 880
4 7 . Peters D P, C eci SJ : Peer- review practices of psy ch ological
j ournals: th e fate of pub lish ed articles, sub m itted again.
B eh avB rain Sci 1 9 82 ; 5: 1 87 - 2 55
4 8. Miller T I: T h e E ffects of D rug T h erapy on Psy ch ological
D isorders, Ph D dissertation, U of C olorado, B oulder, 1 9 7 7
4 9 . Sh rout PE , Fleiss J L: Intraclass correlations: uses in assess-
ing rater reliab ility . Psy ch ol- B ull 1 9 7 9 ; 86 : 4 2 0 - 4 2 8
50 . C oh en J : A coefficient of agreem ent for nom inal scales.
E duc Psy ch ol Meas 1 9 6 0 ; 2 0 : 3 7 - 4 6
51 . H orwitz R I: C om plex ity and contradiction in clinical trial
research . Am J Med 1 9 87 ; 82 : 4 9 8- 51 0
52 . Sack s H , C h alm ers T C , Sm ith H : R andom iz ed versus
h istorical controls for clinical trials. Am J Med 1 9 82 ; 7 2 :
2 3 3 - 2 4 0
53 . Sz e PC , Pincus M, Sack s H S et al: Antiplatelet agents in
secondary strok e prevention. A m eta- analy sis of th e avail-
ab le random iz ed control trials [ ab str] . C lin R es 1 9 86 ; 3 4 :
3 85A
54 . C ooper H M, R osenth al R : Statistical versus traditional
procedures for sum m ariz ing research findings. Psy ch ol B ull
1 9 80 ; 87 : 4 4 2 - 4 4 9
55. Susser M: R eviews and com m entary : th e logic of Sir K arl
Popper and th e practice of epidem iology . Am J E pidem iol
1 9 86 ; 1 2 4 : 7 1 1 - 7 1 8
56 . Guy att GH , Newh ouse MT : Are active and passive sm ok -
ing h arm ful? D eterm ining causation. C h est 1 9 85; 88: 4 4 5-
4 51
C MAi, V O L. 1 3 8, APR IL 1 5, 1 9 88 7 0 3