11/2/2013

www.medscape.com/viewarticle/706399_print

www.medscape.com

HowtoCriticallyAppraiseanArticle
JaneMYoung,MichaelJSolomon NatClinPractGastroenterolHepatol.20096(2):8291.

SummaryandIntroduction
Summary

Criticalappraisalisasystematicprocessusedtoidentifythestrengthsandweaknessesofaresearcharticlein ordertoassesstheusefulnessandvalidityofresearchfindings.Themostimportantcomponentsofacritical appraisalareanevaluationoftheappropriatenessofthestudydesignfortheresearchquestionandacareful assessmentofthekeymethodologicalfeaturesofthisdesign.Otherfactorsthatalsoshouldbeconsidered includethesuitabilityofthestatisticalmethodsusedandtheirsubsequentinterpretation,potentialconflictsof interestandtherelevanceoftheresearchtoone'sownpractice.ThisReviewpresentsa10stepguidetocritical appraisalthataimstoassistclinicianstoidentifythemostrelevanthighqualitystudiesavailabletoguidetheir clinicalpractice.

Introduction

Topracticeevidencebasedmedicine,cliniciansneedtoapplythefindingsofscientificresearchtothe circumstancesofindividualpatientsaspartoftheirclinicaldecisionmakingprocess.Clinicians,therefore,must beabletoselectandappraisescientificliteraturethatisrelevanttotheirfield,understandtheimplicationsof researchfindingsforindividualpatients,elicitpatients'ownpreferencesanddevelopanappropriatemanagement planbasedonthecombinationofthisinformation.Eachofthesetaskspresentsitsownchallenges,butthe sheervolumeofmedicalliteraturemeansthatthefirststep(thatofselectingandappraisingscientificevidence) canbedaunting.Thenumberofnewmedicalresearcharticlespublishedeachyearcontinuallyincreases,and morethan12,000newarticles,includingpapersoninexcessof300randomizedcontrolledtrials(RCTs),are addedtotheMEDLINEdatabaseeachweek.[1,2]Onepracticalwaythatclinicianscanmanagethis'information overload'[2]istodevelopefficientskillsincriticalappraisal,whichenablethemfocusononlythehighestquality studiesthatwillguidetheirclinicalpracticeandtoextrapolateinformationwhennecessaryfromstudiesofless rigorousdesignifhighqualitytrialsareunavailable. Criticalappraisalhasbeendefinedasthe"...applicationofrulesofevidencetoastudytoassessthevalidityof thedata,completenessofreporting,methodsandprocedures,conclusions,compliancewithethicalstandards, etc.Therulesofevidencevarywithcircumstances."[3]Althoughthemethodologicalcriteriabywhichthevalidity ofastudyisassessedwillvaryaccordingtoitsdesign,somegeneralprinciplesunderpintheevaluationofany researchstudy.Variousguidelinesandassessmenttoolshavebeendevelopedtoprovideastructuredapproach totheprocessofcriticalappraisalforclinicians.[414] Despitetheplethoraofdocumentsavailabletoguidetheprocess,no'goldstandard'instrumentforcritical appraisalexists.Thecriteriausedtoassessthevalidityandrelevanceofscientificliteraturearenotstaticthey mustevolvewithimprovementsinunderstandingoftheimportantsourcesofbiasinherentindifferentstudy designs,andincreasedawarenessofthepotentialinfluenceofothernonmethodologicalfactors,suchasconflicts ofinterest.[15]Astructuredapproachtocriticalappraisalcouldpotentiallyimprovethequalityofthisprocess,and simplechecklistscanbeusefultoscreenoutresearchthatisoflowqualityoroflittlerelevance.[16]ThisReview presentsaguidetothecriticalappraisalprocess.

SelectionandCriticalAppraisalofResearchLiterature
Tenkeyquestions()canbeusedtoassessthevalidityandrelevanceofaresearcharticle.Thesequestionscan assistclinicianstoidentifythemostrelevant,highqualitystudiesthatareavailabletoguidetheirclinical

www.medscape.com/viewarticle/706399_print

1/14

11/2/2013

www.medscape.com/viewarticle/706399_print

practice.
Box1.TenQuestionstoAskwhenCriticallyAppraisingaResearchArticle

Isthestudyquestionrelevant? Doesthestudyaddanythingnew? Whattypeofresearchquestionisbeingasked? Wasthestudydesignappropriatefortheresearchquestion? Didthestudymethodsaddressthemostimportantpotentialsourcesofbias? Wasthestudyperformedaccordingtotheoriginalprotocol? Doesthestudytestastatedhypothesis? Werethestatisticalanalysesperformedcorrectly? Dothedatajustifytheconclusions? Arethereanyconflictsofinterest?
IstheStudy'sResearchQuestionRelevant?

Evenifastudyisofthehighestmethodologicalrigor,itisoflittlevalueunlessitaddressesanimportanttopic andaddstowhatisalreadyknownaboutthatsubject.[17]Theassessmentofwhethertheresearchquestionis relevantisinevitablybasedonsubjectiveopinion,aswhatmightbecrucialtosomewillbeirrelevanttoothers. Nonetheless,thefirstquestiontoaskofanyresearcharticleiswhetheritstopicisrelevanttoone'sownfieldof work.

DoestheStudyAddAnythingNew?

Scientificresearchendeavorisoftenlikenedto'standingontheshouldersofgiants',becausenewideasand knowledgearedevelopedonthebasisofpreviouswork.[18]Seminalresearchpapersthatmakeasubstantive newcontributiontoknowledgearearelativerarity,butresearchthatmakesanincrementaladvancecanalsobe ofvalue.Forexample,astudymightincreaseconfidenceinthevalidityofpreviousresearchbyreplicatingits findings,ormightenhancetheabilitytogeneralizeastudybyextendingtheoriginalresearchfindingstoanew populationofpatientsorclinicalcontext.[17]

WhatTypeofResearchQuestionDoestheStudyPose?

Themostfundamentaltaskofcriticalappraisalistoidentifythespecificresearchquestionthatanarticle addresses,asthisprocesswilldeterminetheoptimalstudydesignandhaveamajorbearingontheimportance andrelevanceofthefindings.Awelldevelopedresearchquestionusuallyidentifiesthreecomponents:thegroup orpopulationofpatients,thestudiedparameter(e.g.atherapyorclinicalintervention)andtheoutcomesof interest.[10]Ingeneral,clinicalresearchquestionsfallintotwodistinctcategories,below. QuestionsAbouttheEffectivenessofTreatment.Thesetypesofquestionsrelatetowhetheronetreatmentis betterthananotherintermsofclinicaleffectiveness(benefitandharm)orcosteffectiveness. QuestionsAbouttheFrequencyofEvents.Suchquestionsrefertotheincidenceorprevalenceofdiseaseor otherclinicalphenomena,riskfactors,diagnosis,prognosisorpredictionofspecificclinicaloutcomesand

www.medscape.com/viewarticle/706399_print

2/14

11/2/2013

www.medscape.com/viewarticle/706399_print

investigationsonthequalityofhealthcare.
WastheStudyDesignAppropriatefortheResearchQuestion?

Studiesthatanswerquestionsabouteffectivenesshaveawellestablishedhierarchyofstudydesignsbasedon thedegreetowhichthedesignprotectsagainstbias.MetaanalysesofwellconductedRCTsandindividualRCTs providethemostrobustevidencefollowedbynonrandomizedcontrolledtrials,cohortstudies,casecontrol studies,andotherobservationalstudydesigns.[19,20]However,insomecircumstances,RCTsareeithernot feasibleorconsideredethicallyinappropriate.Theseissuesaremorecommoninnonpharmaceuticaltrials,such asthoseofsurgicalprocedures.Onereviewofgastrointestinalsurgicalresearchfoundthatonly40%ofresearch questionscouldhavebeenansweredbyanRCT,evenwhenfundingwasnotanimpediment.Patients' preferences,therarityofsomeconditions,andtheabsenceofequipoiseamongsurgeonsprovedtobethemajor obstaclestoperformingRCTsofgastrointestinalsurgeryinthissetting.[21]WhenanRCTisnotfeasible,the specificreasonsthatprecludeitsusewilldeterminethetypeofalternatestudydesignthatcanbeused.[21] Observationalstudies,ratherthanRCTs,arethemostappropriatestudydesignforresearchquestionsonthe frequencyofevents.
DidtheStudyMethodsAddresstheKeyPotentialSourcesofBias?

Inepidemiologicalterms,thepresenceofbiasdoesnotimplyapreconceptiononthepartoftheresearcher,but rathermeansthattheresultsofastudyhavedeviatedfromthetruth.[3]Biascanbeattributedtochance(e.g.a randomerror)ortothestudymethods(systematicbias).Randomerrordoesnotinfluencetheresultsinany particulardirection,butitwillaffecttheprecisionofthestudy[22]bycontrast,systematicbiashasadirection andresultsintheoverestimationorunderestimationofthe'truth'.Systematicbiasesarisefromthewayinwhich thestudyisconducted,beithowstudyparticipantswereselected,howdatawascollected,orthroughthe researchers'analysisorinterpretation.[23] Differentstudydesignsarepronetovaryingsourcesofsystematicbias.Oncethestudydesignofagivenarticle hasbeenidentified,werecommendthatcliniciansuseoneoftheavailabledesignspecificcriticalappraisal checkliststodecidewhetherthestudyinquestionisofhighquality.TheCriticalAppraisalSkillsProgramme (CASP)includessuchtoolsandtheprogramcoordinatorshavedevelopedseparatechecklistsfortheappraisalof systematicreviews,RCTs,cohortstudies,casecontrolstudies,diagnosticteststudies,economicevaluations andqualitativeresearchthateachcomprise10questions.[9]TheyhavebeendevelopedfromtheUsers'guidesto themedicalliteratureseriesofarticlesthatwereoriginallypublishedintheJournaloftheAmericanMedical Association.Thesearticlesarenowavailableinbookform[5]andarereadilyaccessibleontheinternet.[9]
SystematicReviewsandMetaanalyses

Ameticulous,standardizedprotocolisusedinasystematicreviewtoidentify,criticallyappraiseandsynthesize alltherelevantstudiesonaparticulartopic.Somesystematicreviewsmaythenproceedtoametaanalysis,in whichtheresultsfromindividualstudiesarecombinedstatisticallytoproduceasinglepooledresult.[3]Although planningtoundertakeasystematicrevieworametaanalysisprospectivelyispossible,[24]themajorityofthese typesofarticleareretrospectiveandariskofbiasexists,whicharisesfromtheselectionofstudiesandthe qualityoftheseprimarysources.[25]Publicationbias,whichresultsfromtheselectivepublicationofstudieswith positivefindings,isofparticularconcern,asitdistortsoverallperceptionsofthefindingsonaparticulartopic.

[26,27]

TheQUORUM(QualityofReportingofMetaAnalyses)statementprovidesacomprehensiveframeworkfor assessmentsofthequalityofreportinginmetaanalysesandsystematicreviews.[25,28]Inaddition,the AMSTAR[29]assessmenttool,whichcomprises11questions,hasbeendevelopedfortheappraisalof systematicreviews,andthistoolortheCASPchecklist[9]couldbemoreusefulthantheQUORUMstatementfor clinicianswhowishtoundertakearapidappraisalofthesetypesofarticles.Keymethodologicalpointsto considerintheappraisalofsystematicreviewsandmetaanalysesarelistedin.

Box2.KeyMethodologicalPointstoConsiderintheAppraisalofSystematicReviewsandMetaanalyses

www.medscape.com/viewarticle/706399_print

3/14

11/2/2013

www.medscape.com/viewarticle/706399_print

Wereallrelevantstudiesincluded(i.e.wasthesearchcomprehensive,diditexcludearticlesonthebasisof publicationstatusorlanguageandwasthepotentialforpublicationbiasassessed)? Wereselectedarticlesappraisedanddataextractedbytwoindependentreviewers? Wassufficientdetailprovidedabouttheprimarystudies,includingdescriptionsofthepatients,interventions andoutcomes? Wasthequalityoftheprimarystudiesassessed? Didtheresearchersassesstheappropriatenessofcombiningresultstocalculateasummarymeasure? SystematicreviewsandmetaanalysesarenotrestrictedtoRCTsalone.TheMOOSE(MetaAnalysisOf ObservationalStudiesinEpidemiology)guidelineshavebeendevelopedasacorollaryoftheQUORUM statementformetaanalysesofnonRCTs.[30]
RandomizedControlledTrials

InanRCT,therandomallocationofparticipantsshouldensurethattreatmentgroupsareequivalentintermsof bothknownandunknownconfoundingfactorsanydifferencesinoutcomesbetweengroupscan,therefore,be ascribedtotheeffectoftreatment.[31]Studydesignalone,however,willnotguardagainstbiasifcrucialaspects ofthestudyprotocolaresuboptimal.Thepotentialforselectiveenrollmentofpatientsintothestudycanbeone animportantsourceofbiasifthegrouptowhichindividualswillbeallocatedisknownorcanbeguessed.[32] Centralizedmethodsofrandomization,forexampleacomputergeneratedallocation,arepreferabletoless concealedmethods,suchasuseofcolorcodedformsorpseudorandomsequencesbasedonmedicalrecord numbersordaysoftheweek.[31]Failuretoconcealtheallocationsequencehasbeenshowntoresultinagreater distortionoftheresultsthanlackofdoubleblindinganothermajorsourceofbiasinRCTs.[33] TheCONSORT(ConsolidatedStandardsofReportingTrials)statementflowchart(Figure1)isfunctionally equivalenttotheQUORUMstatementforsystematicreviews,andprovidesacomprehensivetoolwithwhichto assessthestandardofreportinginrandomizedtrials.[34]KeypointstoconsiderintheappraisalofanRCTare listedin.
Box3.KeyMethodologicalPointstoConsiderintheAppraisalofRandomizedControlledTrials

Wastheprocessoftreatmentallocationtrulyrandom? Wouldparticipantshavebeenabletoknoworguesstheirtreatmentallocation? Wereparticipantsandresearchers'blinded'toparticipants'treatmentgroup? Wereoutcomesassessedobjectively? Wereallparticipantswhowererandomlyallocatedatreatmentaccountedforinthefinalanalysis? Wereallparticipants'dataanalyzedinthegrouptowhichtheywererandomlyallocated?a

aSeesectiononintentiontotreatanalysisunder'Werethestatisticalanalysesperformedcorrectly?'

www.medscape.com/viewarticle/706399_print

4/14

11/2/2013

www.medscape.com/viewarticle/706399_print

Figure1. Consolidatedstandardsofreportingtrials(CONSORT)statementflowchartforthestandardreportingand appraisalofrandomizedcontrolledtrials.WithpermissionfromCONSORT

CohortStudies

Cohort,orlongitudinal,studiesinvolvefollowinguptwoormoregroupsofpatientstoobservewhodevelopsthe outcomeofinterest.Prospectivecohortstudieshavebeenlikenedtonaturalexperiments,asoutcomesare measuredinlargegroupsofindividualsoverextendedperiodsoftimeintherealworld.[35]Cohortstudiescan alsobeperformedretrospectivelysuchstudiesusuallyinvolveidentifyingagroupofpatientsandfollowingup theirprogressbyexaminingrecordsthathavebeencollectedroutinelyorforanotherpurpose,suchasmedical data,deathregistryrecordsandhospitaladmissiondatabases. Themajormethodologicalconcernwithcohortstudiesistheirhighpotentialforselectionbiasandconfounding factors.Theseproblemsareparticularlyrelevantwhencohortstudies(ornonRCTs)areusedtoevaluate therapeuticinterventions.Inthissituation,thetreatmentthatsomeonereceivesisdeterminedbythepatient'sor clinician'spreferences,referralpatterns,currenttreatmentparadigmsorlocalpolicy.[36]Importantdifferencesare likelytoexistbetweenpatientswhoreceivedisparatetreatmentsandthesedifferences,ratherthanthetreatment itself,mightberesponsiblefortheobservedoutcomes.Althoughsomepotentialconfoundingfactorscanbe measuredandaccountedforintheanalysis,[37]suchadjustmentsaremoredifficultinretrospectivethan prospectivestudies,asdataonimportantpotentialconfoundersmightnothavebeencollected,ormightbeof

www.medscape.com/viewarticle/706399_print

5/14

11/2/2013

www.medscape.com/viewarticle/706399_print

poorquality. TheSTROBE(StrengtheningtheReportingofObservationalStudiesinEpidemiology)statementisthecorollary oftheQUORUMandCONSORTstatementsforobservationalstudies,includingcohort,casecontrolandcross sectionalstudies.[38]Keymethodologicalfeaturestoconsiderintheappraisalofcohortstudiesarelistedin.

Box4.KeyMethodologicalPointstoConsiderintheAppraisalofaCohortStudy

Isthestudyprospectiveorretrospective? Isthecohortrepresentativeofadefinedgrouporpopulation? Wereallimportantconfoundingfactorsidentified? Wereallimportantexposuresand/ortreatments,potentialconfoundingfactorsandoutcomesmeasured accuratelyandobjectivelyinallmembersofthecohort? Werethereimportantlossestofollowup? Wereparticipantsfollowedupforasufficientlengthoftime?

CasecontrolStudies

Casecontrolstudiesarealwaysretrospectivebytheirverynaturethecasepatientsareselectedbecausethey havealreadydevelopedtheoutcomeofinterest(e.g.adisease).Dataarethencollectedaboutfactorsthatmight haveinfluencedthisoutcome,andtheseexposuresarecomparedwiththoseofagroupofpeoplewhodifferfrom thecasepatientsonlyinthattheyhavenotdevelopedtheoutcomeofinterest.Casecontrolstudiesareidealfor theinvestigationofriskfactorswhentheoutcomeofinterestisrare,asitwouldtaketoolongtorecruita prospectivecohort. Majormethodologicaldifficultieswithcasecontrolstudiesaretheselectionofappropriatecontrolindividualsand thepossibilityof'recallbias'(apatient'ssubjectiveinterpretationofwhatcausedtheirconditioncanaltertheir recallofcertaineventsorexperiences).Controlsshouldbedrawnfromexactlythesamepopulationasthe cases,andtheonlydifferencebetweencontrolsandcasesshouldbethatthecontrolshavenotdevelopedthe conditionofinterest.Althoughobjectivemeasuresofpossiblecausativefactorsarepreferable,casecontrol studiesoftenrelyonparticipants'recall,andpatientsmightbemorelikelytoremembercertaineventsor experiencesthancontrols.[39]Keyaspectstoconsiderwhenassessingacasecontrolstudyarelistedin.
Box5.KeyMethodologicalPointstoConsiderintheAppraisalofaCaseControlStudy

Werethecasesclearlydefined? Werethecasesrepresentativeofadefinedpopulation? Howwerethecontrolsselectedandweretheydrawnfromthesamepopulationasthecases? Werestudymeasuresidenticalforcasesandcontrols? Werestudymeasuresobjectiveorsubjectiveandisrecallbiaslikelyiftheyweresubjective?

CrosssectionalAnalyses

Crosssectionalstudiesprovidea'snapshot'inwhichallparameters(exposuresandoutcomes)areassessedat

www.medscape.com/viewarticle/706399_print

6/14

11/2/2013

www.medscape.com/viewarticle/706399_print

thesametimeexamplesofcrosssectionaldesignsincludeoneoffsurveysandauditsofpractice.Key methodologicalpointstoconsiderintheappraisalofacrosssectionalstudyarelistedin.
Box6.KeyMethodologicalPointstoConsiderintheAppraisalofaCrosssectionalStudy

Wasthestudysampleclearlydefined? Wasarepresentativesampleachieved(e.g.wastheresponseratesufficientlyhigh)? Wereallrelevantexposures,potentialconfoundingfactorsandoutcomesmeasuredaccurately? Werepatientswithawiderangeofseverityofdiseaseassessed?

CaseSeries

Caseseriesprovidelowlevelevidenceabouttherapeuticeffectivenesshowever,thesearticlesarevery commoninmedicalliterature.Keymethodologicalissuestoconsiderwhenassessingsucharticlesarelistedin.
Box7.KeyMethodologicalPointstoConsiderintheAppraisalofaCaseStudy

Werecasesidentifiedprospectivelyorretrospectively? Arethecasesarepresentativesample(e.g.aconsecutiveseriesofindividualsrecruitedfrommultiplecenters) andsimilartopatientsinyourpractice? Wereallrelevantexposures,potentialconfoundingfactorsandoutcomesmeasuredaccurately?

StudiesthatAssesstheAccuracyofDiagnosticTests

Thesestudiesareusuallycrosssectionalindesign,butpossessanumberofspecificmethodologicalissuesthat shouldbeconsideredinadditiontothosenotedabove.[40]Toinvestigatetheaccuracyofadiagnostictest,itis performedonasampleofpatientsandtheresultsarecomparedwiththoseofareferenceorgoldstandard diagnostictest.[41]Thelevelofagreementbetweentheinvestigatedtestandthegoldstandarddiagnostictest canthenbereportedeitherintermsofthesensitivityandspecificity,orlikelihoodratio.[4,41] TheSTARD(StandardsfortheReportingofDiagnosticAccuracyStudies)websiteprovidesadetailedflowchart (Figure2)and25itemchecklistforstandardizedreportingandappraisalofstudiesthatassesstheaccuracyof diagnostictests.[42,43]TheCASPalsoprovidesasimilar,butmoresimple,toolforthistypeofstudy.[9] Importantfeaturestoconsiderwhenappraisingastudyofdiagnosticaccuracyarelistedin.

Box8.KeyMethodologicalPointstoConsiderintheAppraisalofaStudyofDiagnosticAccuracy

Doesthesampleofpatientsrepresentthefullspectrumofpatientswithandwithoutthediagnosisofinterest? Wasthereacomparisonwithanappropriate'goldstandard'test? Didallpatientsreceiveboththetestunderevaluationandthesame'goldstandard'test? Werethetestsperformedindependentlywithblindingofassessorstotheresultsofthe'goldstandard'test? Werethecutoffsthatwereusedtoclassifypatientsashavingapositivetestresultclearlydescribed?

www.medscape.com/viewarticle/706399_print

7/14

11/2/2013

www.medscape.com/viewarticle/706399_print

Figure2. Standardsforthereportingofdiagnosticaccuracystudies(STARD)statementflowchartforthestandard reportingandappraisalofstudiesexaminingtheaccuracyofdiagnostictests.WithpermissionfromSTARD

EconomicEvaluations

Economicevaluationstudiesfocusoncostefficiency,orwhichtreatmentcanprovidethegreatestbenefitforthe leastcost.[44]Severaltypesofeconomicevaluationstudiesexist,includingcostbenefit,costeffectivenessand costutilityanalyses,allofwhichdifferinhowtheymeasurehealthbenefits.[45]Animportantfeatureofcritical appraisalofanycostanalysisisanassessmentofhowwellthevariouscostsandconsequencesofindividual treatmentshavebeenidentifiedandmeasured.TheCASPhasdevelopedachecklisttoaidwiththeappraisalof economicevaluationstudies.[9]

WastheStudyPerformedinLinewiththeOriginalProtocol?

Deviationsfromtheplannedprotocolcanaffectthevalidityorrelevanceofastudy.Oneofthemostcommon problemsencounteredinclinicalresearchisthefailuretorecruittheplannednumberofparticipants.Anestimate suggeststhatmorethanathirdofRCTsrecruitlessthan75%oftheirplannedsample.[46]Thisdeviationfrom thestudyplannotonlypotentiallyreducestheextenttowhichtheresultsofthestudycanbegeneralizedtoreal worldsituations,becausethosewhoactuallywererecruitedmightbedifferentfromthosewhoweren'tforsome reason,butalsoreducesthepowerofthestudytodemonstratesignificantfindings.Otherdifferencestothe originalprotocolmightincludechangestotheinclusionandexclusioncriteria,variationintheprovidedtreatments orinterventions,changestotheemployedtechniquesortechnologies,andchangestothedurationoffollowup.

www.medscape.com/viewarticle/706399_print

8/14

11/2/2013

www.medscape.com/viewarticle/706399_print

DoestheStudyTestaStatedHypothesis?

Ahypothesisisaclearstatementofwhattheinvestigatorsexpectthestudytofindandiscentraltoany researchasitstatestheresearchquestioninaformthatcanbetestedandrefuted.[3]Anullhypothesisstates thatthefindingsofastudyarenodifferenttothosethatwouldhavebeenexpectedtooccurbychance. Statisticalhypothesistestinginvolvescalculatingtheprobabilityofachievingtheobservedresultsifthenull hypothesisweretrue.Ifthisprobabilityislow(conventionallylessthan1:20orP<0.05),thenullhypothesisis rejectedandthefindingsaresaidtobe'statisticallysignificant'atthatacceptedlevel. Studyhypothesesmustcruciallybeidentifiedapriori(thatis,beforethestudyisconducted,andaredeveloped fromtheoryorpreviousexperience).Ifthestudyinvestigatesthestatisticalsignificanceofassociationsthatwere notprespecifiedintheoriginalhypothesis(posthocanalysis),suchanalysesarepronetofalsepositivefindings because,atasignificancelevelof5%(P=0.05),1in20associationstestedwillbesignificant(positive)by chancealone.Whenalargenumberofsuchtestsareconductedsomefalsepositiveresultsarehighlylikelyto occur.Anotherimportantconsiderationittocheckthatalldatarelevanttothestatedstudyobjectiveshavebeen reported,andthatselectedoutcomeshavenotbeenomitted. Wheretreatmentsforamedicalconditionalreadyexist,trialscanbedesignedtotestwhetheranewtherapyhas similarefficacytoanexistingone.Thistypeoftrialiscalledanequivalenceornoninferioritytrial,asitspurpose istoestablishthatthenewtreatmentisnoworsethantheexistingone.[47]Equivalencestudiesrequirethatthe degreeofoutcomedifferenceatwhichthetwotreatmentswillnotbeconsideredequivalentbedeterminedin advance.[48]Forexample,researchersmightdecidethatiftheprimaryoutcomeforanewtreatmentisnogreater than5%worsethanthatoftheexistingtreatment,thetwotreatmentswillbeconsideredtobeequivalent. Equivalencestudiesdeterminewhetheranewtreatmentisatleastasgoodasanexistingtreatmentsothat decisionsaboutwhichtreatmenttoadministertoagivenpatientcanbemadeonthebasisofcriteria,suchas costoreaseofadministration.[47,48] TheCONSORTstatementforrandomizedtrialshasbeenextendedtoincorporateguidelinesforreporting equivalencestudies.[49]Akeyquestionwhenappraisingthistypeofstudyiswhetherthetrialresultswere analyzedappropriatelyforanequivalencestudy.Ifastudyisdesignedtoshowthatanewtreatmentisatleast asgoodasanexistingtreatment,statisticalmethods,forconventionaltestingofahypothesisthatonetreatment issuperiortoanothershouldnotbeused.Appropriateanalysisoftheresultsinanequivalencestudyoften involvescalculatingconfidenceintervalsforthetreatmenteffect,anddeterminingwhethertheselimitsarewithin thepredeterminedmarginofnoninferiority.[48]Anotherkeyquestioniswhetherthesamplesizewascalculated correctlyforanequivalencestudy,asthesetypesofstudyusuallyrequirealargersamplesizethana correspondingsuperioritytrial.[49]
WeretheStatisticalAnalysesPerformedCorrectly?

Assessingtheappropriatenessofstatisticalanalysescanbedifficultfornonstatisticians.However,all quantitativeresearcharticlesshouldincludeasegmentwithintheir'Method'sectionthatexplainsthetoolsused inthestatisticalanalysisandtherationaleforthisapproach,whichshouldbewrittenintermsthatareappropriate forthejournal'sreadership.Inparticular,theapproachtodealingwithmissingdataandthestatisticaltechniques thathavebeenappliedshouldbespecifiedpatientswhoarelostinfollowupandmissingdatashouldbeclearly identifiedinthe'Results'section.Originaldatashouldbepresentedinsuchawaythatreaderscancheckthe statisticalaccuracyofthepaper. AnimportantconsiderationinthestatisticalanalysisofRCTsiswhetherintentiontotreat(ITT)orperprotocol analyseswereconducted.AccordingtotheITTprinciple,participants'dataareanalyzedwithreferencetothe grouptowhichtheywererandomlyallocated,regardlessofwhethertheyactuallyreceivedtheallocated treatment.ITTanalysesarepreferred,becausetheymaintaintherandomizationandensurethatthetwo treatmentgroupsarecomparableatbaseline.[50]However,ifalotofparticipantsarenonadherantoralarge proportioncrossovertoothertreatments,anITTanalysiswillbesomewhatconservativeandtheresultsmight bedifficulttointerpret.Inthissituation,aperprotocolanalysisthatincludesonlythosepatientswhocomplied

www.medscape.com/viewarticle/706399_print

9/14

11/2/2013

www.medscape.com/viewarticle/706399_print

withthetrialprotocolcanbeusedtosupplementtheITTanalysis.Asperprotocolanalysesareatincreasedrisk ofselectionbias,theyshouldnotusuallybeusedastheprimarymethodofanalysisunlessacompellingreason existstojustifythisapproach.[50]TheCONSORTflowchart(Figure1)enablestheflowofparticipantsandthe groupsusedintheanalysisofthetrialtobeclearlyidentified.[34]

DotheDataJustifytheConclusions?

Thenextconsiderationiswhethertheconclusionsthattheauthorspresentarereasonableonthebasisofthe accumulateddata.Sometimesanoveremphasisisplacedonstatisticallysignificantfindingsthatinvoke differencesthataretoosmalltobeofclinicalvaluealternatively,someresearchersmightdismisslargeand potentiallyimportantdifferencesbetweengroupsthatarenotstatisticallysignificant,oftenbecausesamplesizes weresmall.Otherissuestobewaryofarewhethertheauthorsgeneralizedtheirfindingstobroadergroupsof patientsorcontextsthanwasreasonablegiventheirstudysample,andwhetherstatisticallysignificant associationshavebeenmisinterpretedtoimplyacauseandeffect.

AreThereanyConflictsofInterest?

Conflictsofinterestoccurwhenpersonalfactorshavethepotentialtoinfluenceprofessionalrolesor responsibilities.[51]Membersofaresearchteammustmakejudgmentsthathavethepotentialtoaffectthe safetyoftheparticipantsandthevalidityoftheresearchfindings.Researchersareinapositiontodecidewhich studieswillbeconductedintheirunit,whichpatientswillbeinvitedtoparticipateinastudyandwhethercertain clinicaloccurrencesshouldbereportedasadverseevents.[52]Thesedecisionsrequireresearcherstoactwith integrityandnotforpersonalorinstitutionalgain. Potentialfinancialconflictsofinterestincludethereceiptofsalaryandconsultationfeesfromthecompanythat hassponsoredtheresearchandownershipofstocksandsharesorotherpecuniaryinterests,suchaspatents relatedtotheresearch.[52]Unitsthatrecruitresearchparticipantsmightbepaidapercapitafeeforeverypatient enrolled,whichcanbegreaterthantheexpensesinvolved.[53]Manypotentialfinancialsourcesofconflictsof interest,suchasindustryfundingforeducationalevents,travelorgifts,areincreasinglyrecognizedbothwithin thecontextofdailyclinicalpracticeandresearch.[54]However,otherpotentialconflictsareinherenttothe researchsetting.Anexampleisthatmedicalresearchers'statusandfutureresearchincomeisdependentonthe successoftheirresearch.[55] Identificationofapotentialconflictofinterestisnotsynonymouswithhavinganactualconflictofinterestorpoor researchpractice.Potentialconflictsofinterestareextremelycommon,andthemostimportantquestionsare whethertheyhavebeenrecognizedandhowtheyhavebeendealtwith.[56]Amainmechanismfordealingwith potentialconflictsofinterestisopendisclosure.[56]Intheprocessofcriticallyappraisingaresearcharticle,one importantstepistocheckforadeclarationaboutthesourceoffundingforthestudyand,ifapotentialconflictof interesthadbeenidentifiedforastatementabouthowthisconflictwasmanaged.Forexample,theresearchers mightstatespecificallythatthesponsoringagencyhadnoinputintotheresearchprotocol,dataanalysisor interpretationofthefindings.Manyjournalsnowroutinelyrequireauthorstodeclareanypotentialfinancialor otherconflictsofinterestwhenanarticleissubmitted.Thereadermustthendecidewhetherthedeclaredfactors areimportantandmighthaveinfluencedthevalidityofthestudy'sfindings.

Conclusions
Criticalappraisalisasystematicprocessthroughwhichthestrengthsandweaknessesofaresearchstudycan beidentified.Thisprocessenablesthereadertoassessthestudy'susefulnessandwhetheritsfindingsare trustworthy.Themostimportantcomponentofcriticalappraisaliscarefulassessmentofthestudydesign however,othersteps,suchasevaluationofthestatisticalmethodsused,interpretationofthefindingsand potentialconflictsofinterestarealsoessential.Finally,considerationoftheimportanceoftheresearchtoone's ownpatientswillhelpcliniciansidentifythemostrelevant,highqualitystudiesavailabletoguidetheirclinical practice.

www.medscape.com/viewarticle/706399_print

10/14

11/2/2013

www.medscape.com/viewarticle/706399_print

CLICKHEREforsubscriptioninformationaboutthisjournal.

KeyPoints
Criticalappraisalisasystematicprocessusedtoidentifythestrengthsandweaknessesofaresearch article

Criticalappraisalprovidesabasisfordecisionsonwhethertousetheresultsofastudyinclinicalpractice

Differentstudydesignsarepronetovarioussourcesofsystematicbias

Designspecific,criticalappraisalchecklistsareusefultoolstohelpassessstudyquality

Assessmentsofotherfactors,includingtheimportanceoftheresearchquestion,theappropriatenessof statisticalanalysis,thelegitimacyofconclusionsandpotentialconflictsofinterestareanimportantpartof thecriticalappraisalprocess

References

1. DrussBGandMarcusSC(2005)Growthanddecentralisationofthemedicalliterature:implicationsfor evidencebasedmedicine.JMedLibrAssoc93:499501 2. GlasziouPP(2008)Informationoverload:what'sbehindit,what'sbeyondit?MedJAust189:8485 3. LastJE(Ed.2001)ADictionaryofEpidemiology(4thEdn).NewYork:OxfordUniversityPress 4. SackettDLetal.(2000).EvidencebasedMedicine.HowtoPracticeandTeachEBM.London:Churchill Livingstone 5. GuyattGandRennieD(Eds2002).Users'GuidestotheMedicalLiterature:aManualforEvidencebased ClinicalPractice.Chicago:AmericanMedicalAssociation 6. GreenhalghT(2000)HowtoReadaPaper:theBasicsofEvidencebasedMedicine.London:Blackwell MedicineBooks 7. MacAuleyD(1994)READER:anacronymtoaidcriticalreadingbygeneralpractitioners.BrJGenPract 44:8385 8. HillAandSpittlehouseC(2001)Whatiscriticalappraisal.EvidencebasedMedicine3:18 9. PublicHealthResourceUnit(2008)CriticalAppraisalSkillsProgramme(CASP). 10. NationalHealthandMedicalResearchCouncil(2000)HowtoReviewtheEvidence:Systematic IdentificationandReviewoftheScientificLiterature.Canberra:NHMRC 11. ElwoodJM(1998)CriticalAppraisalofEpidemiologicalStudiesandClinicalTrials(2ndEdn).Oxford: OxfordUniversityPress 12. AgencyforHealthcareResearchandQuality(2002)Systemstoratethestrengthofscientificevidence?

www.medscape.com/viewarticle/706399_print

11/14

11/2/2013

www.medscape.com/viewarticle/706399_print

EvidenceReport/TechnologyAssessmentNo47,PublicationNo02E019Rockville:Agencyfor HealthcareResearchandQuality 13. CrombieIK(1996)ThePocketGuidetoCriticalAppraisal:aHandbookforHealthCareProfessionals. London:BlackwellMedicinePublishingGroup 14. HellerRFetal.(2008)Criticalappraisalforpublichealth:anewchecklist.PublicHealth122:9298 15. MacAuleyDetal.(1998)RandomisedcontrolledtrialoftheREADERmethodofcriticalappraisalin generalpractice.BMJ316:113437 16. ParkesJetal.Teachingcriticalappraisalskillsinhealthcaresettings(Review).CochraneDatabaseof SystematicReviews2005,Issue3.Art.No.:cd001270.doi:10.1002/14651858.cd001270 17. MaysNandPopeC(2000)Assessingqualityinqualitativeresearch.BMJ320:5052 18. HawkingSW(2003)OntheShouldersofGiants:theGreatWorksofPhysicsandAstronomy. Philadelphia,PN:Penguin 19. NationalHealthandMedicalResearchCouncil(1999)AGuidetotheDevelopment,Implementationand EvaluationofClinicalPracticeGuidelines.Canberra:NationalHealthandMedicalResearchCouncil 20. USPreventiveServicesTaskforce(1996)Guidetoclinicalpreventiveservices(2ndEdn).Baltimore,MD: Williams&Wilkins 21. SolomonMJandMcLeodRS(1995)Shouldwebeperformingmorerandomizedcontrolledtrialsevaluating surgicaloperations?Surgery118:456467 22. RothmanKJ(2002)Epidemiology:anIntroduction.Oxford:OxfordUniversityPress 23. YoungJMandSolomonMJ(2003)Improvingtheevidencebaseinsurgery:sourcesofbiasinsurgical studies.ANZJSurg73:504506 24. MargiticSEetal.(1995)Lessonslearnedfromaprospectivemetaanalysis.JAmGeriatrSoc43:435 439 25. SheaBetal.(2001)Assessingthequalityofreportsofsystematicreviews:theQUORUMstatement comparedtoothertools.InSystematicReviewsinHealthCare:MetaanalysisinContext2ndEdition, 122139(EdsEggerM.etal.)London:BMJBooks 26. EasterbrookPHetal.(1991)Publicationbiasinclinicalresearch.Lancet337:867872 27. BeggCBandBerlinJA(1989)Publicationbiasanddisseminationofclinicalresearch.JNatlCancerInst 81:107115 28. MoherDetal.(2000)Improvingthequalityofreportsofmetaanalysesofrandomisedcontrolledtrials:the QUORUMstatement.BrJSurg87:14481454 29. SheaBJetal.(2007)DevelopmentofAMSTAR:ameasurementtooltoassessthemethodologicalquality ofsystematicreviews.BMCMedicalResearchMethodology7:10 30. StroupDFetal.(2000)Metaanalysisofobservationalstudiesinepidemiology:aproposalforreporting. MetaanalysisOfObservationalStudiesinEpidemiology(MOOSE)group.JAMA283:20082012 31. YoungJMandSolomonMJ(2003)Improvingtheevidencebaseinsurgery:evaluatingsurgical

www.medscape.com/viewarticle/706399_print

12/14

11/2/2013

www.medscape.com/viewarticle/706399_print

effectiveness.ANZJSurg73:507510 32. SchulzKF(1995)Subvertingrandomizationincontrolledtrials.JAMA274:14561458 33. SchulzKFetal.(1995)Empiricalevidenceofbias.Dimensionsofmethodologicalqualityassociatedwith estimatesoftreatmenteffectsincontrolledtrials.JAMA273:408412 34. MoherDetal.(2001)TheCONSORTstatement:revisedrecommendationsforimprovingthequalityof reportsofparallelgrouprandomizedtrials.BMCMedicalResearchMethodology1:2 35. RochonPAetal.(2005)Reader'sguidetocriticalappraisalofcohortstudies:1.Roleanddesign.BMJ 330:895897 36. MamdaniMetal.(2005)Reader'sguidetocriticalappraisalofcohortstudies:2.Assessingpotentialfor confounding.BMJ330:960962 37. NormandSetal.(2005)Reader'sguidetocriticalappraisalofcohortstudies:3.Analyticalstrategiesto reduceconfounding.BMJ330:10211023 38. vonElmEetal.(2007)Strengtheningthereportingofobservationalstudiesinepidemiology(STROBE) statement:guidelinesforreportingobservationalstudies.BMJ335:806808 39. SuttonTyrrellK(1991)Assessingbiasincasecontrolstudies:properselectionofcasesandcontrols. Stroke22:938942 40. KnottnerusJ(2003)Assessmentoftheaccuracyofdiagnostictests:thecrosssectionalstudy.JClin Epidemiol56:11181128 41. FurukawaTAandGuyattGH(2006)Sourcesofbiasindiagnosticaccuracystudiesandthediagnostic process.CMAJ174:481482 42. BossyutPMetal.(2003)TheSTARDstatementforreportingstudiesofdiagnosticaccuracy:explanation andelaboration.AnnInternMed138:W1W12 43. STARDstatement(StandardsfortheReportingofDiagnosticAccuracyStudies). 44. RafteryJ(1998)Economicevaluation:anintroduction.BMJ316:10131014 45. PalmerSetal.(1999)Economicsnotes:typesofeconomicevaluation.BMJ318:1349 46. RussSetal.(1999)Barrierstoparticipationinrandomizedcontrolledtrials:asystematicreview.JClin Epidemiol52:11431156 47. TinmouthJMetal.(2004)Areclaimsofequivalencyindigestivediseasestrialssupportedbythe evidence?Gastroentrology126:17001710 48. KaulSandDiamondGA(2006)Goodenough:aprimerontheanalysisandinterpretationofnoninferiority trials.AnnInternMed145:6269 49. PiaggioGetal.(2006)Reportingofnoninferiorityandequivalencerandomizedtrials:anextensionofthe CONSORTstatement.JAMA295:11521160 50. HeritierSRetal.(2007)Inclusionofpatientsinclinicaltrialanalysis:theintentiontotreatprinciple.In InterpretingandReportingClinicalTrials:aGuidetotheCONSORTStatementandthePrinciplesof RandomizedControlledTrials,9298(EdsKeechA.etal.)StrawberryHills,NSW:AustralianMedical

www.medscape.com/viewarticle/706399_print

13/14

11/2/2013

www.medscape.com/viewarticle/706399_print

PublishingCompany 51. NationalHealthandMedicalResearchCouncil(2007)NationalStatementonEthicalConductinHuman Research8990Canberra:NHMRC 52. LoBetal.(2000)Conflictofinterestpoliciesforinvestigatorsinclinicaltrials.NEnglJMed343:1616 1620 53. KimSYHetal.(2004)Potentialresearchparticipants'viewsregardingresearcherandinstitutionalfinancial conflictsofinterests.JMedEthics30:7379 54. KomesaroffPAandKerridgeIH(2002)Ethicalissuesconcerningtherelationshipsbetweenmedical practitionersandthepharmaceuticalindustry.MedJAust176:118121 55. LittleM(1999)Research,ethicsandconflictsofinterest.JMedEthics25:259262 56. LemmensTandSingerPA(1998)Bioethicsforclinicians:17.Conflictofinterestinresearch,education andpatientcare.CMAJ159:960965 ReprintAddress JaneMYoung,SurgicalOutcomesResearchCentre(SOuRCe),RoyalPrinceAlfredHospital,POBoxM157, MissendenRoad,NSW2050,Australia.Email:jyoung@email.cs.nsw.gov.au NatClinPractGastroenterolHepatol.20096(2):8291.2009NaturePublishingGroup

www.medscape.com/viewarticle/706399_print

14/14