Critical Appraisal

Capacity Enhancement Program

Guideline Development Webinars

March 29, 2011

Who we are

Canadian Partnership
Against Cancer

Capacity Enhancement
Program (CEP)
Who we are

www.cancerguidelines.ca
 Who we are

CEP Training Coordinators

•  Caroline Zwaal

•  Karen Spithoff
 Outline

•  Video: Introduction to Critical Appraisal

•  Critical Appraisal of Systematic Reviews
and Meta-Analyses
•  Practice Exercise
•  Critical Appraisal of Practice Guidelines
•  Q&A
•  Online feedback survey
Introductory Video

http://www.youtube.com/user/CEPguidelines#p/a/u/0/KL_yC0RYc6Y
Critical Appraisal
Evidence Synthesis Documents:
Practice Guidelines
Systematic Reviews
Meta-analyses
How did we get here?

 Choose a topic and scope

 Develop guideline questions

 Develop evidence selection criteria

 Search for evidence and apply the evidence

selection criteria

☐ Critically appraise the evidence

Critical Appraisal

“The process of assessing and interpreting

evidence by systematically considering its
validity, results, and relevance.”

www.sahealthinfo.org
Evidence Synthesis Documents

•  Practice guidelines (PGs)

•  Systematic Reviews (±meta-analysis)

PGs
Systematic
Reviews
Randomized
Controlled Trials
Prospective Cohort
Studies
Retrospective Cohort and Case
Controlled Studies

Case Series/ Case Reports

 Definitions
•  Clinical practice guidelines
–  Systematically developed statement to assist practitioners and
patients in making decisions about appropriate health care for
specific clinical circumstances

•  Systematic review
–  A review of specific health questions that uses explicit systematic
methods to identify, select, and critically appraise relevant
evidence, and to collect and analyze data from studies included in
the review

•  Meta-analysis
–  The use of statistical methods in a systematic review to combine
the results of included studies

•  Narrative review (will not be discussed further)

–  A review using selected evidence identified using non-systematic
methods to discuss a topic or support an opinion
Why is critical appraisal of evidence
synthesis documents important?

•  Use of the best available evidence is the foundation

that enables good decision making in areas of
uncertainty

•  Although evidence-based practice guidelines and

systematic reviews are at the top of the evidence
hierarchy, they vary considerably in quality of
development
 Why is critical appraisal of evidence
synthesis documents important?

•  Some guidelines and systematic reviews may be

based on primary evidence that is not appropriate for
the types of questions asked.

•  This increases the likelihood that the results and

conclusions are not an accurate reflection of the true
estimate of effect, potentially leading to greater
harms than benefits to the target population.
Critically appraising systematic
reviews and meta-analyses

What are some of the issues that affect the validity

of a systematic review or meta-analyses?

•  Clarity of objectives and clinical questions

•  Rigour and transparency of methods to identify, select, and

appraise evidence

•  Appropriateness of methods used to summarize and

synthesize evidence

•  Minimization of biases (e.g. publication bias, conflicts of

interest, etc.)

•  Conclusions
 Objectives and clinical questions

•  Objectives should be clearly described

•  Clinical questions should be explicitly reported (using PICO

format if appropriate)

–  Unclear objectives and questions can lead to a disorganized

review and unclear conclusions

–  It is difficult to determine whether a systematic review or

meta-analysis is relevant to your guideline questions if
objectives and questions are not clear
 Identification and selection of evidence

•  Evidence selection criteria should be pre-determined and

specifically reported
•  The literature search, including databases searched, search
dates, and search terms (ideally the full search strategy),
should be reported
•  The search should be comprehensive enough to capture all
(or enough) relevant evidence

–  For the results of an existing systematic review/meta-analysis

to be useful, you need to know that rigorous and transparent
methods were used to identify and select appropriate evidence
 Reporting of Evidence

•  A list of included (and excluded) studies should be provided

in the systematic review report

•  Characteristics and results of included studies should be

clearly presented in the report

•  Included studies should be appraised for methodological

quality

–  Sufficient information about the included evidence must be

provided for you to determine whether the conclusions are
appropriate
 Synthesis of Evidence

•  Results from included studies can be synthesized either

qualitatively using text or quantitatively using meta-
analysis

•  Meta-analysis should be conducted using appropriate

methods. Methods and results should be explicitly reported.

–  Were included studies sufficiently similar to pool? Were all

appropriate studies included?
–  How precise are the results?
–  Are the results statistically and clinically significant?
 Synthesis of Evidence: Meta-analysis

The following should be reported for meta-analyses:

•  Measure of effect

•  Method of combining results (statistical models, how effect

estimates were calculated for each study)

•  Handling of missing data

•  How statistical heterogeneity was assessed

•  Rationale for a-priori sensitivity and subgroup analyses

 Minimization of Bias

•  To minimize bias, study selection and data extraction

should be done in duplicate by two independent reviewers

•  Publication bias: Studies with positive results are more

likely to be published in a peer reviewed journal and
available in English than studies with negative results.
Systematic reviews should assess whether there is a
likelihood of publication bias

•  Authors’ potential conflicts of interest should be reported

 Conclusion and Discussion

•  There should be a summary of the findings for each

outcome

•  The conclusions should be based on or supported by the

evidence reported

•  There should an interpretation of the results and

implications for research and clinical practice
 Critical Appraisal Tools

•  Measurement tools designed to appraise the

methodological quality or reporting of evidence

•  Can be in the form of checklists, scales, or domain-

based tools
 Critical Appraisal Tools: Systematic
reviews and meta-analysis

•  AMSTAR: 11-item checklist to appraise the

methodological quality of systematic reviews

•  PRISMA: 27-item checklist to guide the reporting of

systematic reviews (not specifically designed to be a
critical appraisal tool)

•  MOOSE: A checklist to guide the reporting of meta-

analyses of observational studies

See the Resources section of www.cancerguidelines.ca for references and more tools
AMSTAR
Critically appraising practice guidelines

All quality elements discussed for systematic reviews also

apply to the appraisal of evidence-based practice guidelines

•  Clear objectives
•  Clear clinical questions
•  Rigorous and transparent methods to identify and
select evidence
•  Appropriate summarization and synthesis of evidence
•  Reporting conflicts of interest
More issues for practice guidelines

Description of target population

•  If relevant, the description should include age, sex,

clinical condition, severity/stage of disease,
comorbidities, and excluded population
Example
 Description of target users

•  e.g., The intended guideline audience may include

specialists, family physicians, patients, clinical or
institutional leaders/administrators
Example
 Development group that includes all
relevant stakeholders

•  For each guideline development group member,

guidelines should list name, discipline or content
expertise, institution, geographical location, and a
description of the member’s role in guideline
development
Example
 Incorporation of target population
preferences

•  Preferences may be captured from evidence in the

literature, surveys, focus groups, or inclusion of a
representative of the target population in the
guideline development group
Example
 Description of how
recommendations were developed

•  Methods may include a voting system, informal

consensus, or formal consensus techniques.
Descriptions should include methods, results, and
how the process influenced the recommendations.
Example
 Clear recommendations explicitly linked
to evidence

•  Recommendations should be specific, unambiguous,

and easy to find
•  Different options for management should be clearly
presented
•  Each recommendation should be linked to a description
of the key evidence or a reference list
Example
 External review process

•  Guidelines should be reviewed by individuals who

were not involved in developing the guidelines.
External review methods, outcomes, and how the
feedback was used to inform the guideline
development process should be reported.
Example
 Updating procedure

•  Guidelines should report when and how the guideline

will be updated
 Implementation and monitoring
strategies

•  Advice and/or tools on how the recommendations can be

put into practice should be provided.
•  The guideline report should also identify criteria to
assess guideline implementation or adherence to
recommendations and how these criteria should be
measured.
Examples
 Statement of editorial independence
from the funding body

•  The guideline should report the name of the funding

body and should state that the funding body did not
influence the content of the guideline.
Example
 Critical Appraisal Tools:
Clinical Practice Guidelines

AGREE II

•  The current gold standard for appraising the reporting

quality of clinical practice guidelines

•  23 items organized within six domains

1.  Scope
and
Purpose

2.  Stakeholder
Involvement

3.  Rigour
of
Development

4.  Clarity
of
PresentaAon

5.  Applicability

6.  Editorial
Independence


www.agreetrust.org
 AGREE II: User’s Manual

AGREE II User’s Manual content:

•  Definition of each item

•  Examples of each item
•  Where information can be found
•  Criteria and considerations to use
when rating each item
 AGREE II: Rating Scale

•  Score of 1 (Strongly Disagree):

–  When there is an absence of information (i.e., there is no guideline
content relevant to the AGREE item) or if the concept is very poorly
reported.

•  Score of 7 (Strongly Agree):

–  If the quality of reporting is exceptional and where the criteria and
considerations articulated in the users guide have been met.
 AGREE II: Domain Scores

•  Each guideline must be assessed by a minimum of 2

reviewers

•  Reviewers’ scores for items within each domain are

combined to give overall domain scores expressed as
percentage. The formula can be found in the AGREE II
Instrument.
AGREE II: 23 Items
Domain Item
Scope and purpose 1. The overall objective(s) of the guideline is (are) specifically described.
2. The health question(s) covered by the guideline is (are) specifically described.
3. The population (patients, public, etc.) to whom the guideline is meant to apply is specifically described.
Stakeholder involvement 4. The guideline development group includes individuals from all the relevant professional groups.
5. The views and preferences of the target population (patients, public, etc.) have been sought.
6. The target users of the guideline are clearly defined.
Rigor of development 7. Systematic methods were used to search for evidence.
8. The criteria for selecting the evidence are clearly described.
9. The strengths and limitations of the body of evidence are clearly described.
10. The methods for formulating the recommendations are clearly described.
11. The health benefits, side effects and risks have been considered in formulating the recommendations.
12. There is an explicit link between the recommendations and the supporting evidence.
13. The guideline has been externally reviewed by experts prior to its publication.
14. A procedure for updating the guideline is provided.
Clarity of presentation 15. The recommendations are specific and unambiguous.
16. The different options for management of the condition or health issue are clearly presented.
17. Key recommendations are easily identifiable.
Applicability 18. The guideline describes facilitators and barriers to its application.
19. The guideline provides advice and/or tools on how the recommendations can be put into practice.
20. The potential resource implications of applying the recommendations have been considered.
21. The guideline presents monitoring and/ or auditing criteria.
Editorial independence 22. The views of the funding body have not influenced the content of the guideline.
23. Competing interests of guideline development group members have been recorded and addressed.
 AGREE II: Limitations

•  Does not assess clinical content

•  Does not assess the quality of evidence that supports the

recommendations

•  Its role in appraising guidelines tackling health care system

or organizational issues and health technology assessments
has not been evaluated
 What to do with the results

Options:

•  Results can be used to exclude low quality practice

guidelines, systematic reviews, or meta-analyses
−  If pre-specified in the evidence selection criteria

•  Or, include all documents but focus discussion on high

quality evidence synthesis documents
 What to do with the results

Options:

•  If only poor quality guidelines and systematic reviews are

available:

–  Conduct your own review of the primary literature

–  Reduce strength of recommendations as appropriate to
reflect lack of high quality evidence

–  Consider using a formal expert opinion consensus

process to develop recommendations
How to report critical appraisal

•  Describe critical appraisal methods in the Methods

section of your guideline

•  Include a table summarizing critical appraisal results

•  Provide a summary of the overall quality of the evidence

in the text of the guideline
How to report critical appraisal

•  Discuss evidence derived from synthesis documents in

the context of its methodological quality

•  Provide a grade of the evidence in the recommendations

section of your guideline if appropriate
 Example

CCO: Cancer pain management standards

Discussion

•  Questions?

•  Experiences to share?
 Optional Follow-up Exercises

1. Critically appraise a guideline using the AGREE II.

•  Pick any guideline from the SAGE Inventory of Cancer
Guidelines to evaluate:
http://cancerguidelines.ca/Guidelines/inventory/index.php

2. Critically appraise a systematic review.



•  Wulff CN, et al. Case management used to optimize cancer care
pathways: A systematic review. BMC Health Services Research
2008;8:227.
•  Use any systematic review critical appraisal tool or develop
your own list of important quality criteria.
Feedback

Brief online feedback survey:

http://fhswedge.csu.mcmaster.ca/cep/
index.php?sid=85392&lang=en
 Next webinar

Tuesday, April 19, 12:00 pm EDT

•  Topic: Critical appraisal of randomized controlled trials

and phase I and II clinical trials

•  Register at www.cancerguidelines.ca

•  Invite your colleagues