An introduction to guidelines and

GRADE

With acknowledgement to Prof Davina Ghersi, NHMRC and Prof Holger

Schünemann, McMaster University

Presented by
Sally Green & Steve McDonald
(The Australasian Cochrane Network)
Detty Nurdiati
(Indonesian Cochrane Node)

Evidence Based Clinical Practive Guideline Workshop

Yogyakarta, November 2014
 What is a guideline?
"Guidelines are recommendations intended to
assist providers and recipients of health care
and other stakeholders to make informed
decisions. Recommendations may relate to
clinical interventions, public health activities, or
government policies."

WHO

http://www.sign.ac.uk/images/publications-graphic.jpg
 CPG Definition
• Systematically developed statements to assist
practitioner and patient decisions about appropriate
health care for specific clinical circumstances

• CPG can play an improtant role in health policy

formation and have evolved to cover topics across
the health care continuum (health promotion,
screening and diagnosis)
 What are evidence-based CPGs?
• Systematically developed documents designed to
support clinical decision-making based on the best
available evidence

• What does
“systematically” mean?
– Clear, well explained
process for finding,
evaluating and
summarising evidence

http://biomedicalcomputationreview.org/sites/default/files/u7/second_opinion_cartoon.jpg
 CPG development: The theory
1. Define topic
& scope
11. Revise & update 2. Create multidisciplinary
group with consumers
10. Implement &
evaluate
3. Search for
existing CPGs
9. Finalise
CPG 4. Develop
questions
8. Get consensus
5. Systematically
7. Draft search for research
recommendations 6. Appraise &
synthesise research
 Wisely’s five questions for patients to ask
doctors/health personnels
• Do I really need this test or procedure?
– Medical tests help you and your healthcare provider decide how to
treat a problem. And medical procedures help to actually treat it
• What are the risks?
– Ask if there will be side effects, the chances of getting results that
aren’t accurate, and whether that leads to more testing or another
procedure
• Are there simpler, safer options?
– Sometimes all you need to do is make lifestyle changes, such as
eating healthier foods or exercising more
• What happens if I don’t do anything?
– Ask if your condition might get worse — or better — if you don’t
have the test or procedure rightaway
• How much does it cost?
– Ask if there are less expensive tests, treatments or procedures,
what your insurance may cover, and about generic drugs instead of
brand name drugs
 When do we need guidelines?
• Knowledge gap?
– Is a guideline the right approach?
• Diagnosis?
– Too many cases? Too few? Variation?
• Treatment?
– Under? Over? Variation? Something new?
• Screening?
• Quality of care? Integration of care?
• Other?
 Other reasons?
• Utilisation data?
• Costs?
• Health outcomes?
• Complaints?
• Requests?
• Other?
 What healthcare workers
want…
• A guideline is not a textbook or a cookbook
• To KNOW that the guideline is evidence
based
• But not necessarily all of the evidence…
• To have it easy to use and accessible
• Clear recommendations
New methods in guideline development:
• Involve assessing evidence
K
transparently: e
– How confident are we in an estimate of y
effect?
– Starts with single studies
– Ends with a body of evidence by outcome m
and a recommendation e
s
• Develop structured health care
recommendations: s
– Evidence to recommendation frameworks a
– Comprehensive list of factors that influence a g
recommendation
– Clearly developed and formulated message e
 What are the standards
• World Health Organization methods
• Guideline International Network
• Institute of Medicine (US)
• Professional societies
• GRADE Working Group
Guideline International Network
Guideline International Network
 Institute of Medicine
Report on Trustworthy guidelines

• Be based on a systematic review of the

existing evidence;
• Be developed by a knowledgeable,
multidisciplinary panel of experts and
representatives from key affected groups;
• Consider important patient subgroups and
patient preferences as appropriate;
• Be based on an explicit and transparent
process that minimizes distortions, biases, and
conflicts of interest;
• Provide a clear explanation of the logical
relationships between alternative care options
and health outcomes, and provide ratings of
both the quality of evidence and the strength
of recommendations; and
• Be reconsidered and revised as appropriate
when important new evidence warrants
modifications of recommendations.
 Guideline enterprise
1. Organization, Budget, Planning and Training
2. Priority Setting
3. Guideline Group Membership
4. Establishing Guideline Group Processes
5. Identifying Target Audience and Topic Selection
6. Consumer and Stakeholder Involvement
7. Conflict of Interest Considerations
8. (PICO) Question Generation
9. Considering Values and Preferences
10. Deciding what Evidence to Include and Searching for Evidence
11. Summarizing Evidence and Considering Additional Information
12. Judging Quality, Strength or Certainty of a Body of Evidence
13. Developing Recommendations and Determining their Strength
14. Wording of Recommendations and of Considerations of Implementation, Feasibility
and Equity
15. Reporting and Peer Review
16. Dissemination and Implementation
17. Evaluation and Use
18. Updating
 Background: GRADE
• - International contributors (>300)
with diversity in background – since year 2000
• Developed a unifying, transparent and sensible system
for grading the quality of evidence and developing
recommendations
• Over 70 organizations adopted or use GRADE

CMAJ 2003, BMJ 2004, BMC 2004, BMC 2005,

AJRCCM 2006, Chest 2006, BMJ 2008, JCE 2011-
2012
 Randomization raises
initial quality
RCTs: high
Observational: low
1. Risk of bias
P Outcome Critical

Grade down
High 2. Inconsistency
I Outcome Critical Moderate O3. Indirectness
C Outcome Important Low OO4. Imprecision
Very low OOO5. Publication bias
O Outcome Not
Summary of findings & 1. Large effect
estimate of effect for

Grade up
2. Dose response
each outcome 3. Opposing bias &
Confounders
Evidence synthesis

Recommendation

Grade recommendations Grade overall

(Evidence to Recommendation) quality of evidence
• For or against (direction)  across outcomes based on
• Strong or conditional/weak (strength) lowest quality
of critical outcomes
Guideline
By considering balance of consequences Formulate Recommendations ( | …)
(evidence to recommendations): “The panel recommends that ….should...”
 Quality of evidence
“The panel suggests that ….should...”
 Balance benefits/harms
“The panel suggests to not ...”
 Values and preferences
“The panel recommends to not...”
 Resource use (cost, feasibility)
Transparency, clear, actionable
 Acceptability
 Simple hierarchies are (too)
simplistic
STUDY DESIGN BIAS
 Randomized Controlled
Trials

Expert Opinion
 Cohort Studies and Case
Control Studies
 Case Reports and Case
Series, Non-systematic
observations
Expert Opinion
Schünemann & Bone, 2003
 Determinants of confidence
• RCTs 
• observational studies 
• 5 factors that can lower quality
1. limitations in detailed study design and
execution (risk of bias criteria)
2. Inconsistency (or heterogeneity)
3. Indirectness (PICO and applicability)
4. Imprecision
5. Publication bias
• 3 factors can increase quality
1. large magnitude of effect
2. opposing plausible residual bias or
confounding
3. dose-response gradient
Assessing Quality of Evidence by
Outcome
 Lowering confidence in RCTs
Table: GRADE's approach to rating quality of evidence (aka confidence in effect estimates)
For each outcome based on a systematic review and across outcomes (lowest quality across the outcomes critical for decision making)
1. 2. 3.
Establish initial Consider lowering or raising Final level of
level of confidence level of confidence confidence rating
Study design Initial Reasons for considering lowering Confidence
confidence or raising confidence in an estimate of effect
in an estimate across those considerations
of effect  Lower if  Higher if*

High Risk of Bias Large effect High

Randomized trials
confidence 
Inconsistency Dose response
Indirectness All plausible Moderate
confounding & bias 
Imprecision
 would reduce a
Low demonstrated effect Low
Observational studies Publication bias
confidence or 
 would suggest a
spurious effect if no Very low
effect was observed


*upgrading criteria are usually applicable to observational studies only.

 Altering confidence in observational
studies
Table: GRADE's approach to rating quality of evidence (aka confidence in effect estimates)
For each outcome based on a systematic review and across outcomes (lowest quality across the outcomes critical for decision making)
1. 2. 3.
Establish initial Consider lowering or raising Final level of
level of confidence level of confidence confidence rating
Study design Initial Reasons for considering lowering Confidence
confidence or raising confidence in an estimate of effect
in an estimate across those considerations
of effect  Lower if  Higher if*

High Risk of Bias Large effect High

Randomized trials
confidence 
Inconsistency Dose response
Indirectness All plausible Moderate
confounding & bias 
Imprecision
 would reduce a
Low demonstrated effect Low
Observational studies Publication bias
confidence or 
 would suggest a
spurious effect if no Very low
effect was observed


*upgrading criteria are usually applicable to observational studies only.

 Evidence profiles
Question and source of evidence (systematic review)
Population, intervention, comparator, outcomes
Confidence/quality by
Methods and evaluation Effect estimates
Outcomes outcome:
High
Moderate
Low
Very low
 From evidence to recommendations
Balance
Obser- between
RCTs vational benefits, harms
studies & burdens

Quality of
evidence
Patients’
Resource
values &
use
preferences
High level Lower level
recommen- recommen-
dation dation

Old systems GRADE

 From evidence to recommendations

By considering balance of consequences

(evidence to recommendations):
• Quality of evidence
• Balance benefits/harms
• Values and preferences
• Resource use (cost, feasibility)
• Acceptability