Melissa Brouwers, PhD Caroline Zwaal, MSc

!!

To present the various options for appraising the reporting and quality the evidentiary base as it relates to clinical practice guideline development In other words, theres no one right way to appraise evidence we just need to be explicit in our use of evidence to guide the development of recommendations There are many different scales to use, we are showing only a few today

!!

!!

!!

Purpose:
!! To estimate the accuracy and validity of the results of the study !! Inadequate reporting can hamper the assessment of strengths and weaknesses of the study and so one cant..

!!

Study design
!! !! !! !!

randomization quasi-randomization observational study Any other evidence

!!

Detailed design and execution

!! !! !! !! !! allocation concealment balance in known prognostic factors intention to treat principle observed blinding (single or double blinding) completeness of follow-up

STROBE Strengthening the Reporting of Observational studies in Epidemiology

!!

22 item checklist for observational research

!! Cohort studies !! Case-control studies !! Cross-sectional studies

!!

Urges reporting of what was planned and what was not, how the research was conducted, what was found, and what the results mean

CONSORT Consolidated Standards of Reporting Trial (RCT)

!!

A minimum set of recommendations in reporting randomized controlled trials Focus on reporting how the trial was designed, analyzed, and interpreted The checklist includes 22 items pertaining to the content of: title, abstract, introduction, methods, results and discussion The flow diagram displays the progress of all participants through the trial

!!

!!

!!

!!

Cochrane Tool to measure the risk of bias in the final results

!! Whether bias in the methods causes the results to be not true so !! the risk that the true effect of the intervention will be overestimated or underestimated

!!

This comprises a description and a judgement for each entry in a Risk of bias table, where each entry addresses a specific feature of the study. It is a two-part tool, addressing the six specific domains.
!! Part 1 describe what happened !! Part 2 judge whether there is bias or not or unclear

!!

!!

Overall plots (pretty pictures) of Risk of bias assessments for the whole systematic review can be created in RevMan.

!!

Sequence generation
!! Randomization

!!

Allocation concealment

!! Concealing the upcoming allocation from those involved in enrolment into the trial (the participant doesnt know which group theyre in before being a part of the study) !! Of participants, personnel and outcome assessors !! Not always possible

!!

Blinding

!!

Incomplete outcome data

!! Missing data so maybe effect estimates biased !! Maybe some participants data omitted exclusions !! Maybe some data not available attrition !! Final report didnt report all of the outcomes stated in the methods non-significant differences not reported !! Certain circumstances -early stopping, clinical setting

!!

Selective outcome reporting

!!

Other sources of bias

!!

Consider the magnitude and direction of bias

!!

The judgement for each entry involves answering the specific question for each entry !! Yes indicating low risk of bias, !! No indicating high risk of bias, !! Unclear indicating either lack of information or uncertainty over the potential for bias.

Entry Sequence generation

Outcome

Judgement

-stated randomly allocated -unclear

!!

So, now you have a bunch of individual studies all rated how can you summarize them? -Tables and tables of information! Group studies into outcomes Developer needs to judge which domains are most important
!! i.e. Is blinding important? !! Subjective outcomes like pain yes it is !! For obvious interventions like surgery tougher to hide so maybe blinding not as important

!! !!

!!

The assessment of the overall risk of bias involves consideration of the relative importance of different domains. How judgements are reached should be made explicit and be informed by:
!! Empirical evidence of bias association between domains and the magnitude of effect !! Likely direction of bias: overestimation or underestimation of the effect
!! can make a domain more or less important

!!

!! Likely magnitude of bias

!! magnitude of effect for certain domains may be greater for certain outcomes ie blinding and pain !! magnitude of bias relative to the magnitude of effect ie minor bias with a small effect would impact ones confidence in effect more than a minor bias with a large effect

!!

Summarizing risk of bias for a study across outcomes: Some domains affect the risk of bias across outcomes in a study: e.g. sequence generation and allocation sequence concealment. Summarizing risk of bias for an outcome within a study (across domains): This is the recommended level at which to summarize the risk of bias in a study, because some risks of bias may be different for different outcomes. Summarizing risk of bias for an outcome across studies (e.g. for a metaanalysis) in summary of findings table Summarizing risk of bias for a review as a whole (across studies and outcomes): It may be tempting to summarize the overall risk of bias in a review, but this should be avoided for two reasons
!! Usually missing data concerning important outcomes critical to a decision ie adverse effects !! Outcomes critical to a decision may very from setting to setting so need context to inform decisions

!!

!!

!!

Risk of Bias Low Risk of Bias Unclear Risk of Bias

Interpretation Plausible bias unlikely to seriously alter the results Plausible bias that raises some doubts about the results Plausible bias that seriously weakens confidence in the results

Within a Study Low risk of bias for all key domains Unclear risk of bias for one or more key domains High risk of bias for one or more key domains

Across Studies Most information is from studies with low risk of bias Most information is from studies is low or unclear risk of bias
The proportion of information from studies at risk of bias is sufficient to affect the interpretation of results

High risk of Bias