CRITICAL APPRAISAL OF

THE MEDICAL LITERATURE

Partini Pudjiastuti T
Child Health Department
Faculty of Medicine University of Indonesia

Critical appraisal
(Making Reading More Worthwhile)
What is Critical Appraisal?
1. Critical appraisal = quality assessment
2. .process of weighing up evidence to see
how useful it is in decision making
3. .a process of assessing the validity, reliability
and usefulness of evidence
4. Critical appraisal is about considering,
evaluating and interpreting information in a
systematic and objective way

Critically Appraise What You Read.

Separating the wheat from the chaff.
Time is limited you should aim to quickly
stop reading the dross.
Others contain useful information mixed
with rubbish.
Simple checklists enable the useful
information to be identified.

Critical Appraisal Critical Thinking

Appraising (evaluating/reviewing) the
available evidence to construct clinical
reasoning strategies and to make
decisions
Finding strengths and limitations of written
evidence
You need to decide what evidence to pay
attention to (what is worthy of your
attention) versus what to ignore

Why critically appraise?

Supports sound decision making based on
best available evidence
Helps us determine (three Rs):
How rigorous a piece of research is
What the results are telling us
How relevant it is to our patient

What is Evidence?
People disagree on what constitutes
evidence
Evidence - what is generally regarded as a
scientific fact
Evidence - a combination of information
obtained from 3 sources: research, clinical
experience, and client preferences (Kitson,
Harvey, & McCormack, 1998)

Why do we need evidence?

Resources should be allocated to things that
are EFFECTIVE
The only way of judging effectiveness is
EVIDENCE
In God we trust all others bring data

Why do we need evidence?

Move towards:
EVIDENCE-BASED MEDICINE
Move away from:
EMINENCE- BASED MEDICINE
What we really, really want is

EVIDENCE-INFORMED MEDICINE

Sources of Evidence
Primary sources
Based on experiments and published
research

Secondary sources
Systematic reviews
Clinical guidelines
Journals of secondary publication e.g.
Evidence Based Medicine

5S Pyramid of Evidence
Resources

Levels of evidence
1. Systematic reviews of RCTs and high quality
RCTs
2. Systematic reviews of Cohort studies, lower
quality RCTs, Outcomes research
3. Systematic reviews of case controls, case
control studies
4. Case Series
5. Expert opinion
See http://www.cebm.net/levels_of_evidence.asp for full
descriptions

Types of Evidence
Question Types
Type of Question
Best Evidence
Health care interventions:
treatment, prevention

Quantitative:
Systematic Review of RCTs or RCT

Harm or Etiology

Quantitative: Observational Study Cohort or Case Control

Prognosis

Quantitative: Observational Study Cohort, Case Control

Diagnosis or Assessment

Quantitative:
Comparison to Gold Standard

Economics

Quantitative:
Cost-effectiveness Study

Meaning

Qualitative:
case study, ethnography, grounded
theory, phenomenologic approach

KEY QUALITY PARAMETERS

VALIDITY
RELIABILITY
IMPORTANCE

VALIDITY
INTERNAL
Is the study designed in such a way that I
can trust the findings?
EXTERNAL
Is the study designed in such a way that I
can generalize the findings?

RELIABILITY
If the study was conducted again,
would the results be the same?
Usually interpreted as the accuracy
of measurement.

IMPORTANCE
What was the effect size
or magnitude of effect?
Clinical vs. statistical
significance.

Tools for Critical Appraisal

EBM simplified approach:

What are the results?

Are the results valid?

Will the results help me in patient care?

Evidence based medicine

5 steps
Formulate question
Efficiently track
down best
available
evidence
Critically review the
validity and usefulness
of the evidence

Evaluate
performance
Implement
changes in
clinical practice

CHECK LIST FOR MEDICAL LITERATURE

(COMPLETENESS)
1.
2.
3.
4.

Title
Authors
Abstract: structured? Informative? Abbreviation?
Introduction: length? Relevant references? Target
population?
5. Methods:

Design
Inclusion criteria
Exclusion criteria
Sample size, sampling method
Randomization technique
Intervention: masking?
Outcome measurement: blinding?

Primary outcome: type of variable

Secondary outcome: type of variable

Analysis

CHECK LIST FOR MEDICAL LITERATURE

(COMPLETENESS)
6. Results
Baseline characteristics
Main outcome
Secondary outcome

7. Discussion
General
Strength and weakness
Conclusions

8. References
Vancouver style
Constant

9. Acknowledgment
10.Ethics approval
11.Conflict interest

What to assess?
(in study of cause-effect relationship)
A. General description
Type of design
Target population, source population,
sample
Sampling method
Dependent and independent variables
Main results?

What to assess?
(in study of cause-effect relationship)
B. Internal validity, non-causal relationship
Influence of bias
Influence of chance
Influence of confounders

Bias
What is a bias? A process that tends to produce

results that depart systematically from the true values

existing in the study population

Types of bias
1. Sample (subject selection) biases, which may result
in the subjects in the sample being unrepresentative
of the population which you are interested in
2. Measurement (detection) biases, which include
issues related to how the outcome of interest was
measured
3. Intervention (performance) biases, which involve how
the treatment itself was carried out.

What to assess?
(in study of cause-effect relationship)
C. Internal validity, causal relationship
Temporality (cause precedes effect)
Strength of association (large difference, RR, OR, etc) or
small p value or narrow confidence interval
Biological gradient (dose dependence)
Consistency among studies (diff. populations or designs)
Specificity (certain factor results in certain effect)
Coherence (does not conflict with current knowledge)
Biological plausibility: can be explained with current
knowledge (at least in part)

What to assess?
(in study of cause-effect relationship)
D. External validity
Applicable to study subjects
Applicable to source population
Applicable to target population

11 items, each with 3 sections

1. Can you find this information in the
paper?
2. Is the way this was done a problem?
3. Does this problem threaten the validity of
the study?

11 items
1. What is the research question?
2. What is the study type?
3. What are the outcome factors and how are they measured?
4. What are the study factors and how are they measured?
5. What important confounders are considered?
6. What are the sampling frame and sampling method?
7. In an experimental study, how were the subjects assigned to
groups? In a longitudinal study, how many reached final followup? In a case control study, are the controls appropriate? (Etc)
8. Are statistical tests considered?
9. Are the results clinically/socially significant?
10. Is the study ethical?
11. What conclusions did the authors reach about the study
question?

1.What is the research

question?
(Is the way this was done a problem?)

Is it concerned with the impact of an

intervention, causality or determining the
magnitude of a health problem?
(Does this problem threaten the validity of the study?)

Is it a well stated research

question/hypothesis?

2. What is the study type?

(Is the way this was done a problem?)

Is the study type appropriate to the research

question?
(Does this problem threaten the validity of the study?)

If not, how useful are the results produced by

this type of study?

3. What are the outcome factors

and how are they measured?
(Is the way this was done a problem?)

a) are all relevant outcomes assessed

b) is there measurement error?
(Does this problem threaten the validity of the study?)

a) how important are omitted outcomes

b) is measurement error an important source
of bias?

4. What are the study factors

and how are the measured?
(Is the way this was done a problem?)

Is there measurement error?

(Does this problem threaten the validity of the study?)

Is measurement error an important source of

bias?

5. What important potential

confounders are considered?
(Is the way this was done a problem?)

Are potential confounders examined and

controlled for?
(Does this problem threaten the validity of the study?)

Is confounding an important source of bias?

6. What are the sampling frame

and sampling method?
(Is the way this was done a problem?)

Is there selection bias?

(Does this problem threaten the validity of the study?)

Does this threaten the external validity of the

study?

7. Questions of internal validity

(Is the way this was done a problem?)

In an experimental study, how were the subjects

assigned to groups?
In a longitudinal study, how many reached follow-up?
In a case control study, are the controls appropriate?
Note: other issues of relevance to internal validity are
considered under the other headings in this critical appraisal
system. You can add your own questions, and also design
your own questions for other study types such as cross
sectional studies and systematic reviews
(Does this problem threaten the validity of the study?)

Does this threaten the internal validity of the study?

8. Are statistical tests

considered?
(Is the way this was done a problem?)

Were the tests appropriate for the data?

Are confidence intervals given?
Is the power given if a null result?
In a trial, are results presented as absolute
risk reduction as well as relative risk
reduction?
(Does this problem threaten the validity of the study?)

If not, how useful are the results?

9. Are the results

clinically/socially significant?
(Is the way this was done a problem?)

Was the sample size adequate to detect a

clinically/socially significant result?
Are the results presented in a way to help in
health policy decisions?
(Does this problem threaten the validity of the study?)

Is the study useful?

10. Are ethical issues

considered?
(Is the way this was done a problem?)

Does the paper indicate ethics approval?

Can you identify potential ethical issues?
(Does this problem threaten the validity of the study?)

Are the results or their application

compromised?

11. What conclusions did the

authors reach about the study
question?
(Is the way this was done a problem?)

Do the results apply to the population in which

you are interested?
(Does this problem threaten the validity of the study?)

Will you use the results of the study?

Appraisal Tools
Tools from the Critical Appraisal Skills
Programme (CASP)

Systematic Reviews
Randomised Controlled Trials
Qualitative Research Studies
Cohort Studies
Case-Control Studies
Diagnostic Test Studies
Economic Evaluation Studies

Available at: http://

www.phru.nhs.uk/casp/critical_appraisal_tools.htm

Study Designs Recap

Effectiveness of
Therapy

Randomised Controlled Trial

Risk Factors /
Prognosis

Cohort Study

Diagnosis

Survey using Gold Standard

Attitudes & Beliefs

Qualitative (Interviews,
Observations, etc)

CRITICAL APPRAISAL

METHODS
- VALID
- IMPORTANT RESULTS
- APPLICABLE DISCUSSION

THANKS

From Data to Wisdom

Data are what researchers collect
Information results when data is analyzed
and interpreted (EVIDENCE)
Knowledge results when information is
shared, acquired, and used
Wisdom is the ability to make the right
use of knowledge

Types of evaluations
Efficacy
treatment does more good than harm when
offered to those who adhere to treatment
recommendations
Does it work under ideal conditions?

Types of evaluations
Effectiveness
treatment does more good than harm in those
to whom it is offered, under ordinary (clinical)
circumstances
Can it work in the real world?

Critical review protocol - quantitative

research

Study purpose
Application to occupational therapy
Study design
Bias
Sampling issues/ sample size/ drop outs
Outcome measurement (reliability, validity)
Intervention description & implementation
Results - statistical & clinical significance
Implications