1

Peninsula Technology Assessment Group (PenTAG), University of Exeter Medical School, Veysey Building, Salmon Pool Lane, Exeter EX2 4SG
Cite: L.Long Routine piloting in systematic reviews a modified approach? Syst Rev, 2014 Jul 18.3(1):77


Stages of Proposed Method

In this paper it is proposed that current
guidelines in conducting systematic reviews
are followed up to the stage where papers
have been assessed for inclusion in the
review. The pilot method is then employed
(fig 1) and a purposive sample (reporting a
wide range of outcome measure scales and
time points) of included papers are chosen
from the total number of included studies to
pilot up to synthesis stage (step 1). Data from
the sample papers are extracted (step 2) and
critically appraised for quality and validity
(step 3) and a sample synthesis is then
performed (step 4). Results from this pilot
synthesis are then used to inform modification
of data extraction forms in the light of these
preliminary synthesis findings (step 5), to
ensure efficient and meaningful extraction of
data from all included papers.


Future Research Recommendations
It is hoped that this paper will serve as stimulation for further discourse on the subject of maximising
validity and efficiency in systematic reviews, given the increasing volume of research papers available both
now and in the future. Details of approaches developed by other research teams to address this issue, or
evaluation of the above routine piloting method, would be most welcome. Contact L.Long@exeter.ac.uk
Theoretical Case Study
A systematic review of adults with type 2
diabetes exploring the association
between biomedical outcomes (e.g. HbA1c,
BMI/weight and blood pressure) and
quality of life (including low mood and
depression) following a diabetic drug
intervention. Extraction of quantitative
data would be performed using a
structured data extraction form to include
key study details, patient characteristics,
diabetes-related factors, intervention,
setting and outcome measures. It may
emerge through the piloting process that in
addition to baseline diagnosis of
depression, a patients history of
depression is important in predicting
changes in blood glucose levels (as
measured by HbA1c) and so the data
extraction form would need to be modified
to extract history of depression data from
all the review papers. It may also emerge
during the piloting process that some
studies record single follow-up points for
biomedical outcome measures, while
others have multiple follow-up time points.
After consideration of the pilot synthesis
stage, the data extraction form could be
modified to ensure that only the most
clinically relevant time points required to
answer the review question are extracted
in the full review.
Strengths and Weaknesses of Piloting Systematic
Reviews Prior to Full Review
ADVANTAGES WEAKNESSES
Efficient and potentially
time-saving when
processing large numbers
of studies
Not necessary for reviews
with small numbers of
included studies
Greater flexibility for
optimally efficient data
extraction
Need to have access to most
of the potentially eligible
studies in order to draw the
purposive sample.
Can be used in large
umbrella overviews
(a review of reviews)
Not necessary for overviews
with small numbers of
included systematic reviews
Background
A continuous growth in the publication of research papers
mean that there is an expanding volume of data available to the
systematic reviewer. Sometimes, researchers can become
overwhelmed by the sheer volume of data processed, leading
to inefficient data extraction. The paper seeks to address this
problem by proposing a modification to the current systematic
review methodology.

Proposed Method
This paper details the routine piloting of systematic reviews all
the way through to evidence synthesis stage using data from a
sample of included papers.

Results and Discussion
The result of piloting a sample of papers through to evidence
synthesis stage is to produce a mini-systematic review.
Insights from such a pilot review may be used to modify criteria
in the data extraction form. It is proposed that this approach
will ensure that in the full review the most useful and relevant
information is extracted from all the papers in one phase,
without needing to re-visit the individual papers at a later
stage.

Conclusions
Routine piloting in systematic reviews has been developed in
response to advances in information technology and the
subsequent increase in rapid access to clinical papers and data.
It is proposed that the routine piloting of large systematic
reviews will enable themes and meaning in the data to become
apparent early in the review process. This, in turn, will facilitate
the efficient extraction of data from all the papers in the full
review. It is proposed that this approach will result in increased
validity of the review, with potential benefits for increasing
efficiency.
Sample Data
Extraction /
Checking
Refine Data
Extraction /
Checking
Included n=41 Papers
(36 individual studies)
Pilot Paper Sample
(e.g. n=10)
Full Data
Extraction /
Checking
Appraise /
Analyse All
Individual
Studies
Full Synthesis
Dissemination
Appraise /
Analyse
Sample Studies
Sample
Synthesis
Fig 1: Theoretical Process Model
For Pilot Method
STEP 1
STEP 2
STEP 3
STEP 4
STEP 5