[ editorial ]

The Distinction Between

Randomized Clinical Trials
(RCTs) and Preliminary
Feasibility and Pilot Studies:
What They Are and Are Not
Downloaded from www.jospt.org by 112.215.154.65 on 02/26/19. For personal use only.

J. HAXBY ABBOTT, DPT, PhD, FNZCP

Associate Editor
J Orthop Sports Phys Ther 2014;44(8):555-558. doi:10.2519/jospt.2014.0110

A
n underpowered trial is not a pilot study. Let’s just get that were available in the literature8; how-
out there to begin. But a pilot study is an underpowered trial. ever, recent years have seen some useful
Feasibility and pilot studies are preliminary studies conducted guidance become available.2,13 The TABLE
provides a summary of the objectives,
specifically for the purposes of establishing whether or not a
outcomes, and interpretation of feasibili-
full trial will be feasible to conduct, and is between such preliminary studies and ty and pilot studies. Again, the distinction
J Orthop Sports Phys Ther 2014.44:555-558.

that all the necessary components of a
trial will work properly together. A ran-
domized clinical trial (RCT) is conducted
to compare 2 (or more) treatments, or a
treatment to a control or comparison
group. Preliminary studies should not
attempt this, as they will almost always
be woefully underpowered, and thus are
very likely to give misleading answers.
Feasibility and pilot studies play an
important role in the preliminary plan-
ning of a proposed full-size RCT. In es-
sence, feasibility studies are used to help
develop trial interventions or outcome
measures, whereas pilot studies replicate,
in miniature, a planned full-size RCT.2,4
However, the terms used for these pre-
liminary studies are sometimes consid-
ered synonymous,13 and in practice may
overlap considerably or be combined.11,13
The important issue is not the distinction
between these: the important distinction
a proper RCT (TABLE).
Unfortunately, however, feasibility
and pilot studies are very often misused
or misrepresented.8 The most common
transgressions are the characterization of
a too-small RCT as a pilot study, and/or
reporting treatment effectiveness results
from a pilot or feasibility study in the
same way that those from RCTs are re-
ported.2,11 Although it is not an appropri-
ate purpose of either a pilot or feasibility
study to compare the efficacy of an inter-
vention with a comparator,2,11 more than
80% of all pilot studies published from
2007 to 2008 reported hypothesis testing
of treatment effect using inferential sta-
tistics,2 and more than 80% of nondrug
pilot or feasibility trial articles published
from 2000 to 2009 made statements
about efficacy in their abstract.11
Until 2004, no formal methodologi-
cal guidelines for preliminary studies
between them is not critical, as in prac-
tice, often only one or the other is con-
ducted, they are combined, or only the
latter gets published.11,13 However, there
is a strong case, particularly in complex
interventions and multicenter trials, for
far greater attention to preliminary stud-
ies than is currently the case.6
The purpose of a feasibility study is to
assess whether or not it will be feasible
to conduct an RCT of a particular in-
tervention in a particular setting. Some
of the key feasibility issues that must be
considered (TABLE) include the willing-
ness of clinicians to recruit participants,
the willingness of participants to be
randomized to the proposed treatment
groups, whether the intervention can be
delivered as intended within the health
care system and clinical setting, the ad-
herence/compliance of participants to
the proposed treatments, response rates

journal of orthopaedic & sports physical therapy | volume 44 | number 8 | august 2014 | 555

44-08 Editorial-Abbott.indd 555 7/16/2014 3:35:01 PM

 [ editorial ]

Objectives and Outcomes of Feasibility and Pilot Studies in Contrast to
TABLE
Randomized Clinical Trials

Preliminary Studies Randomized Clinical Trials

Feasibility Studies
Objectives—to determine:
• Access to participants (eg, willingness of clinicians
to recruit participants, response of participants to
advertising, proportion of respondents who meet the
eligibility criteria)
• Barriers to participation
• The feasibility and suitability of assessment procedures
and outcome measures
• The necessary time and resources required to conduct
assessments
• Willingness of participants to be randomized to the
proposed treatment groups
• Need for stratification
• Whether a complex treatment is deliverable in the
clinical setting
• Clinician training needs and competence
• Barriers to clinical delivery of the intervention in the
Downloaded from www.jospt.org by 112.215.154.65 on 02/26/19. For personal use only.
Pilot Studies
Objectives—to assess:
• Whether recruitment and screening/eligibility pro-
cesses are working well
• Recruitment rates per week/month
• Randomization processes are working smoothly
• Selection bias
• Effectiveness of blinding
• Capacity and resources to conduct all trial processes
• Access to equipment, space, personnel time
• Processes to ensure and/or audit treatment fidelity
• Adequate time, intensity, dose, effects of interventions
• Assessment processes are timely and complete
• Participant retention among the allocation groups
• Data completeness
• Data variability
• Whether treatment effects/outcomes are consistent
with expectations, previous literature
Efficacy or Effectiveness Trials
Objectives—to investigate:
• T he null hypothesis that treatment A is not more effec-
tive than a comparison (typically either a control group
or another intervention), in the case of a superiority
trial
• O r, less commonly, the null hypothesis that the effect
of treatment A is not equivalent to, or treatment A is
inferior by x margin (MID) to, a comparison (in the
case of an equivalence trial or a noninferiority trial,
respectively)

health care system • Challenges faced by sites, personnel

• Clinicians’ adherence to protocols
• The acceptability to patients of the treatment(s)
• Participants’ adherence to treatment
• Appropriateness of target group for intervention
• Risk of treatment contamination
• Multicenter and multisite issues (eg, comparability and
generalizability of sites and participants, correlations
among site clusters, centers’ commitment and willing-
ness, barriers, differential loss to follow-up)
Outcomes: Outcomes: Outcomes:
• x potential participants per week can be accessed • Recruitment methods are accessing x potential partici- • P  ≤α: the null hypothesis is rejected
• x % of potential participants meet inclusion criteria pants per week • P
 >α: cannot reject the null hypothesis
J Orthop Sports Phys Ther 2014.44:555-558.

• x % of eligible participants are willing to be randomized
• Identified barriers to participation are...
• Problems encountered with assessment procedures
were...
• Etc...
Interpretation:
(a) A full study is not feasible: stop
(b) Problems identified are surmountable: a full
study may be feasible with modifications
(c) Few or no major problems, but some unknowns:
a pilot study without modifications that aims to
complete the information needed to plan a full
study can proceed
(d) Few or no major problems: a full study can
proceed without modifications or further testing,
provided all study processes are known to be
fail safe
Abbreviations: α, the prespecified probability of making the type I error of concluding that there is a difference when in truth there is not; MID, minimal impor-
tant difference.
• x % of potential participants meet inclusion criteria
• x % of eligible participants are willing to be randomized
• x % of consenting participants complete all baseline
assessments within (benchmark) days
• x % of consenting participants commence their ran-
domly allocated treatment within (benchmark) days
• x % of consenting participants complete their randomly
allocated treatment with a minimum of (benchmark)
adherence
• x % of participants complete all follow-up assessments
within (benchmark) days of target date
• Etc...
Interpretation: Interpretation:
(a) Insurmountable problems with necessary processes: In the case of a superiority trial:
stop (a) Treatment A is probably more effective than the
(b) Problems identified are surmountable: modifications comparison
will be necessary in full study protocol, and further (b) Treatment A is probably not more effective than the
pilot testing may be necessary comparison
(c) Few or no major problems: a full study can proceed
without modifications or further testing
NOT: treatment A is effective/not effective

556 | august 2014 | volume 44 | number 8 | journal of orthopaedic & sports physical therapy

44-08 Editorial-Abbott.indd 556 7/16/2014 3:35:02 PM

 to questionnaires, and loss to follow-up.2
Feasibility studies play a critical role in
developing complex interventions and/
or multidisciplinary interventions.3,6
Feasibility studies often don’t necessarily
resemble the final design the researchers
envisage for the RCT, instead being more
flexible and less rigorous in design.2
A pilot study is useful for the purposes
of establishing whether the key compo-
nents necessary for conducting the pro-
posed main RCT, such as the processes
for assessing eligibility, conducting base-
line assessments, randomization proce-
dures, treatment fidelity, and follow-up
assessments, all function well together
(TABLE).2 Pilot studies typically replicate
what the researchers hope will be the fi-
nal design of the full RCT, and serve the
role of a “dress rehearsal.”
Downloaded from www.jospt.org by 112.215.154.65 on 02/26/19. For personal use only.
In both feasibility and pilot studies,
the objectives and specific aims should
be restricted to those practicality issues
(TABLE). Criteria for success should be
specified a priori. Ribeiro et al10 provide
an excellent example of a preliminary
study addressing feasibility issues. They
tried 2 variations of a novel, untried in-
tervention; multisite recruitment; man-
agement of the time and equipment
J Orthop Sports Phys Ther 2014.44:555-558.
necessary to run the intervention at
multiple sites; and participant accept-
ability, adherence, and retention. They
established that their outcome measure
was able to detect change within and dif-
ference between groups, and that at least
1 of their treatment protocols showed an
effect size promising enough to indicate
that it would be worthwhile developing
and testing that intervention further.10
There is some debate in the literature
regarding whether or not it is appropri-
ate to assess for treatment effect in a
feasibility or pilot study, or to estimate
the size and variability of differences
between groups.2,8,11 The estimation of
sample size is generally considered a le-
gitimate purpose of a pilot study7; how-
ever, this should only be the case when
there are no data available from previ-
ous studies to indicate the likely base-
line status of, and variability among, a
journal of orthopaedic & sports physical therapy | volume 44 | number 8 | august 2014 |
44-08 Editorial-Abbott.indd 557
representative sample, and no evidence
informing the minimal important dif-
ference (MID) on the most appropriate
outcome measure.13 These are the data
required for calculating the sample size
for a full RCT.9 Unavailability in the
literature will only seldom be the case.
It is not advisable to calculate sample
size for a full RCT on the basis of the
effect size seen in a pilot study; assess-
ing treatment effect in a feasibility or
pilot study should only be for the pur-
pose of indicating whether a full RCT of
the intervention will be worthwhile, or if
the intervention should be either devel-
oped further or sent back to the drawing
board. When MID and data estimating
the population standard deviation of the
most appropriate outcome measure are
available elsewhere, they should be used
in the first instance, and the data from
the pilot study should then be used to
confirm whether these estimates hold up
in the population and setting intended
for the full trial. Considerable cau-
tion is advised in the use of pilot study
outcomes data to generate sample size
calculations, as the estimates may be
biased due to factors that may not be
reproduced in a full trial, or may be un-
realistic due to chance factors related to
the small sample size.9,13
If there is genuinely no indication of
whether or not a treatment might plausi-
bly be effective, then some form of proof-
of-concept is necessary. In drug trials,
this takes the form of phase 1 or phase
2 trials, conducted to provide some evi-
dence of treatment effects or efficacy.
These are often not randomized, and of-
ten use a surrogate marker of physiologi-
cal effect rather than a patient-reported
outcome.13 In nondrug studies, such as
physical therapy interventions, it might
take the form of a case series or a small
trial with a short-term, physiological
outcome, such as within-group changes
in range of motion or strength,1 rather
than a patient-reported outcome such
as a functional outcome questionnaire
or recurrence,5 or a surrogate outcome
like a risk score shortly after interven-
tion7 rather than an observed, patient-
relevant, longer-term outcome.12 It may
be reasonable to assess these aspects of
treatment effect in the context of a fea-
sibility study2; however, the design of a
full-size trial to assess either efficacy or
effectiveness may still require a subse-
quent pilot study to get an indication of
whether the intervention has promising-
enough effects in longer-term, patient-
relevant outcomes to warrant the time
and expense of an RCT.
Though the distinction between
proof-of-concept studies, phase 2 stud-
ies, feasibility studies, and pilot studies
may be fairly hazy,11,13 the distinction be-
tween these preliminary studies and a
proper RCT should not be. That distinc-
tion lies in the aims: the aim of a true
RCT is to investigate the efficacy or ef-
fectiveness of an intervention (or inter-
ventions) compared with a comparison
group; the aims of preliminary studies
are to assess whether and how a full RCT
can be done (TABLE).
The results of preliminary studies
should, therefore, be mostly descriptive.8
They should focus on important practi-
calities such as recruitment, ability to
deliver the treatment properly in the in-
tended setting, treatment acceptability,
treatment fidelity, adherence, partici-
pant retention, and completeness and
adequacy of outcomes data.10 They may
include within-group treatment effects,
such as effect size, but these should not
be the basis of sample-size calculations
unless the MID for the outcome measure
is genuinely not known.9,10 Statistical
comparisons between groups are likely
to be misleading due to the many poten-
tial problems arising from small samples
and low power. Any results from statisti-
cal comparisons between groups should
be treated with great caution, and re-
ported only with explicit reference to
statistical power. Ideally, no statements
should be made regarding treatment ef-
ficacy, particularly in the abstract, and
the primary interpretation made from
the trial should be with regard to the
conduct of a fully powered RCT. t
557
7/16/2014 3:35:03 PM

REFERENCES
1. A
 bbott JH, Patla CE, Jensen RH. The initial ef-
fects of an elbow mobilization with movement
technique on grip strength in subjects with
lateral epicondylalgia. Man Ther. 2001;6:163-169.
http://dx.doi.org/10.1054/math.2001.0408
2. A
 rain M, Campbell MJ, Cooper CL, Lancaster
GA. What is a pilot or feasibility study? A review
of current practice and editorial policy. BMC
Med Res Methodol. 2010;10:67. http://dx.doi.
org/10.1186/1471-2288-10-67
3. B
 ennell KL, Egerton T, Pua YH, Abbott JH, Sims
K, Buchbinder R. Building the rationale and
structure for a complex physical therapy inter-
vention within the context of a clinical trial: a
multimodal individualized treatment for patients
with hip osteoarthritis. Phys Ther. 2011;91:1525-
1541. http://dx.doi.org/10.2522/ptj.20100430
4. C
 ocks K, Torgerson DJ. Sample size calculations
for pilot randomized trials: a confidence interval
approach. J Clin Epidemiol. 2013;66:197-201.
Downloaded from www.jospt.org by 112.215.154.65 on 02/26/19. For personal use only.
[ editorial ]
http://dx.doi.org/10.1016/j.jclinepi.2012.09.002
5. C oombes BK, Bisset L, Brooks P, Khan A, Vicen-
zino B. Effect of corticosteroid injection, physio-
therapy, or both on clinical outcomes in patients
with unilateral lateral epicondylalgia: a random-
ized controlled trial. JAMA. 2013;309:461-469.
http://dx.doi.org/10.1001/jama.2013.129
6. Craig P, Dieppe P, Macintyre S, Michie S, Naza-
reth I, Petticrew M. Developing and evaluating
complex interventions: the new Medical Re-
search Council guidance. BMJ. 2008;337:a1655.
http://dx.doi.org/10.1136/bmj.a1655
7. Hale LA, Waters D, Herbison P. A randomized
controlled trial to investigate the effects of water-
based exercise to improve falls risk and physical
function in older adults with lower-extremity os-
teoarthritis. Arch Phys Med Rehabil. 2012;93:27-
34. http://dx.doi.org/10.1016/j.apmr.2011.08.004
8. Lancaster GA, Dodd S, Williamson PR. Design
and analysis of pilot studies: recommendations
for good practice. J Eval Clin Pract. 2004;10:307-
312. http://dx.doi.org/10.1111/j.2002.384.doc.x
9. Noordzij M, Tripepi G, Dekker FW, Zoccali C,
Tanck MW, Jager KJ. Sample size calculations:
basic principles and common pitfalls. Nephrol
Dial Transplant. 2010;25:1388-1393. http://
dx.doi.org/10.1093/ndt/gfp732
10. Ribeiro DC, Sole G, Abbott JH, Milosavljevic S.
The effectiveness of a lumbopelvic monitor and
feedback device to change postural behavior:
a feasibility randomized controlled trial. J Or-
thop Sports Phys Ther. In press. http://dx.doi.
org/10.2519/jospt.2014.5009
11. Shanyinde M, Pickering RM, Weatherall M.
Questions asked and answered in pilot and
feasibility randomized controlled trials. BMC
Med Res Methodol. 2011;11:117. http://dx.doi.
org/10.1186/1471-2288-11-117
12. Taylor D, Hale L, Schluter P, et al. Effectiveness
of tai chi as a community-based falls prevention
intervention: a randomized controlled trial. J
Am Geriatr Soc. 2012;60:841-848. http://dx.doi.
org/10.1111/j.1532-5415.2012.03928.x
13. Thabane L, Ma J, Chu R, et al. A tutorial on
pilot studies: the what, why and how. BMC
Med Res Methodol. 2010;10:1. http://dx.doi.
org/10.1186/1471-2288-10-1

EARN CEUs With JOSPT’s Read for Credit Program

JOSPT’s Read for Credit (RFC) program invites readers to study and analyze
selected JOSPT articles and successfully complete online exams about
them for continuing education credit. To participate in the program:
J Orthop Sports Phys Ther 2014.44:555-558.

1. Go to www.jospt.org and click on Read for Credit in the top blue

navigation bar that runs throughout the site.
2. Log in to read and study an article and to pay for the exam by credit
card.
3. When ready, click Take Exam to answer the exam questions for that
article.
4. Evaluate the RFC experience and receive a personalized certificate of
continuing education credits.

The RFC program offers you 2 opportunities to pass the exam. You may
review all of your answers—including your answers to the questions you
missed. You receive 0.2 CEUs, or 2 contact hours, for each exam passed.

JOSPT’s website maintains a history of the exams you have taken and the
credits and certificates you have been awarded in My CEUs and Your Exam
Activity, located in the right rail of the Read for Credit page listing
available exams.

558 | august 2014 | volume 44 | number 8 | journal of orthopaedic & sports physical therapy

44-08 Editorial-Abbott.indd 558 7/16/2014 3:35:03 PM