Anda di halaman 1dari 6

ORIGINAL REPORTS

Validation of a Web-Based Curriculum for


Resident Education in Orthopedic Surgery$

Barrett Boody, MD,* Patrick Johnson, DO, Andrew Pugely, MD, Daniel Miller, MD,
Jeffrey Geller, MD, William Payne, MD, James Boegener, MD, Michael Schafer, MD,*
and Matthew Beal, MD*

*
Department of Orthopaedic Surgery, Northwestern Memorial Hospital, Chicago, Illinois; Department of
Orthopaedic Surgery, Francisican St. James Health, Chicago Heights, Illinois; Department of Orthopaedic
Surgery, University of Iowa, Iowa City, Iowa; Department of Orthopaedic Surgery, Columbia University
Medical Center, New York, New York; and Department of Orthopaedic Surgery, University of
MissouriKansas City, Kansas City, Missouri

BACKGROUND: The Orthopedic In-Training Examina- RESULTS: While OITE scores incrementally increased for
tion is an annual standardized examination with multiple- the group as a whole (n 71, RPGY mean improvement
choice questions focused on application of orthopedic surgery 2.5%, p 0.406), the junior resident subgroup (post-
core knowledge and principles. The outcome of this test can graduate year 1-2) produced a statistically signicant increase
be used to both predict how residents are progressing in their in scores (n 28, RPGY mean increase 10.1%,
orthopedic knowledge as well as assess their likelihood to pass p 0.0260). Nearly 90% (38/42) of curriculum participants
the Orthopedic Board Examinations, that is the American surveyed reported a preference to complete a similar review
Boards of Orthopedic Surgery Part 1, following completion curriculum for future OITE preparation. The participants
of residency. Preparation for the examination can be difcult, completing greater than 150 PASS questions (n 57) were
as residents commonly have limited study time to review the analyzed for OITE predictive capacity of the PASS curricu-
vast amounts of available published literature. The objective lum. Pearson analysis with PASS questions percent answered
of our study is to evaluate the effectiveness of the Orthopae- correctly and 2014 OITE raw score (independent and
dic In Training Examination (OITE) scores and the partic- dependent variables, respectively) suggests moderate correla-
ipants perceived utility of the curriculum for OITE tion of the variables (R 0.682, p o 0.05).
preparation.
CONCLUSION: We propose that the Orthobullets PASS
METHODS: Residents from 5 US Orthopedic residencies curriculum is a useful tool for OITE preparation, especially
(4 M.D. and 1 D.O.) were included in a pilot program of for junior residents, with a focus on achieving competency
the Orthobullets PASS curriculum in the academic year for a broad fund of orthopedic knowledge, whereas mastery
2013 to 2014. Only residents enrolled in the PASS of these topics would be better suited through using
curriculum who completed both the 2013 and 2014 OITEs complementary sources. ( J Surg Ed ]:]]]-]]]. J C 2016
were included in the nal analysis (n 71). We used the Association of Program Directors in Surgery. Published by
OITE 2013 and 2014 rank postgraduate year (RPGY) Elsevier Inc. All rights reserved.)
reported scores to assess for efcacy of the PASS curriculum,
KEYWORDS: orthopedic surgery, web-based curriculum,
as the RPGY score provides postgraduate year-of-training
in-training examination, resident self-assessment
matched analysis to control for expected increased levels of
knowledge with subsequent retesting. COMPETENCIES: Medical Knowledge


An author (D.J.M.) received a monetary reward from his residency program for
his high level of performance in the Orthobullets PASS curriculum. No authors have INTRODUCTION
nancial ties to Orthobullets or any other direct or indirect conicts of interest related
to this article. Program directors nor residency programs were compensated for their The Orthopedic In-Training Examination is an annual
utilitzation of the PASS curriculum.
standardized examination administered to all US orthopedic
Correspondence: Inquiries to Barrett Boody, MD, Department of Orthopaedic
Surgery, Northwestern Memorial Hospital, 676 N St. Clair St, Ste 1350, Chicago, residents with multiple-choice questions focused on appli-
IL 60611; fax: (312) 926-4643; e-mail: barrett.s.boody@gmail.com cation of orthopedic surgery core knowledge and principles.

Journal of Surgical Education  & 2016 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 1
Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2016.05.020
The purpose of the examination is to provide an objective made aware of participants results, no consequences for
assessment of a residents gross fund of orthopedic knowl- poor performance or noncompliance were reported. Results
edge, reporting results regarding both the overall testing of PASS curriculum monthly quizzes were provided for
population, and a level-of-training matched analysis (rank analysis by the Orthobullets editorial team. Program direc-
postgraduate year [RPGY]). The outcome of this test can tors from the 5 institutions submitted their programs
be used to both predict how residents are progressing in OITE results from the 2013 to 2014 years. Once each
their orthopedic knowledge as well as assess their likelihood residents OITE and Orthobullets PASS program data were
to pass the Orthopedic Board Examinations, that is, the correlated for statistical analysis, identifying information was
American Boards of Orthopaedic Surgery Part 1, following deleted for condentiality. Only residents enrolled in the
completion of residency.1-3 PASS curriculum who completed both the 2013 and 2014
Our study evaluates the utility and efcacy of a new web- OITEs were included in the nal analysis. All residents
based curriculum designed for orthopedic residentsPASS participating in the PASS program from the 5 institutions
curriculum. Preparation for the examination can be dif- were asked to complete a brief survey about their OITE
cult, as residents commonly have limited study time to study preparation and their perceived utility of the Ortho-
review the vast amounts of available published literature. bullets PASS curriculum for OITE preparation. The North-
The Orthobullets PASS curriculum provides access to daily western University Institutional Research Board approved
comprehensive reading curriculum and optional timed the study. Neither the participating programs nor the
subject quizzes and mock examinations. The curriculum authors were compensated for the utilization of the PASS
used scheduled monthly quizzes in ofce of intramural curriculum or publication of this article.
training and education (OITE) format with the intended
purpose of assisting residents identify their progression and
deciencies in orthopedic knowledge, as well as facilitate Statistics
program directors in identifying at-risk residents for poor Correlated OITE and PASS data for 71 residents were
performance on the OITE, and thus more importantly, available for statistical analysis. We used the OITE 2013
failure risk for residents preparing to take American Board and 2014 RPGY reported scores to assess for efcacy of the
of Orthopaedic Surgery (ABOS) Part 1. PASS curriculum, as the RPGY score provides postgraduate
The objective of our study is to evaluate the effectiveness year (PGY) year-of-training matched analysis to control for
of the Orthobullets PASS curriculum on improving OITE expected increased levels of knowledge at subsequent
scores and the participants perceived utility of the curric- retesting. Subgroup analysis was performed to determine
ulum for OITE preparation. We compared residents OITE the effect of the PASS curriculum for junior resident (PGY
performance from both the year before (2013) and year of 1 and 2 at the time of the 2013 OITE) versus senior
the Orthobullets PASS curriculum (2014) to assess the resident (PGY 3 and 4 at the time of the 2013 OITE) and
comparative effectiveness of the new curriculum technique individual PGY subgroups. Paired t-tests were performed for
over conventional examination preparation techniques. We overall and subgroup analyses as participants OITE scores
additionally compared residents PASS curriculum and allow for matched pair analysis and 2-tailed p-values were
OITE performance to evaluate the PASS curriculum for reported. Pearson correlations were performed for the PASS
predictive capabilities regarding future OITE performance. curriculum performance to determine predictive relation-
Additionally, we surveyed the residents to understand how ships with performance on the 2014 OITE.
participants using the Orthobullets PASS curriculum used
the resource, perceived the efcacy of the resource, and their
likelihood to use the curriculum again. RESULTS
An overall (n 71) RPGY score increase of 2.5% (p
DESIGN 0.406) was seen after participation in the PASS curriculum,
with raw scores on average increasing from 163 to 176
Residents from 5 US orthopedic residencies (4 M.D. and 1 (maximum score of 275). On subgroup analysis, the junior
D.O.) were included in a pilot program of the Orthobullets resident subgroup (n 28, PGY 1 and 2 at the time of the
PASS curriculum in the academic year 2013 to 2014. 2013 OITE) demonstrated signicant improvements in
Residency programs integrated the PASS curriculum into OITE performance, with raw score increases from 139 to
their core OITE preparation curriculae and received com- 164 and an average 10.1% improvement in RPGY score
plementary access to the PASS curriculum for their partic- (p 0.0259). The senior resident subgroup (n 43, PGY
ipation in pilot testing. Residents were required to 3 and 4 at the time of the 2013 OITE) demonstrated an
participate in the PASS curriculums monthly multiple- average raw score improvement from 178 to 183an average
choice tests designed to reect similar material previously of 2.4% decrease of their RPGY score on the 2014 OITE,
presented on the OITE. Although program directors were which lacked statistical signicance (p 0.5445). The

2 Journal of Surgical Education  Volume ]/Number ]  ] 2016


Overall Analyses Including Junior
Versus Senior Resident
Subgroups: Basic Stascs
RPGY 2013 RPGY 2014
PGY 2013 Mean Std Mean Std
PGY 1&2
(n=28) 48.18% 29.89% 58.25% 28.41%

PGY 3&4
61.07% 27.32% 58.65% 27.57%
(n=43)
Overall
55.99% 28.86% 58.49% 27.70%
(n=71)

Overall Analyses Including Junior


Versus Senior Resident
Subgroups: Paired t-tests with p-
values
Group DF t Value Pr > |t|
PGY 1&2 27 2.356 0.026
PGY 3&4 42 0.611 0.545
Overall 70 0.835 0.406

FIGURE 1. Overall and junior versus senior resident subgroup performances are shown. Basic statistics of mean scores and standard deviation are
provided for the 2013 and 2014 OITE administrations. As the overall and subgroup analyses are comprised of subsequent retesting of the same
participants, a paired t-test with p-values are provided.

decrease in mean RPGY score for the senior residents OITE with an average of 69% answered correctly (SD
appeared to be largely from the 2013 PGY 4 subgroup 12%). The results of 70 participants who completed at least
(mean decrease of 9.8%, p 0.067). When excluding 1 PASS test (25 questions per test) were subsequently
the 2013 PGY 4 subgroup from the overall analysis, the analyzed for OITE predictive capacity of the PASS curric-
2013 PGY 1 to 3 subgroup improved an average 7.3% ulum. Pearson analysis with PASS questions percent
(p 0.041). Score analyses that could not be meaningfully answered correctly and 2014 OITE raw score (independent
correlated to previous OITE performances without account- and dependent variables, respectively) suggests moderate
ing for the interval improvement in OITE scores correlation of the variables (R 0.649, p o 0.05) (Fig. 3).
as expected with increased experience (i.e., raw scores) Totally, 42 residents from the 5 participating residency
were provided for context, but not statistically analyzed program responded to surveys regarding their OITE study
(Figs. 1 and 2). habits and utility of the PASS program. Residents reported
Participants completed an average of 194 PASS questions high perceived utility of the PASS program for OITE
(standard deviation [SD] 75.4) of a possible 275 PASS preparation. When asked to rate the utility of the PASS
milestones mock examination questions before the 2014 curriculum using a global rating scale from 1 (no utility) to

Subgroups by PGY level:


Basic Stascs
RPGY 2013 RPGY 2014
PGY 2013 Mean Std Mean Std
1 49.36% 30.44% 55.91% 26.37%
2 47.41% 30.45% 59.77% 30.35%
3 60.74% 29.86% 64.70% 32.20%
4 61.45% 25.86% 51.70% 19.62%

Subgroups by PGY level:


Paired t-tests with p-
values
PGY2013 DF t Value Pr > |t|
1 10 0.722 0.487
2 16 3.02 0.008
3 22 0.689 0.499
4 19 1.944 0.067

FIGURE 2. Subgroup analysis by PGY level. Basic statistics of mean scores and standard deviation are provided for the 2013 and 2014 OITE
administrations. As the subgroup analyses are comprised of subsequent retesting of the same participants, a paired t-test with p-values is provided.

Journal of Surgical Education  Volume ]/Number ]  ] 2016 3


Correllaon of PASS quesons
answered correctly versus 2014 OITE
RAW score

240

220

200
2014 OITE RAW Score

180

160

140

120

100
0.4 0.5 0.6 0.7 0.8 0.9 1
PASS % Correct

Pearson Correlation Coefficient 0.649


R Standard Error 0.009
t 6.98
p-value <0.05
H0 (5%) rejected

FIGURE 3. Linear regression model demonstrating correlation between percentage of PASS milestone questions answered correctly and 2014 OITE
raw scores. Participants who had completed at least 1 PASS test (Z25 questions answered) were included in the analysis (n 70).

10 (signicantly useful) for OITE preparation, respondents States Medical Licensing Examination steps 1 and 2 scores
reported on average 6.9 (SD 2.24). Furthermore, nearly with OITE and ABOS performance.1,2,4
90% (38/42) noted they would prefer to complete a similar Although success on the ABOS Part 1 may be predicted
structured review curriculum before future OITEs. When by residents raw test-taking abilities, a residents orthopedic
asked specically about study habits, practice questions fund of knowledge and problem-solving capabilities may
appeared to be the most common study technique, with demonstrate the most robust predictive capacity for board
35.7% of respondents (15/42) using practice questions for examinations. Multiple authors have correlated OITE
51% to 75% of their OITE preparation time and 50.0% of performance with success on board examinations such as
respondents (21/42) using practice questions for 76% to the ABOS Part 1 and the Royal College of Physicians and
100% of their OITE preparation time (Figs. 4-6). Surgeons of Canada (a Canadian orthopedic board

DISCUSSION
Various publications have sought to establish correlations
and predictive capacity of standardized tests taken by
orthopedic residents to the ABOS Part 1 licensing exami-
nation. Although orthopedic residents historically have
performed well on standardized examinations, the interest
in predicting ABOS outcomes is because of the high stakes
nature of board examinations and the desire to predict
FIGURE 4. Selected responses from the resident participation survey.
likelihood of success on the boards. For example, several Responses for Would you prefer a similarly structured curriculum before
authors have demonstrated predictive correlations of United future OITEs? (n 42 responses).

4 Journal of Surgical Education  Volume ]/Number ]  ] 2016


(i.e., eliminating the 2013 PGY 4 group from the overall
analysis) would see statistically signicant improvements in
OITE performance; however, this was not one of our
intended subgroups for analysis. The reported utility of
the PASS curriculum was high, as 90% of participants
reviewed the curriculum positively and likely to use again
with even distributions of reported satisfaction across each
PGY class. After analyzing our results, we propose the
material available on Orthobullets PASS curriculum may be
better suited to achieving competence within the general
knowledge base of orthopedics, whereas senior residents
who have already mastered these topics may not see similar
incremental improvement with merely a review of high-
yield topics. Although review of performance on PASS
FIGURE 5. Selected responses from the resident participation survey. curriculum questions demonstrated good correlative
Responses for Please rate the utility of the PASS curriculum for OITE
preparation (1no utility, 10signicant utility), results stratied by capacity with 2014 OITE raw scores, we are hesitant to
PGY level. (n 42 responses). report predictive capacity of the PASS curriculum for OITE
performance from our small cohort (n 57, participants
certication examinations).2,3,5,6 However, as the ABOS with 150 completed PASS questions or greater).
Part 1 is a licensing examination with a pass/fail format, Although multiple studies have been conducted on
more interest has been focused on identifying predictive review materials and curricula for OITE preparation, we
tools for residents at risk of failing. Dougherty et al.1 believe ours is the highest quality data to be reported.11,12
reported a 57% chance of failing the ABOS Part 1 for We report the rst multicenter data from East Coast and
residents with mean OITE scores below the 27th percentile. Midwest residency programs, including both D.O. and M.
Similarly, Swanson et al.6 noted PGY 5 residents with D. programs, undergoing a structured curriculum with the
OITE scores below the 10th percentile were more likely to purpose of improving OITE scores and core orthopedic
fail the ABOS Part 1 than those who scored above the 50th knowledge. Our study improves upon prior research
percentile (18% versus 3%, respectively). Accordingly, through increased sample sizes (n 71 residents, 5 residency
residency programs have placed signicant emphasis on programs) and use of a well-dened, reproducible curricu-
OITE preparation, as programs with an increased emphasis lum, improving generalizability, and relevance of our studys
on OITE performance perform better on the OITE.7 outcomes. Although comparative studies have demonstrated
Multiple review materials are available for OITE prepa- improvements in resident OITE performance with imple-
ration, ranging from review books (e.g., American Academy mentation of a standardized curriculum, most are under 25
of Orthopaedic Surgeons [AAOS] Comprehensive Review residents and limited to 1 residency program.11,12
and Millers Review of Orthopaedics), practice questions The Orthobullets PASS curriculum study does demon-
from Orthobullets and AAOS, and review websites such as strate several limitations. Our sample size (n 71) limits
Orthobullets.com and WheelessOnline.com. The most reporting of subgroup analysis and predictive capacity of the
commonly used and recommended sources are practice
questions modeled on previous OITE questions, such as
Orthobullets and AAOS self-assessment examinations.8,9
Our data similarly suggested a preference for practice
questions, with 85% of participants using practice questions
for over 50% of their examination preparation time.
Beyond providing practice questions, the PASS curriculum
uses a structured curriculum to maintain steady progress
with examination preparation, as daily reading and more
hours committed to studying have correlated to increased
OITE scores.10 In addition, a daily reminder with a more
proactive format suggests a more consistent approach with
more focused topical learning.
Our results demonstrate improved performance on OITE
examinations with the PASS curriculum intervention, with
the junior residents subgroup (2013 PGY 1-2) displaying
FIGURE 6. Selected responses from the resident participation survey.
statistically signicant improvement in scores. Further Responses for Would you prefer a similarly structured curriculum before
review of the data suggest a 2013 PGY 1 to 3 subgroup future OITEs?, results stratied by PGY level (n 42 responses).

Journal of Surgical Education  Volume ]/Number ]  ] 2016 5


PASS curriculum. Furthermore, as the project did not and ABOS Part I examination performance. South Med
collect participant ABOS Part 1 data, we cannot directly J. 2014;107(12):746-750.
comment on the efcacy of the PASS curriculum for
4. Black KP, Abzug JM, Chinchilli VM. Orthopaedic in-
improving ABOS Part 1 outcomes. We also were not able
training examination scores: a correlation with
to correlate survey results and OITE/PASS scores, limiting
USMLE results. J Bone Joint Surg Am. 2006;88(3):
our ability to associate subjective perceptions of utility, and
671-676.
objective efcacy of the PASS curriculum. Despite these
limitations, our data suggest a valuable role for the PASS 5. Yen D, Athwal GS, Cole G. The historic predictive
curriculum in the preparation of residents for the OITE value of Canadian orthopedic surgery residents ortho-
with demonstrated efcacy through improved OITE scores pedic in-training examination scores on their success
and high perceived utility from curriculum participants. on the RCPSC certication examination. Can J Surg.
2014;57(4):260-262.
6. Swanson D, Marsh JL, Hurwitz S, et al. Utility of
CONCLUSION AAOS OITE scores in predicting ABOS Part I out-
Although the scores incrementally increased for the group as comes: AAOS exhibit selection. J Bone Joint Surg Am.
a whole, the junior resident subgroup (PGY 1-2) produced a 2013;95(12):e84.
statistically signicant increase in OITE scores after com- 7. Evaniew N, Holt G, Kreuger S, et al. The orthopaedic
pletion of the PASS curriculum. Furthermore, the residents in-training examination: perspectives of program direc-
have largely reported positive feedback regarding the per- tors and residents from the United States and Canada.
ceived utility of the curriculum with 90% preferring to J Surg Educ. 2013;70:528-536.
repeat a similar curriculum for future OITE preparation.
We propose that the Orthobullets PASS curriculum is a 8. Krueger CA, Aden J. Is there an association between
useful tool for OITE preparation, especially for junior study materials and scores on the American Board of
residents, with a focus on achieving competency for a broad Orthopaedic Surgeons Part 1 examination? J Surg
fund of orthopedic knowledge, whereas mastery of these Educ. 2014;71:375-384.
topics would be better suited through using complementary 9. LaPorte DM, Marker DR, Seyler TM, Mont MA,
sources. Frassica FJ. Educational resources for the Orthopedic
In-Training Examination. J Surg Educ. 2010;67(3):
135-138.

REFERENCES 10. Miyamoto RG Jr, Klein GR, Walsh M, Zuckerman


JD. Orthopedic surgery residents study habits and
1. Dougherty PJ, Walter N, Schilling P, Najibi S, performance on the orthopedic in-training examina-
Herkowitz H. Do scores of the USMLE Step 1 and tion. Am J Orthop. 2007;36(12):E185-188.
OITE correlate with the ABOS Part I certifying
11. Weglein DG, Gugala Z, Simpson S, Lindsey RW.
examination? A multicenter study. Clin Orthop Relat
Impact of a weekly reading program on orthopedic
Res. 2010;468(10):2797-2802.
surgery residents in-training examination. Orthopedics.
2. Dyrstad BW, Pope D, Milbrandt JC, Beck RT, 2015;38(5):e387-393.
Weinhoeft AL, Idusuyi OB. Predictive measures of a
12. Klena JC, Graham JH, Lutton JS, Temple JL, Beck
residents performance on written Orthopaedic Board
JD. Use of an integrated, anatomic-based, orthopaedic
scores. Iowa Orthop J. 2011;31:238-243.
resident education curriculum: a 5-year retrospective
3. Ponce B, Savage J, Momaya A, et al. Association between review of its impact on orthopaedic in-training exami-
orthopaedic in-training examination subsection scores nation scores. J Grad Med Educ. 2012;4:250-253.

6 Journal of Surgical Education  Volume ]/Number ]  ] 2016