ORIGINAL ARTICLE
Does simulation booster impact retention of resuscitation
procedural skills and teamwork?
J Bender1,2, K Kennally1, R Shields1 and F Overly2,3
OBJECTIVE: The Neonatal Resuscitation Program (NRP) has transitioned to a simulation-based format. We hypothesized that
immersive simulation differentially impacts similar trainee populations resuscitation knowledge, procedural skill and teamwork
behavior.
STUDY DESIGN: Residents from NICU and non-NICU programs were randomized to either control or a booster simulation
7 to 10 months after NRP. Procedural skill and teamwork behavior instruments were validated. Individual residents resuscitation
performance was assessed at 15 to 18 months. Three reviewers rated videos.
RESULT: Fifty residents were assessed. Inter-rater reliability was good for procedural skills (0.78) and team behavior (0.74)
instruments. The intervention group demonstrated better procedural skills (71.6 versus 64.4) and teamwork behaviors (18.8
versus 16.2). The NICU program demonstrated better teamwork behaviors (18.6 versus 15.5) compared with non-NICU program.
CONCLUSION: A simulation-enhanced booster session 9 months after NRP differentiates procedural skill and teamwork behavior
at 15 months. Deliberate practice with simulation enhances teamwork behaviors additively with residents clinical resuscitation
exposure.
Journal of Perinatology (2014) 34, 664668; doi:10.1038/jp.2014.72; published online 24 April 2014
1
Department of Pediatrics, Women & Infants' Hospital, Providence, RI, USA; 2Warren Alpert School of Medicine at Brown University, Providence, RI, USA and 3Lifespan Medical
Simulation Center, Rhode Island Hospital, Providence, RI, USA. Correspondence: Dr J Bender, Department of Pediatrics, Women & Infants' Hospital, 101 Dudley St, Providence,
RI 02905, USA.
E-mail: gbender@wihri.org
Received 2 December 2013; revised 7 March 2014; accepted 13 March 2014; published online 24 April 2014
Simulation improves procedural skills and teamwork
J Bender et al
665
even with this more conservative target the study may be underpowered were coached to resuscitate realistically, request help as necessary, do all
to detect procedural skill effects, teamwork behavioral effects or subgroup steps and verbalize actions and plans. They were encouraged to explore
comparisons. the code cart, practice intubation and recognize mannequin cues of
cyanosis, heart rate, breath sounds and movement before the simulation.
Each resident individually managed one scenario for 20 min. The other 2 to
Subjects 4 residents in each session rotated through educational stations on airway
Investigators recruited all medical school graduates at two Brown practice, umbilical venous catheter placement or unrelated Pediatric
University afliated residency programs (Memorial Hospital of Rhode Advanced Life Support (PALS) scenarios. Residents did not receive
Island Family Practice, Hasbro Childrens Hospital categorical Pediatrics) feedback or observe each others performance until post-simulation22
over two consecutive years. Each resident was followed for 2 years after group debrieng. Portions of each residents run were highlighted during a
experiencing the standard NRP Provider course at the start of residency. 75- min round-table intermediate23 level-facilitated discussion. Trained24
NRP instructors taught the courses in the fth edition format, consisting of debriefers guided residents to explore specic procedural and teamwork
6 to 8 h of standardized lecture intermixed with skills practice and multiple learning objectives. This instructional design provided individual perfor-
choice tests, culminating with a mega code covering the entire algorithm. mance assessments while optimizing the time allotted for reective
The two residency programs provided different exposure to clinical learning during debriengs.25
resuscitation events. Pediatric residents spent 2 months in their rst year in
a high volume level III neonatal intensive care unit (NICU) at Women and
Infants Hospital of Rhode Island, including 2 weeks on a busy delivery Validated performance instruments
service. Annual delivery volume was ~ 9000 deliveries, with a 32% cesarean Performance was evaluated with novel customized instruments. Assessment
section rate and 230 infants born with o 1500 g birth weight. Residents scales evaluating procedural skills and team behaviors during simulated
received consistent mentoring from neonatal nurse practitioners on the scenarios have been developed17,26,27 and validated with acceptable inter-
delivery service during day shifts, with frequent independent practice on rater reliability (kappa 0.42 to 0.59).2830 To improve discrimination of subtle
night shifts. The curriculum included regular lectures and case review on procedural differences, the existing NRP Megacode Assessment Form20 was
delivery room stabilization and management. For this study, these modied to reect specic procedural learning objectives. Iterative
residents were labeled NICU program. renement (see Supplementary Appendix I) by four NRP instructors resulted
The Family Practice residents experienced 2 months in a level II nursery. in good concordance between raters (kappa 0.62 to 0.83).
Typical patient volumes included fewer than 600 deliveries annually, 23% The teamwork behavior instrument was similarly rened from global
cesarean section rate and fewer than ve infants with o 1500 g birth team competency scales31,32 and delivery room-specic scales16,33 to t
weight. Monthly mock codes with case reviews formed the core of their our scenarios. Supplementary Appendix II describes the development and
resuscitation experience. Given the clear difference in clinical resuscitation validation of the instrument with key behavioral attributes organized into
exposure, these residents were labeled Non-NICU program. ve teamwork subscales. The overlapping Venn diagram construct may
complicate interpretation, but clearly benetted rater determination at
assessment. Each subscale was scored as predominantly (1) undermining
Randomization behaviors, (2) missed opportunities, (3) neutral, (4) good examples or (5)
Computer-generated a priori random assignment selected intervention consistent outstanding behaviors. The instrument discriminated favorably
residents, stratied within each program to reduce bias from pre-existing between similar performances when two videos were independently
skills. Randomization was revealed to schedule the intervention for half of reviewed by four raters.
the Pediatric residents and half of the Family Practice residents. These
residents had the simulation-enhanced booster 7 to 9 months after the
NRP course. The control arm consisted of the remaining rst year residents Data collection
from each program. All residents were assessed with simulation 15 to The three domains of assessment were knowledge, procedural skills
18 months after taking the NRP course. Residents in the intervention and teamwork behaviors. All participants had knowledge determined at
arm were exposed to simulation twice, once at the booster and once at baseline using the NRP questions scored from their initial provider course,
assessment. Residents in the control arm performed routine clinical duties as well as just before assessment using a subset of the same questions.
on the booster days. Residents did not receive feedback on any written components during the
study. Video recordings of simulations were retained for determination of
procedural skills and teamwork behaviors.
Simulation experience Residents evaluated their simulation experience on a 5-point Likert scale
Residents were briefed on the study in an open forum. Condentiality was using our routine simulation evaluation form. Residents also self-reported
emphasized, limiting both participant content sharing and research team related experiences that could alter their learning curve during study
sharing evaluations. Participants signed informed consent. Residents were simulations. These included familiarity with simulation, proximate certica-
scheduled into one of four half-day booster sessions and one of eight half- tions, clinical resuscitation events and intubation successes. Each of these
day assessment sessions. Each session included residents from NICU and was measured on a unique scale estimated to discriminate clinical relevance.
non-NICU programs, and assessment sessions inter-mixed residents from Familiarity with simulation-based medical education was scored per
intervention and control. Residents were not requested to review NRP or exposure to standardized patients, mannequins, animal models, mock
prepare for simulation. codes and high-delity simulations (1 to 5). Proximate certications included
Simulation exposures were standardized. Each booster simulation was NRP, Advanced Cardiac Life Support, or PALS scenarios (scored 1 to 4). The
an emergent cesarean section for fetal heart tone decelerations in a high- impact of clinical resuscitation events was scored (1 to 4) by the proximity of
delity simulated operating room. Two minutes after the residents arrival, the most recent NICU rotation for never, >6 months, 3 to 6 months, o3
a stressed obstetric team handed off a non-vigorous term infant covered months. Successful endotracheal intubation of a neonate scored (0 to 3) for
with blood and meconium. Each assessment simulation was a shoulder never, o4, 4to 7 or >7 intubations.
dystocia scenario in a chaotic labor room, delivering an unresponsive Condence in resuscitation may both modify the primary outcomes
neonate requiring resuscitation. Scenarios were designed around the and be affected by the booster intervention. Condence at the time of
essential steps of NRP. Vignettes were varied to reduce predictability, assessment was an average of response scores for the following questions.
changing descriptive elements (for example, gravida, social history) from How condent are you (1) resuscitating a non-vigorous term baby with
resident to resident without altering underlying physiologic corrections. meconium-stained amniotic uid, (2) managing an apneic baby, (3)
Residents who moved efciently through the scenario within 15 min were intubating a late-preterm baby, (4) managing a cyanotic, spontaneously
challenged with a tension pneumothorax, without impacting assessment breathing newborn and (5) assessing effective bag-mask ventilation.
score. Interventions were done on a mannequin (Laerdal, SimNewB) Participants used a scale 0 to 4, with higher numbers representing an
preprogrammed for scenario-specic physiologic responses. One NRP increased level of condence.
instructor consistently performed the central support role of newborn
nurse. She was scripted to enable progress without excessive prompting
and mistakes were not corrected during the simulation run. Data analysis
Each simulation-enhanced booster session started with a 30-min Three raters were standardized on the instruments by separate review of
orientation to simulation. Stated learning objectives were to (1) follow three videos followed by group feedback. They independently scored each
NRP guidelines and (2) demonstrate effective team leadership. Residents assessment video recording. They were all NRP instructors; two had been
Resident Enrollment
Excluded (n=2)
Not meeting inclusion criteria (n=0)
Declined to participate (n=0)
Other reasons (n=2)
Randomized (n=58)
Baseline
Allocation
Allocated to intervention (n= 29) Allocated to control (n= 29)
Received allocated intervention (n= 27) Received allocated control (n= 29)
Did not receive allocated intervention (n= 2) Did not receive allocated control (n=0)
Booster
NICU program (n= 15) Non-NICU (n= 12) NICU program (n= 16) Non-NICU (n= 13)
Follow-Up
Lost to follow-up (n= 6) Lost to follow-up (n=2)
Assessment
NICU program (n=14) Non-NICU (n= 9) NICU program (n=16) Non-NICU (n= 11)
Analysis
Analysed (n= 23) Analysed (n= 27)
Excluded from analysis (n= 0) Excluded from analysis (n= 0)
NICU program (n= 14) Non-NICU (n= 9) NICU program (n= 16) Non-NICU (n= 11)
Proximate certications 2.4 (0.7) 2.6 (0.9) 0.37 2.3 (0.7) 2.9 (0.9) 0.005
Simulation familiarity 2.9 (1.3) 2.5 (1.1) 0.33 3.0 (1.3) 2.2 (0.9) 0.03
Recent NICU exposure 2.5 (1.3) 2.0 (1.1) 0.17 3.0 (0.9) 1.0 (0) o0.001
Intubation successes 0.8 (0.6) 0.6 (0.8) 0.38 0.8 (0.7) 0.3 (0.6) 0.02
Resuscitation condence 2.5 (0.6) 2.2 (0.7) 0.14 2.5 (0.6) 2.0 (0.7) o0.007
Abbreviation: NICU, neonatal intensive care unit.
procedural skills (71.6 versus 64.4) and teamwork behaviors (18.8 Teamwork training integrated into the NRP course improves
versus 16.2) compared with control. The NICU program demon- behaviors in multiple disciplines37 that may persist 6 months after
strated better teamwork behaviors (18.6 versus 15.5) compared 2 h of simulation-based training.38 This study explored a separate
with non-NICU program. Subgroup analysis (Table 2) supported booster, as neither simulation nor teamwork was integrated into
these group and program effects. Procedural skills correlated well NRP at the time. We found simulation followed by attention to
(r = 0.86) with teamwork behaviors. Knowledge scores correlated teamwork principles during facilitated debrieng to be sufcient
weakly with either procedural skills (r = 0.66) or behavior (r = 0.51), to distinguish behaviors 6 months later. Subgroup analysis
and all correlated poorly (r = 0.28 to 0.55) with related experiences identied independent effects on teamwork behaviors from
and condence. simulation and clinical resuscitation exposure. Yet the residents
with the least clinical exposure did not display the best gains from
simulation, refuting our secondary hypothesis. The data support
DISCUSSION that deliberate practice has an incremental benet on teamwork
Effective resuscitation of the critically ill neonate requires behaviors regardless of source. As observed clinical performance
integrated tiers of knowledge, procedural skills and teamwork was outside the study scope, we can only presume that
behaviors. Essential NRP procedural skills successfully reach the persistently superior procedural skills and teamwork behaviors in
neonate given effective knowledge and teamwork. Simulation the simulation arena correlate with patient benet. We remain
may enhance each component.34 This study was done just before skeptical of over-interpreting knowledge or condence scores,
a fundamental transition in sixth edition NRP, shifting learner given their poor correlation with either skills or behavioral
contact hours focus from {knowledge and procedure} to competence.
{procedure to teamwork}. Deliberate practice with three simula- Several study limitations affect interpretation of the results.
tions within 6 months of NRP may improve performance.35 The Twenty percent attrition by intervention residents introduces
impact of a single booster session on performance or behavior potential selection bias. Rater bias was minimized by reviewing
is less clear. Two studies failed to demonstrate improved NRP videos more than 2 years after the simulations. Second, the study
performance with validated instruments immediately following36 was underpowered for procedural skill or teamwork behaviors,
or 2 weeks after (Van der Heide (2006)) a single simulation session. nding statistically signicant results less than the 15% targeted
The present study shows that a single simulation-enhanced difference for knowledge scores. Third, individual resident pro-
booster session 9 months after the initial NRP course measurably cedure and teamwork improvement is not knowable without
differentiated 15-month procedural skills and teamwork behaviors baseline measurement. Group outcome comparisons are reason-
from those not receiving the booster. Knowledge drifted equally able given randomization into equivalent cohorts. The study
in both groups. This did not undermine the intervention groups groups are balanced (Table 1) with respect to previous NRP and
superiority with essential procedural skills. While the measured related certications, simulation familiarity, NICU exposure and
benets from simulation (+7.2) may be clinically nominal, it was at live procedural success with intubation. Fourth, as high versus low
least as signicant (+5.5) as more extensive interim clinical tech simulation was not compared, we can only conclude that
resuscitation exposure. the environment was sufciently immersive. Fifth, alternate
Supplementary Information accompanies the paper on the Journal of Perinatology website (http://www.nature.com/jp)