A Methodology for Enhancing Crew Resource Management Training

Eduardo Salas and Carolyn Prince, Naval Air Warfare Center Training Systems Division, Orlando, Florida, Clint A. Bowers, University of Central Florida, Orlando, Florida, and Rene J. Stout, Randall L. Oser, and Janis A. Cannon-Bowers, Naval Air Warfare Center Training Systems Division, Orlando, Florida
Human error is an ever-present threat to the safe conduct of flight. Recently, applied psychologists have developed an intervention, crew resource management (CRM) training, designed to help prevent human error in the cockpit. However, as it is commonly applied within the aviation community, CRM lacks standardization in content, design, delivery, and evaluation. This paper presents a discussion of an applied program of research aimed at developing a methodology for the design and delivery of CRM training within the Navy. This long-term, theoretically based program of aviation team research included identification of skills to be trained, development of performance measures, application of instructional design principles, and evaluation of the training delivery. Our conclusion indicates that a systematic methodology for developing CRM training can result in better performance in the cockpit. Actual or potential applications of this research include any task environment in which teams are interdependent.

INTRODUCTION In the 1970s, hundreds of airline passengers on routine, scheduled flights lost their lives because each of three aircrews committed an error. In one incident, the crew failed to take fuel levels into consideration during problem solving; in the second incident, the crew did not monitor the altitude; and in the third incident, the crew misinterpreted an air traffic control communication. The crew members who committed the errors had tens of thousands of hours of flight experience, yet the errors committed should have been avoided by even the most inexperienced pilots. The crews were not members of a country in which standards of pilot training and certification were questionable, and each of these crews worked for a major air carrier. Two of the crews were flying domestic operations within the United States. In both domestic accidents, the crew members committed the

errors while responding to a potentially unsafe problem with the plane by taking extra time and care to troubleshoot or prepare (or both) for this unplanned circumstance. As a result of their lapses, their planes ended the flight in one case by crashing into a stand of trees in Oregon, and in the other case by crashing into the Florida Everglades. The third plane, on an international flight, had results so catastrophic that it sent shock waves throughout the world. This plane collided with another aircraft, immediately ending the lives of everyone aboard both planes. After 20 years, the aviation industry is still challenged by a haunting question: Why is the number of take-offs not equal to the number of safe landings? In the past 20 years, it has been commonly acknowledged that almost 60% to 80% of aviation incidents and accidents were attributable to human error in the cockpit (Foushee, 1984). This recognition led a number of applied psychologists to suggest an intervention aimed at improving human

Address correspondence to Eduardo Salas, Code 4961, Naval Air Warfare Center Training Systems Division, 12350 Research Parkway, Orlando, FL 32826-3275; salasea@navair.navy.mil. HUMAN FACTORS, Vol. 41, No. 1, March 1999, pp. 161172.

162 performance and, in particular, teamwork in the cockpit. This intervention, commonly referred to as crew resource management (CRM) training, now has a long history of research and practice in the air carrier industry (Wiener, Kanki, & Helmreich, 1993). On the military side, CRM training developments have also emerged. (The military labeled this team training intervention aircrew coordination training, but we will use the term CRM in this paper because it is most common in the airline industry and government regulatory agencies.) The U.S. Navy (in particular, the Marine Corps) enlisted the help of the Naval Air Warfare Center Training Systems Division about 10 years ago in improving the safety of its rotary wing fleet. Our response was to design and conduct a longterm program of research that began with theory building and moved into development of measures of performance, design of instruction, empirical testing, and evaluation. The purpose of this paper is to describe our efforts in this regard. To do this, we organized the presentation around the critical questions that have guided our research:
1. What is CRM, and, more specifically, what is CRM training? 2. Which theories provide a basis to develop CRM training? 3. Which skills underlie effective CRM? 4. Which instructional approaches and strategies are appropriate to impart CRM skills? 5. What evidence exists to support the effectiveness of CRM training?

March 1999 Human Factors how to conduct emergency procedures. It was also clear that on its own, human redundancy in the cockpit had not worked to make the system more secure; three people together in a single cockpit still overlooked a basic flight parameter. This led researchers to expand their view of what was required for effective aviation beyond technical aspects of the task (i.e., flying the aircraft) to include nontraditional competencies such as teamwork. According to Lauber (1984), this new way of thinking (labeled first as cockpit resource management and later as crew resource management) is defined as using all available resources information, equipment, and people to achieve safe and efficient flight operations (p. 20). Foushee and Helmreich (1988) added that
CRM includes optimizing not only the personmachine interface and the acquisition of timely, appropriate information, but also interpersonal activities including leadership, effective team formation and maintenance, problem-solving, decision-making, and maintaining situation awareness. (p. 4)

We conclude with a presentation of our methodology for developing CRM training and a word about reciprocity between training science and practice. What Is CRM and CRM Training? The three accidents discussed in the previous section help illustrate a persistent threat to safe aviation: human error caused by inadequate coordination among team members. Although there were minor mechanical failures in two of the accidents, each occurred because of the crews error. It is clear that the errors made in these accidents were not the result of inadequate technical training. All of those involved knew how to read the gauges and

Given this definition, researchers began to examine what training for CRM might include. According to Foushee and Helmreich (1988, p. 4), such training involves communicating basic knowledge of human factors concepts that relate to aviation and providing the tools necessary to apply these concepts operationally. At the time, CRM represented a new focus on crew-level (as opposed to individual-level) aspects of training and operations (p. 4). The major goal of the resulting training was to help stem the tide of accidents caused by so-called human error (Stone & Babcock, 1988, p. 553) by addressing situational, sociopsychological and other factors that influence aircrew performance (Caro, 1988, p. 258). When it came to actually developing training to address CRM issues, early attempts held almost exclusively that CRM training should target crew members attitudes toward teamwork (Chidester & Foushee, 1988; Helmreich & Wilhelm, 1991). In keeping with the definition of CRM, this approach to CRM training emphasized the social-psychological factors that influence crew performance. In fact, this work by Foushee, Helmreich, and others paved the way

CRM TRAINING for the aviation industry to consider softer human performance and teamwork issues to be legitimate concerns. This was no small accomplishment. The success of these scientists was to shift the emphasis in aviation training onto social interactions in the cockpit (i.e., crew coordination). However, early CRM training practices did not go far enough; we believe that there were two fundamental problems. First, initial CRM training overemphasized the affective, personality, and attitudinal aspects of crew coordination (i.e., the right stuff) at the expense of the behavioral aspects of the problem of crew coordination. Second, little guidance was available regarding how to train crew coordination skills. So, although these early efforts had the effect of enlightening the aviation community, they did not address the CRM training problem sufficiently. In the 10 years or so that have followed, the lack of a standardized methodology for developing CRM training (and the associated problem of disagreement about exactly what needed to be trained) caused a host of diverse CRM training programs to be developed. Whereas early programs emphasized attitudes toward teamwork, as described earlier, others were based on personality or skill, and still others on a combination of these (Byrnes & Black, 1993; Helmreich & Foushee, 1993; Yamamori & Mito, 1993). Furthermore, programs have varied in length from 1 h to 2 weeks. Training has been given in lectures, discussions, videotape observations, game-playing, classroom roleplay, mishap analyses, and both low- and highfidelity simulations. Some courses have used only one technique (e.g., lecture) and others have used several. Content has shown diversity as well and has included topics such as interaction styles, stress reduction, and automation issues, in addition to the subjects of workload management, advocacy, and situation awareness that are included in many programs. Some programs are clearly based on attitude change or skill development, but others, for which training has been composed of elements copied from the programs of other organizations, have no discernible basis. In sum, the lack of agreement regarding what CRM training should include and how it should be accomplished has led to confusion

163 and, in many cases, adoption of atheoretical and suboptimal programs. To address these deficiencies, our work for the U.S. Navy adopted a more systematic approach to developing CRM training that was consistent with our team training work in similar domains. As a starting point, we clarified the concept of CRM by extracting the notion of teamwork (or crew coordination) from the definitions offered earlier as the essential ingredient of CRM. We selected this focus because the evidence gleaned mostly from accident reports suggested that it was the lack of coordination of team member resources or the underutilization of team member resources (or both) that accounted for most of the errors (Foushee, 1984). Hence, we define CRM as being a set of teamwork competencies that allow the crew to cope with situational demands that would overwhelm any individual crew member (we elaborate on these competencies in a later section). Flowing from this definition, we consider CRM training to be a family of instructional strategies designed to improve teamwork in the cockpit by applying well-tested training tools (e.g., performance measures, exercises, feedback mechanisms) and appropriate training methods (e.g., simulators, lectures, videos) targeted at specific content (i.e., teamwork knowledge, skills, and attitudes). It should be noted that our initial emphasis in CRM training was on skills (in part because other programs emphasized attitudes). More recently, our work has been focused more directly on the knowledge underlying effective teamwork (see, e.g., Stout, Salas, & Kraiger, 1997), but this work is not covered here. In the following sections, we describe the program of research that we conducted to enhance the training of CRM skills and the resulting methodology that we developed. As noted, we do this by addressing a number of crucial questions that guided our work. Which Theories Provide a Basis for CRM Training Development? The problem of improving teamwork in a task as complex as aviation provides a formidable challenge. Clearly, no single approach is likely to address the entire problem. Moreover, without a strong theoretical foundation

164 one that rests on an understanding of individual and team performance, human learning and skill acquisition, and pedagogy it is not likely that effective training will be developed or fielded. In fact, we submit as others have that there is nothing more practical than a good theory (Lewin, as cited in Marrow, 1969). This statement embraces the philosophy behind the design of all of the team training strategies that we have developed and tested (Salas & Cannon-Bowers, 1997). Fortunately, there are a number of theoretical perspectives that can provide a strong foundation for CRM training development. For example, Tannenbaum, Beard, and Salas (1992) described a theoretical model of team performance that includes consideration of a variety of factors that might influence the performance of flight crews. This model has already been used as a foundation for research in CRM for automated cockpits (see Bowers, Thornton, Braun, & Salas, 1998). Recent theoretical advances in cognitive psychology also offer promise in guiding what to train and how to train it. For example, Jentsch (1997) developed a metacognitive training strategy for junior first officers aimed at improving their ability to determine the appropriate time to act. The training focused on teaching the pilots monitoring skills and improving their ability to mentally simulate possible outcomes of either action or inaction in a particular situation. Pilots who had received the training performed significantly better in those situations that required judgment (whether to act or not) than those who did not receive the training. Similarly, using shared mental model theory (Cannon-Bowers, Tannenbaum, Salas, & Volpe, 1995) as a foundation, Stout, Salas, and Fowlkes (1997) found that participants who received CRMtype training improved their knowledge structures related to teamwork as compared with a control group that did not receive training. In addition, the trained group demonstrated 8% more teamwork skills when it was appropriate to do so during a simulated mission. Thus, training positively influenced aviators knowledge structures and performance. Although our work in CRM training has been influenced in one way or another by all

March 1999 Human Factors of the theories just described, we based our notions about training teamwork in the cockpit largely on the body of work conducted in the 1980s by Salas, Morgan, Glickman, and colleagues (see, e.g., Morgan, Salas, & Glickman, 1994). Briefly, these researchers studied military command and control teams and found behaviors that were consistent across effective teams. Specifically, they found that using closed-loop communication, predicting each others behavior, performing self-correction, and providing motivational and task reinforcement led to better team performance (for details, see McIntyre & Salas, 1995). Which Skills Underlie Effective CRM? We often found that aviators who attended early CRM training commented that the course was interesting and that they learned a lot about themselves, but they did not know what to do to change the way they operated when they got back in the cockpit. As Ryle (1949) pointed out, there is a difference between knowing that something should be done and knowing how to do it. Therefore, based on our notions about teamwork, we adopted a behavioral perspective to complement the attitude and personality-based work that was ongoing in the commercial sector in the early 1990s. This meant that from a learning standpoint, we needed to identify the requisite skills that enable effective performance in naval aviation. This training was designed to provide aviators with behaviors that they could take back to the cockpit to improve their performance. Thus, our vision for CRM training emphasizes what to do rather than how to feel. In order to fulfill this vision, we synthesized the scientific literature (Salas, Dickinson, Converse, & Tannenbaum, 1992), made multiple observations of crews from several communities performing in full-mission simulations (Prince & Salas, 1993), and conducted structured interviews with over 200 aviators from a variety of operational communities. Through an integration of information gained from these three divergent sources, we identified a set of teamwork constructs with associated behaviors that we predicted would lead to effective teamwork in the cockpit (Prince & Salas, 1993). Along the way, we developed and

CRM TRAINING tested a variety of approaches to identifying fleetspecific coordination behaviors (see Bowers, Baker, & Salas, 1994; Bowers, Morgan, Salas, & Prince, 1993). These tools exist and can be used to develop skills-based training today (Salas & Cannon-Bowers, 1997; Salas & CannonBowers, in press). In sum, by integrating theoretical models of teamwork and human learning with training needs-analysis tools, we addressed the first deficiency of CRM training noted previously by identifying the specific skills and associated behaviors that should improve coordination in the cockpit. The emphasis on specific skills offers several advantages from the theoretical and evaluation perspective (described in a later section). However, training success is also dependent upon ones ability to train those skills, which brings us back to the issue of how to train CRM skills. How Can CRM Skills Best Be Trained? Pertinent to the question of how best to train teamwork in the cockpit, we contend that the failure to create opportunities to practice newly acquired skills was the greatest shortcoming of early CRM training. In essence, early approaches failed to take advantage of literature that relates to human learning and skill acquisition (Kolb, 1984). Consequently, our efforts have endeavored to improve upon the traditional simulator-based practice experience, which typically lacked specific measurement and feedback, to more directly reinforce the targeted teamwork skills. For example, a program of research on expanded opportunities for teamwork skill practice (and feedback) has shown that CRM skills trained in one situation transfer to another (Brannick, Prince, Salas, & Stout, 1997; Prince, Brannick, Prince, & Salas, 1997). In addition, we have shown that this method of instruction provides enhanced training transfer as compared with game-playing, a technique currently used in many CRM programs (Brannick, Prince, et al., 1997; see Jentsch & Bowers, 1998 for a thorough discussion). Once our instruction was developed, it was necessary to assess the effectiveness of the training. The methods we employed to do this are described next. What Evidence Exists to Suggest That CRM Training Is Effective?

165

As with any other complex human performance problem, the issue of establishing a criterion against which to evaluate the success of CRM training is difficult. At first glance, it is tempting to use broad performance indices to evaluate CRM effectiveness. Common examples include the use of accident rates or the number of mishaps attributed to CRM errors. However, there are several concerns regarding these types of indices: (a) CRM has become part of aviation vernacular (perhaps accident investigators are citing it as a cause more frequently); (b) aircraft are becoming increasingly automated, which places considerable additional coordination demands on crews (Bowers, Deaton, Oser, Prince, & Kolb, 1995; Thornton, Braun, Bowers, & Morgan, 1992); and (c) accidents that happened are a poor index of accidents that did not happen. There are numerous anecdotal accounts of accidents that were prevented with the use of CRM skills, but do the accidents that did happen negate these reported benefits? A better approach is one that is multifaceted and considers several levels of evaluation, including trainee reactions, extent of learning, extent of performance change, and impact on organizational effectiveness (Cannon-Bowers et al., 1989; Ford, Kozlowski, Kraiger, Salas, & Teachout, 1997). This type of approach includes a variety of data points for consideration, such as trainee evaluation, specific indices of learning, evidence of application, and so forth. A multifaceted approach is not only more thorough than an approach focusing only on reactions or attitudes, but it also provides an improved ability to assess which aspects of training work and which do not work. Hence, it is also useful in identifying specific areas in need of remediation, because it assesses multiple aspects of learning/skill-acquisition. Using a multifaceted approach, we conducted formal evaluations of our CRM training interventions in four aircraft communities with a total of 55 crews. In each of the evaluations we used a control group as a comparison so that we could actually assess the value of our training. We also used raters or assessors of performance who were

166 blind to conditions. Finally, we used a measure of performance that was designed to identify whether aviators were demonstrating behaviors (identified through task analysis and assessment of training manuals and operating procedures) that had been delineated as more effective by fleet subject-matter experts (as described in Fowlkes, Lane, Salas, Franz, & Oser, 1994). We obtained evidence that our variety of CRM training is effective, as evidenced by average performance improvements of 8% to 20% (for empirical evidence see Salas, Fowlkes, Stout, Milanovich, & Prince, in press; SmithJentsch, Salas, & Baker, 1996; Stout, Salas, & Fowlkes, 1997). That is, on average we observed 8% to 20% more teamwork behaviors exhibited in the cockpit by crews that were trained than by crews that were not trained. This may not seem like much, but it is actually impressive considering that many of the participants in these studies were experienced crews who actually changed their cockpit performance based on a relatively short training intervention. Furthermore, we submit that performing even one more teamwork behavior (when required) can make a difference between mission success and failure (and sometimes between life and death). Similar efforts in other military communities have also shown positive results (Cannon-Bowers & Salas, 1998; Leedom & Simon, 1995; Salas, Cannon-Bowers, & Johnston, 1997). In addition, we have conducted studies to test specific components of the training. For example, Smith-Jentsch et al. (1996), using a variety of measurement approaches, demonstrated the effectiveness (74% performance improvement for the trained group) of skillsbased assertiveness training, which is a component of our CRM program. Jentsch (1997) also used a multidimensional approach in assessing the effectiveness of metacognitive training for junior first officers. Thus, the data are encouraging. We have shown that even experienced aviators can learn new ways to behave in the cockpit and improve their performance. In the future, we hope that others will also attempt to evaluate the CRM training they develop; in fact, this should be an ongoing process. Given the diffi-

March 1999 Human Factors culties noted earlier, it is the responsibility of all of the experts (researchers and practitioners), the airlines, the fleets, and cognizant agencies to devise acceptable methods for evaluating CRM training effectiveness and then to make the necessary commitment to, and investment in, conducting these studies. A METHODOLOGY FOR DEVELOPING CRM TRAINING Table 1 shows an overview of the methodology that we employed for developing effective CRM training. It is based on the work we conducted to develop CRM training as well as guidelines from the literature (Cannon-Bowers & Salas, 1998; Salas & Cannon-Bowers, in press; Salas, Cannon-Bowers, & Blickensderfer, 1997; Swezey & Salas, 1992). It is similar to the event-based approach to training that we have applied in other domains (Dwyer, Fowlkes, Oser, Salas, & Lane, 1997; Johnston, Cannon-Bowers, & Smith-Jentsch, 1995; Oser, Dwyer, & Fowlkes, 1995). For any particular community that we worked with, we followed the steps illustrated in Table 1 to develop theoretically rooted, behaviorally based, communityspecific CRM training (for examples, see Prince & Salas, 1989; Stout, Salas, & Fowlkes, 1997). The process delineated in Table 1 began with a thorough analysis of the aircraft mission and procedures and their impact on crew coordination demands. The second step was to assess the coordination demands directly (for details, see Bowers et al., 1993). This analysis allowed us to identify specific tasks that had a team component so that we could focus attention on these tasks in training. Next, we employed our theoretical notions regarding the nature of teamwork competencies to arrive at a set of coordination skills to be trained. Each skill was then translated into a training objective and used as the basis to develop scenarios or exercises that would allow the skill to be practiced. At the same time, we developed measures of performance and associated measurement tools that were linked to each of the training objectives (Baker, Prince, Shrestha, Oser, & Salas, 1993; Dwyer et al., 1997; Prince, Oser, Salas, &

CRM TRAINING Woodruff, 1993). Our goal was to focus on observable aspects of teamwork behavior so that raters could readily assess whether the behavior was demonstrated sufficiently (Fowlkes et al., 1994). We then used the performance measurement data as a basis for developing feedback. Because feedback mechanisms were crucial to the training, we spent considerable effort tailoring our assessment tools so that they could form the basis of effective feedback (Fowlkes et al., 1994). Finally, whenever possible, we conducted training-effectiveness evaluations to determine the impact of our training on cockpit performance. Description of the Training Employing the methodology just described, we developed an instructional strategy for CRM training with behavioral modeling as its basis (Salas et al., in press; Stout, Salas, & Fowlkes, 1997). The training sessions followed an information-demonstration-practice-feedback sequence. The training began with a lecture (delivered live) explaining important teamwork skills. Next, videotaped models of effective and ineffective behavior were shown to the trainees. Following this demonstration phase, trainees were provided with the opportunity to practice on carefully crafted scenarios (as described earlier), and their performance was tracked. Finally, we provided feedback that was relevant to the behaviors that we sought to train. Using this general strategy, we were able to train the repertoire behaviors that allowed crew members to exhibit effective teamwork in the cockpit, as evidenced by the effectiveness data described earlier. It should be noted that the methodology presented here should also be useful in developing other types of CRM training strategies. In other words, there may be other ways to train various aspects of teamwork in the cockpit; our argument is that it is the systematic process for devising it that helps ensure success. Further, we are aware that many CRM training programs have been developed that are consistent with one or even several of the steps shown in Table 1. Again, we argue that in order to develop and evaluate effective CRM training, all of the activities in Table 1 must be complet-

167 ed. Otherwise, there is a risk of suboptimizing; that is, there is a risk of developing training that addresses only parts of the problem or lacks a sound theoretical foundation. We also submit that this methodology may be useful in guiding the development of team training for domains other than aviation. RECIPROCITY BETWEEN TRAINING RESEARCH AND PRACTICE Given what we have presented thus far, it is clear that we have made progress in understanding CRM and applying sound training principles to improve it. However, as with other areas of training, the results of training research often do not influence training practice as much as they could. We suggest that there is still difficulty infiltrating operational communities with the results of team performance and training efforts. This problem is certainly not limited to CRM training; we have maintained that the application of research results to practice is a problem confronted by many areas of applied psychology (Cannon-Bowers, Tannenbaum, Salas, & Converse, 1991; Salas, CannonBowers, & Blickensderfer, 1997). Too often we find that training developers fail to make use of the results of training research. There may be many reasons for this. In the past, we have contended that it is attributable to a lack of translation mechanisms. That is, researchers and practitioners lack a common forum and language for communicating. Moreover, given that our work is conducted in a military context, it is sometimes dismissed as not generalizing to many commercial tasks, even though it could offer a substantial reduction in costs incurred in re-inventing the wheel. Furthermore, by taking advantage of the research done with military participants who often become commercial pilots after their military careers it is likely that the state of the art will improve more quickly than it has in the past decade. CONCLUDING REMARKS In conclusion, the science and practice of CRM training has evolved considerably over the past 10 years. There are now viable (and tested)

TABLE 1: A Methodology to Design and Deliver CRM Training Products -Mission specific context/examples -General understanding of coordination demands within the task Sources Fowlkes, Lane, Salas, Franz, & Oser (1994); Hartel, Smith, & Prince (1991); Prince & Salas (1989) Bowers, Morgan, & Salas (1991); Bowers, Morgan, Salas, & Prince (1993) Cannon-Bowers & Salas (1997); Cannon-Bowers, Tannenbaum, Salas, & Volpe (1995); Prince & Salas (1993); Stout, Salas, & Fowlkes (1997) -List of targets for training -Full list of generic and taskspecific training objectives Salas & CannonBowers (1997); Stout, Salas, & Kraiger (1997); Swezey, Llaneras, Prince, & Salas (1991)

Step

Description

1. Identify operational/mission requirement

Review existing training curriculum, including course master material lists, instructor guides, standard operating procedures (SOPs); interview aviation subject matter experts (SMEs); observe crews performing missions; review relevant mishap/accident reports.

2. Assess team training needs and coordination demand

Use same data sources as in Step 1, with emphasis on identifying deficiencies in existing team training and specifying all tasks required to perform missions that involve a teamwork element.

-Coordination demand analyses -List of tasks requiring coordination

168

3. Identify teamwork competencies Link team training needs to a theory of team performance and knowledge, skills, and that allows delineation of competencies (our emphasis attitudes (KSAs) initially was on skills) required to perform each of the team tasks identified in Step 2.

-Set of knowledge, skills, and attitudes

4. Determine team training objectives

For each teamwork KSA, develop a training objective that can be empirically evaluated to determine whether or not it was accomplished.

5. Determine instructional delivery method

The method for accomplishing the instruction should be specified (e.g., information, demonstration, or practice and feedback, or all) in this step. Consideration should be given to costs and availability of simulators.

-Lectures -Videos -Role-play exercises -PC-based methodologies -High-fidelity simulators -Accident reviews as analysis -Training curriculum -Valid realistic scenario(s), including all relevant peripheral support Dwyer, Fowlkes, Oser, Salas, & Lane (1997); Fowlkes, Dwyer, Oser, & Salas (1998); Prince, Oser, Salas, & Woodruff (1993)

Baker, Prince, Shrestha, Oser, & Salas (1993); Beard, Salas, & Prince (1995)

6. Design scenario exercises and create opportunities for practice

Design scenarios or exercises in which events are embedded to provide trainees an opportunity to demonstrate each of the required KSAs identified in objectives in which accomplishment requires practice and feedback.

7. Develop performance assessment/measurement tools

169

In conjunction with scenario design, develop measures that can reliably assess whether each of the KSAs was demonstrated at an observable behavioral level.

-Behaviorally-based checklists -Subjective evaluation forms -Outcome metrics and criteria

Baker & Salas (1992); Brannick, Salas, & Prince (1997); Fowlkes et al. (1994) -Instructor training -Debriefing checklists and guides Salas & CannonBowers (1997); Smith-Jentsch, Zeisig, Acton, & McPherson (1998)

8. Design and tailor tools for feedback

Design or tailor (or both) measurement tools for use in debrief, in which trainees are made aware of those required team behaviors that they did and those that they did not perform successfully. This tool should also help instructors diagnose the causes of poor performance and provide guidance for specific improvement in future operations.

9. Evaluate the extent of improved teamwork in the cockpit

Design experiments to assess the effectiveness of the training. Because of operational constraints, quasiresearch methods may need to be applied.

-Reaction data -Learning data -Knowledge-acquisition data -Transfer data -Performance data

Fowlkes et al. (1994); Salas et al. (1998); Stout, Salas, & Fowlkes (1997)

170 theories, methods, principles, approaches, and content available to shape the design and delivery of CRM training. Much has been done and much remains to be done. There is little doubt that our colleagues performing research in decision making (Orasanu, 1993), situation awareness (Endsley, 1995), leadership (Pettit & Dunlap, 1997), and other topics will contribute useful elements to CRM training as well. In fact, our own interests have now turned toward training the cognitive components of teamwork in the cockpit (Salas, Bowers, & CannonBowers, 1995; Salas, Prince, Baker, & Shrestha, 1995; Shrestha, Prince, Baker, & Salas, 1995; Stout, Cannon-Bowers, & Salas, 1996). We believe that by exploiting advances in training technology and methods and, perhaps more important, adopting a systematic methodology for developing training teamwork in the cockpit will improve. It is up to all of us scientists and practitioners to see that the potential benefits of CRM training are realized in the aviation community. ACKNOWLEDGMENTS The views expressed here are those of the authors and do not reflect the official position of the organization with which the authors are affiliated. Several colleagues throughout the years have contributed to this program of research: David Baker, Rebecca Beard, Maureen Bergondy, Mike Brannick, Jim Driskell, Jennifer Fowlkes, Florian Jentsch, Danielle Merket, Dana Milanovich, Ben Morgan, Jr., Elizabeth Muiz, Jerry Owens, Ashley Prince, Lisa Shrestha, Bob Swezey, Kimberly Smith-Jentsch, Mike Lilienthal, and Scott Tannenbaum. We appreciate their efforts in making this program of research offer solutions to the aviation community. We would also like to thank the 1400 or so aviators who have donated their time over the years to help us conduct this research. REFERENCES
Baker, D. P., Prince, C., Shrestha, L., Oser, R. L., & Salas, E. (1993). Aviation computer games for crew resource management training. International Journal of Aviation Psychology, 3(2), 143156. Baker, D. P., & Salas, E. (1992). Principles for measuring teamwork skills. Human Factors, 34, 469475.

March 1999 Human Factors

Beard, R. L., Salas, E., & Prince, C. (1995). Enhancing transfer of training: Using role-play to foster teamwork in the cockpit. International Journal of Aviation Psychology, 5(2), 131143. Bowers, C. A., Baker, D. P., & Salas, E. (1994). Measuring the importance of teamwork: The reliability and validity of job/task analysis indices for team-training design. Military Psychology, 6, 205214. Bowers, C. A., Deaton, J., Oser, R. L., Prince, C., & Kolb, M. (1995). Impact of automation on aircrew communication and decision-making performance. International Journal of Aviation Psychology, 5(2), 145168. Bowers, C. A., Morgan, B. B., Jr., & Salas, E. (1991). The assessment of aircrew coordination demand for helicopter flight requirements. In Proceedings of the 6th International Symposium on Aviation Psychology (pp. 308313). Mahwah, NJ: Erlbaum. Bowers, C. A., Morgan, B. B., Jr., Salas, E., & Prince, C. (1993). Assessment of coordination demand for aircrew coordination training. Military Psychology, 5, 95112. Bowers, C. A., Thornton, C., Braun, C. C., & Salas, E. (1998). Automation, task difficulty, and aircrew performance. Military Psychology, 10, 259274. Brannick, M. T., Prince, C., Salas, E., & Stout, R. J. (1997). Development and evaluation of a team training tool . Unpublished manuscript. Brannick, M. T., Salas, E., & Prince, C. (Eds.). (1997). Team performance assessment and measurement: Theory, methods, and applications. Mahwah, NJ: Erlbaum. Byrnes, R. E., & Black, R. (1993). Developing and implementing CRM programs: The Delta experience. In E. L. Wiener, B. G. Kanki, & R. L. Helmreich (Eds.), Cockpit resource management (pp. 421443). San Diego: Academic. Cannon-Bowers, J. A., Prince, C., Salas, E., Owens, J. M., Morgan, B. B., Jr., & Gonos, G. H. (1989). Determining aircrew coordination effectiveness. In Proceedings of the 11th Interservice/Industry Training Systems Conference (pp. 128136). Arlington, VA: National Defense Industrial Association. Cannon-Bowers, J. A., & Salas, E. (1997). Teamwork competencies: The intersection of team member knowledge, skills, and attitudes. In H. F. ONeil (Ed.), Workforce readiness: Competencies and assessment. Mahwah, NJ: Erlbaum. Cannon-Bowers, J. A., & Salas, E. (1998). Individual and team decision making under stress: Theoretical underpinnings. In J. A. Cannon-Bowers & E. Salas (Eds.), Making decisions under stress: Implications for individual and team training (pp. 1738). Washington, DC: American Psychological Association. Cannon-Bowers, J. A., Tannenbaum, S. I., Salas, E., & Converse, S. A. (1991). Toward an integration of training theory and technique. Human Factors, 33, 281292. Cannon-Bowers, J. A., Tannenbaum, S. I., Salas, E., & Volpe, C. E. (1995). Defining team competencies and establishing team training requirements. In R. Guzzo & E. Salas (Eds.), Team effectiveness and decision-making in organizations (pp. 333380). San Francisco: Jossey-Bass. Caro, P. W. (1988). Flight training and simulation. In E. L. Wiener & D. C. Nagel (Eds.), Human factors in aviation (pp. 229261). San Diego: Academic. Chidester, T. R., & Foushee, H. C. (1988). Leader personality and crew effectiveness: Factors influencing performance in full-mission air transport simulation. In Proceedings of the 66th Meeting of the Aerospace Medical Panel on Human Stress Situations in Aerospace Operations (pp. 7-17-9). The Hague, Netherlands: Advisory Group for Aerospace Research and Development. Dwyer, D. J., Fowlkes, J. E., Oser, R. L., Salas, E., & Lane, N. E. (1997). Team performance measurement in distributed environments: The TARGETs methodology. In M. T. Brannick, E. Salas, & C. Prince (Eds.), Team performance assessment and measurement: Theory, methods, and applications (pp. 137153). Mahwah, NJ: Erlbaum. Endsley, M. R. (1995). Measurement of situation awareness in dynamic systems. Human Factors, 37, 6584. Ford, J. K., Kozlowski, S. W. J., Kraiger, K., Salas, E., & Teachout, M. S. (Eds.). (1997). Improving training effectiveness in work organizations. Mahwah, NJ: Erlbaum. Foushee, H. C. (1984). Dyads and triads at 35,000 feet: Factors affecting group process and aircrew performance. American Psychologist, 39, 885893.

CRM TRAINING
Foushee, H. C., & Helmreich, R. L. (1988). Group interaction and flight crew performance. In E. L. Wiener & D. C. Nagel (Eds.), Human factors in aviation (pp. 189227). San Diego: Academic. Fowlkes, J. E., Dwyer, D. J., Oser, R. L., & Salas, E. (1998). Event-based approach to training. International Journal of Aviation Psychology, 8(3), 209221. Fowlkes, J. E., Lane, N. E., Salas, E., Franz, T., & Oser, R. L. (1994). Improving the measurement of team performance: The TARGETs methodology. Military Psychology, 6, 4761. Hartel, C., Smith, K. A., & Prince, C. (1991). Defining aircrew coordination: Searching mishaps for meaning. Paper presented at the 6th International Symposium on Aviation Psychology, Columbus, OH. Helmreich, R. L., & Foushee, H. C. (1993). Why crew resource management? Empirical and theoretical bases of human factors training in aviation. In E. L. Wiener, B. G. Kanki, & R. L. Helmreich (Eds.), Cockpit resource management (pp. 345). San Diego: Academic. Helmreich, R. L., & Wilhelm, J. A. (1991). Outcomes of crew resource management training. International Journal of Aviation Psychology, 1(4), 287300. Jentsch, F. (1997). Meta-cognitive training for junior team members: Solving the copilots catch-22. Unpublished doctoral dissertation, University of Central Florida, Orlando. Jentsch, F., & Bowers, C. A. (1998). Evidence of validity of lowfidelity simulations in studying aircrew coordination. International Journal of Aviation Psychology, 8(3), 243260. Johnston, J. H., Cannon-Bowers, J. A., & Smith-Jentsch, K. A. (1995). Event-based performance measurement system for shipboard command teams. In Proceedings of the 1st International Symposium on Command and Control Research and Technology (pp. 274276). Washington, DC: Department of Defense. Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and development. Englewood Cliffs, NJ: Prentice-Hall. Lauber, J. K. (1984). Resource management in the cockpit. Air Line Pilot, 53, 2023. Leedom, D. K., & Simon, R. (1995). Improving team coordination: A case for behavior based training. Military Psychology, 7, 109122. Marrow, A. J. (1969). The practical theorist: The life and work of Kurt Lewin. New York: Basic Books. McIntyre, R. M., & Salas, E. (1995). Measuring and managing for team performance: Emerging principles from complex environments. In R. Guzzo & E. Salas (Eds.), Team effectiveness and decision making in organizations (pp. 149203). San Francisco: Jossey-Bass. Morgan, B. B., Jr., Salas, E., & Glickman, A. S. (1994). An analysis of team evolution and maturation. Journal of General Psychology, 120, 277291. Orasanu, J. L. (1993). Decision-making in the cockpit. In E. L. Wiener, B. G. Kanki, & R. L. Helmreich (Eds.), Cockpit resource management (pp. 137172). San Diego: Academic. Oser, R. L., Dwyer, D. J., & Folkes, J. E. (1995). Team performance in multi-service distributed interactive simulation exercises: Initial results. In Proceedings of the 17th Interservice/Industry Training Systems and Education Conference (pp. 163171). Arlington, VA: National Defense Industrial Organization. Pettit, M. A., & Dunlap, J. H. (1997). Cockpit leadership and followership skills: Theoretical perspectives and training guidelines (Tech. Report). Washington, DC: Federal Aeronautics Administration. Prince, A., Brannick, M. T., Prince, C., & Salas, E. (1997). The measurement of team process behaviors in the cockpit: Lessons learned. In M. T. Brannick, E. Salas, & C. Prince (Eds.), Team performance assessment and measurement (pp. 289310). Mahwah, NJ: Erlbaum. Prince, C., Oser, R., Salas, E., & Woodruff, W. (1993). Increasing hits and reducing misses in CRM/LOS scenarios: Guidelines for simulator scenario development. International Journal of Aviation Psychology, 3(1), 6982. Prince, C., & Salas, E. (1989). Aircrew performance: Coordination and skill development. In D. E. Daniel, E. Salas,

171
& D. M. Kotick (Eds.), Independent research and independent exploratory development (IR/IED) programs: Annual report for fiscal year 1988 (NTSC Tech. Report 89-009). Orlando, FL: Naval Training Systems Center. Prince, C., & Salas, E. (1993). Training and research for teamwork in the military aircrew. In E. L. Wiener, B. G. Kanki, & R. L. Helmreich (Eds.), Cockpit resource management (pp. 337366). San Diego: Academic. Ryle, G. (1949). The concept of mind. Chicago: University of Chicago Press. Salas, E., Bowers, C. A., & Cannon-Bowers, J. A. (1995). Military team research: Ten years of progress. Military Psychology, 7, 5575. Salas, E., & Cannon-Bowers, J. A. (1997). Methods, tools, and strategies for team training. In M. A. Quiones & A. Ehrenstein (Eds.), Training for a rapidly changing workplace: Applications of psychological research (pp. 249279). Washington, DC: American Psychological Association. Salas, E., & Cannon-Bowers, J. A. (in press). The anatomy of team training. In L. Tobias & D. Fletcher (Eds.), Handbook on research in training. New York: Macmillan. Salas, E., Cannon-Bowers, J. A., & Blickensderfer, E. L. (1997). Enhancing reciprocity between training theory and practice: Principles, guidelines, and specifications. In J. K. Ford, S. W. J. Kozlowski, K. Kraiger, E. Salas, & M. S. Teachout (Eds.), Improving training effectiveness in work organizations (pp. 291322). Mahwah, NJ: Erlbaum. Salas, E., Cannon-Bowers, J. A., & Johnston, J. H. (1997). How can you turn a team of experts into an expert team? Emerging training strategies (pp. 359370). In C. Zsambok & G. Klein (Eds.), Naturalistic decision making. Mahwah, NJ: Erlbaum. Salas, E., Dickinson, T. L., Converse, S. A., & Tannenbaum, S. I. (1992). Toward an understanding of team performance and training. In R. W. Swezey & E. Salas (Eds.), Teams: Their training and performance (pp. 329). Norwood, NJ: Ablex. Salas, E., Fowlkes, J. E., Stout, R. J., Milanovich, D., & Prince, C. (in press). Does CRM training improve teamwork skills in the cockpit?: Two evaluation studies. Human Factors. Salas, E., Prince, C., Baker, D. P., & Shrestha, L. (1995). Situation awareness in team performance: Implications for measurement and training. Human Factors, 37, 123136. Shrestha, L. B., Prince, C., Baker, D. P., & Salas, E. (1995). Understanding situation awareness: Concepts, methods, and training. In W. B. Rouse (Ed.), Human/technology interaction in complex systems (Vol. 7, pp. 4583). Greenwich, CT: JAI Press. Smith-Jentsch, K. A., Salas, E., & Baker, D. (1996). Training team performance-related assertiveness. Personnel Psychology, 49, 909936. Smith-Jentsch, K., Zeisig, R. L., Acton, B., & McPherson, J. (1998). Team dimensional training: A strategy for guided team self-correction. In J. A. Cannon-Bowers & E. Salas (Eds.), Making decisions under stress: Implications for individual and team training (pp. 271297). Washington, DC: American Psychological Association. Stone, R. B., & Babcock, G. L. (1988). Airline pilots perspective. In E. L. Wiener & D. C. Nagel (Eds.), Human factors in aviation (pp. 529560). San Diego: Academic. Stout, R. J., Cannon-Bowers, J. A., & Salas, E. (1996). The role of shared mental models in developing team situational awareness: Implications for training. Training Research Journal, 2, 85116. Stout, R. J., Salas, E., & Fowlkes, J. E. (1997). Enhancing teamwork in complex environments through team training. Group Dynamics: Theory, Research, & Practice, 1, 169182. Stout, R. J., Salas, E., & Kraiger, K. (1997). The role of trainee knowledge structures in aviation team environments. International Journal of Aviation Psychology, 7(3), 235250. Swezey, R., Llaneras, R., Prince, C., & Salas, E. (1991). Instructional strategy for aircrew coordination training. In Proceedings of the 6th International Symposium on Aviation Psychology (pp. 302307). Mahwah, NJ: Erlbaum. Swezey, R. W., & Salas, E. (1992). Guidelines for use in teamtraining development. In R. W. Swezey & E. Salas (Eds.), Teams: Their training and performance (pp. 219245). Norwood, NJ: Ablex.

172
Tannenbaum, S. I., Beard, R. L., & Salas, E. (1992). Team building and its influence on team effectiveness: An examination of conceptual and empirical developments. In K. Kelley (Ed.), Issue, theory, and research in industrial/organizational psychology (pp. 117153). Amsterdam: Elsevier. Thornton, C., Braun, C. C., Bowers, C. A., & Morgan, B. B., Jr. (1992). Automation effects in the cockpit: A low-fidelity investigation. In Proceedings of the Human Factors and Ergonomics Society 36th Annual Meeting (pp. 3034). Santa Monica, CA: Human Factors and Ergonomics Society. Wiener, E. L., Kanki, B. G., & Helmreich, R. L. (Eds.). (1993). Cockpit resource management. San Diego: Academic. Yamamori, H., & Mito, T. (1993). Keeping CRM is keeping the flight safe. In E. L. Wiener, B. G. Kanki, & R. L. Helmreich, Cockpit resource management (pp. 399420). San Diego: Academic.

March 1999 Human Factors

Clint A. Bowers is an associate professor of psychology and director of the Team Performance Laboratory at the University of Central Florida. He received his Ph.D. in psychology from the University of South Florida in 1987. Rene J. Stout is a research psychologist who works in the Aviation Team Training Lab at NAWCTSD. She received her Ph.D. in human factors psychology from the University of Central Florida in 1994. Randall L. Oser is a research psychologist at NAWCTSD. He received his M.S. degree in industrial/ organizational psychology from the University of Central Florida in 1990. Janis A. Cannon-Bowers is a research psychologist at NAWCTSD. She received her Ph.D. in 1988 from the University of South Florida in industrial/ organizational psychology. Date received: November 5, 1997 Date accepted: July 13, 1998

Eduardo Salas is a senior research psychologist and head of the Training Technology Development branch of the Naval Air Warfare Center Training Systems Division (NAWCTSD). He received his Ph.D. in industrial/organizational psychology from Old Dominion University, Norfolk, Virginia in 1984. Carolyn Prince is a former research psychologist at NAWCTSD. She holds a Ph.D. in industrial/organizational psychology from the University of South Florida, Tampa, which she received in 1984.