Toward Formative Measures For Instructors in Outcomes Based Training & Education

Marceau, R., Jackson, C., Dean, C., Diedrich, F., Artis, S., Wiese, E., & Riccio, G. (2010).
Formative Measures for Instructors. In: Riccio, G.,

Diedrich, F., & Cortes, M. (Eds.). An Initiative in Outcomes-Based Training and Education: Implications for an Integrated Approach to
Values-Based Requirements (Chapter 2). Fort Meade, MD: U.S. Army Asymmetric Warfare Group. [Cover art by Wordle.net represents word
frequency in text.]
Development of Instructor Measures 20

Asymmetric Warfare Group
Chapter 2. Formative Measures for Instructors

Ryan Marceau, Cullen Jackson, Courtney Dean, Fred Diedrich, Sharnnia Artis, Emily Wiese
Aptima, Inc.

Gary E. Riccio
The Wexford Group International

2.1 Development of Formative Measures

2.1.1 The COMPASS Methodology

In order to assess training effectiveness and provide developmental feedback, it is important to
assess instructor performance in relation to the knowledge, skills, and attitudes that are being
developed. A bottom-up process for developing performance measures grounds the assessment in
behavior illustrated by experts and individuals who have attained proficiency in a particular job
(MacMillan, Garrity, and Wiese, 2005). This helps ensure that behavior-based measures are
meaningful and pertain to constructs for which data can be collected reliably.

Our methodology for measure development combines subject matter expertise with established
psychometric practices to produce measures of observable behavior. This methodology, originally
developed by Aptima with the Air Force Research Laboratory, is referred to as COmpetency-
based Measures for Performance ASsessment Systems (COMPASS
SM
) (MacMillan, Entin,
Morley, & Bennett, in press). The COMPASS methodology was used to operationalize the
principles and practices of OBTE into observable behavior that can be measured in evaluation of
a training program and that can provide formative feedback to instructors.

The COMPASS methodology employs an iterative series of three workshops with subject-matter
experts to develop and validate observation-based performance measures (for complete details,
see MacMillan et al., in press). In the first workshop, working as a group with a facilitator,
subject-matter experts identify behavior upon which the performance measures are based. These
performance indicators refer to observable behavior that allows an individual to rate the quality of
individual or team performance. For this workshop, the objective is to identify behavior that can
be observed rather than inferred. The performance indicators allow identification, at a high level,
of behavior for which it is most important to develop specific measures. In addition, a workflow
or a series of tasks typically is addressed to provide context for the performance indicators and
associated measures. This context helps identify behavior that is diagnostic or critical to quality
performance.

While a few performance indicators are readily translated into performance measures, more
detailed information is generally needed to create behaviorally anchored performance measures
that coincide with the performance indicators from the first workshop. For the anchors, it is useful
to consider specific behavior that is related to effective and ineffective performance for each
performance indicator. Therefore, the second COMPASS Workshop consists of a series of one-
on-one interviews with subject matter experts to identify explicit behavior that illustrates
superior, average, or poor performance for each of the performance indicators. Conducting this
workshop with individual experts allows for documentation of multiple opinions and, hence,
constant comparison and a more thorough examination of the various assumptions and
considerations for the behaviorally anchored measures.

21 Marceau et al.

The goal of the third workshop is a detailed review and modification of a set of draft performance
measures. The development of an observer-based rating instrument involves concerns regarding
observability (i.e., Will there be an opportunity to observe this behavior?), rating scale (i.e., How
much variability in behavior will be observed?), and wording of the behavioral anchors to reduce
ambiguity and promote inter-rater agreement. In order for performance measures to be
informative, the third workshop also includes a review of relevance and a confirmation that the
performance measures capture the behavior described in the performance indicators derived in
first workshop. Based on the results of the third workshop, the set of performance measures is
further refined, subsequently reviewed, and tested for feasibility of data collection.

2.1.2 Development of Measures for OBTE

The three COMPASS workshops were conducted to develop performance measures for assessing
the extent to which OBTE principles and practices are represented in instructor behavior.
Accordingly, the first COMPASS workshop was conducted in a group setting with the
progenitors of OBTE from the Asymmetric Warfare Group (AWG). These subject matter experts
developed the Combat Applications Training Course (CATC) that the AWG offers to
stakeholders in Army training and education who are interested in OBTE. This train-the-trainer
course presents OBTE in the context of marksmanship training. CATC was used as a point of
departure in the first workshop for most of the discussion about specific behavior and broader
instructional strategies that exemplify OBTE. The performance indicators developed in this
workshop were categories of behavior that reflect OBTE.

The second COMPASS workshop consisted of individual interviews with a selection of the
subject matter experts who were instructors for CATC. The interviews focused on identifying
examples of instructor behavior that would illustrate the novice, average, and expert level of
performance for each performance indicator. The topic of conversation focused on what
instructors should do to be consistent with OBTE. It addressed behavior indicative of effective
performance and behavior indicative of ineffective performance. The information gathered in this
workshop informed the creation of three behavioral anchors to develop a given performance
indicator into a behaviorally anchored measures for instructor performance. Anchors were
developed for most of the performance indicatorsall of the ones for which a rating scale would
be appropriate.

The third and final COMPASS workshop presented the participants from workshop with the final
list of behaviorally anchored measures as they relate to each performance indicator. Performance
measures were reviewed to ensure that the items were seen as meaningful, relevant, and
observable. Subject matter experts reviewed the measures for the performance indicators with
respect to the following criteria: relevance, observability, measurement type (e.g., scale, yes/no,
checkboxes), measure wording, scale type, and scale wording. Consequently, the third workshop
resulted in a complete set of observation-based performance measures for assessing instructors
with respect to OBTE.

2.2 Description of Formative Measures

2.2.1 Results of the COMPASS Process

The product of the COMPASS effort was a set of 65 measures that reflect the principles and
practices of OBTE (see Appendix A for the complete set). Many of the measures also possess
behaviorally anchored rating scales that describe observable instructor behavior at the novice,
average, and expert levels. These measures operationalize OBTE in terms of a formative

assessment, that is, assessments that are actionable in the continuing improvement of training and
education. The COMPASS process yielded a set of measures that have meaning within a
community of practice. Disciplined use of the measures provides the opportunity for the user to
become increasingly oriented to the values and best practices of the community. This provides
additional meaning and validity to the endeavors of anyone who uses the measures for formative
feedback. It thus is an important source of confidence for instructors. Given that the measures
refer to observable behavior, they also tend to ground confidence in a sense of competence; that
is, sources of confidence are verifiable by oneself and others.

These formative measures for instructors are intended to be general. For some instructor
measures, a particular focus such as rifle marksmanship training helped reduced ambiguity. In
such cases, it should be relatively straightforward to translate the particulars into those of another
skill or knowledge domain. We believe the measures can be employed to assess instructor
behavior in any domain to provide feedback with respect to the principles and practices of OBTE.

Many of measures developed for planning and execution of instruction are described below. They
are grouped under high-level headings that were instrumental in the development of the measures.
These headings were intermediate products of the initial COMPASS workshop. They resulted
from initial discussions for particular performance measures that eventually became focused and
refined to a level of detail sufficient for identification of specific behavior that exemplifies
OBTE. Figure 1 shows an example of a behaviorally anchored performance measure (Does the
instructor articulate the why to the Soldiers?) in the context of its nested superordinate
headings (in gray). The twelve highest-level headings are used below to group and describe the
measures. In subsequent development of a grounded theory for OBTE, the measures have been
regrouped into practical categories that facilitate their selection and use with respect to broad
outcomes around which OBTE is organized (see Chapter 3 and Epilogue).

4 Communicate the parameters of learning ("why are we out here" / "what, why, and how")
4.1 Communicate the "right" problem (i.e. what is the real problem they're trying to solve)
4.1.1 Combat or mission applications vs. meeting the minimum standard

1. Does the instructor articulate the why to the Soldiers?

Relies only on tasks,
conditions and
standards; focus is on
completing the event
Explains the why but not
in the context of mission
success/problem solving;
states solution in the
context of the problem
Lays the foundation of
why at the beginning of
training; states the
problem, then guides the
Soldiers to discovery of
the tactical relevance

c N/A Comments/Notes:
c N/O

Figure 1. Sample Performance Measure. N/A refers to not applicable and N/O refers to not
observed.
23 Marceau et al.

2.2.2 Elaboration on the Description of Measures

In the description of measures below, we utilized two techniques that are not part of the
COMPASS methodology but that help in utilizing the products of COMPASS. In particular, the
description of each group of measures includes an instructional vignette that characterizes the
thinking and purposeful behavior of notional instructors in planning and executing a learning
event. The vignettes are based on first-hand observations of instructor behavior in CATC and our
discussions with the instructors. Thus they generally refer to marksmanship training. It is
important to reemphasize, however, that the measures are not limited to marksmanship training.
They can and should be adapted for other skills and knowledge sets (see e.g., Chapters 12 and
13). The notions about the thinking and purpose of instructors were influenced by discussions
with CATC instructors and the progenitors of OBTE (see Chapter 3).

For each group of measures described below, we elaborate on the practical and scientific
implications of OBTE based on our interactions with subject matter experts in OBTE and given
our theoretical dispositions and experience as scientists and educators. The descriptions reflect
our constant comparison between themes from our discussions with experts over many months
and well-established lines of thought in relevant scientific disciplines (Camic, Rhodes, &
Yardley, 2003; Denzin & Lincoln, 2003). The citations make the comparisons explicit; they point
from our representations of OBTE to the relevant literature as opposed to being representations of
the literature. The citations represent interrelated bodies of workmost of which evolved over
many decadesthat can provide anchors and verifiable foundations for a grounded theory of
OBTE. In Chapters 4 and 5, we took greater license in exploring the reciprocal relationships
between our inquiry into OBTE and the relevant literature. Together, the connections described in
Chapters 2 through 5 reveal opportunities for continuing improvement of the measures and the
associated practices of OBTE. This is important given the raison dtre for the formative
measures and the empirical investigations that utilize them (see Chapter 1).

As in our development of a grounded theory for OBTE, identifying implications and personal
meaning in the measures is an important part of how they should be used. To be consistent with
OBTE, particular measures should not be imposed on the user, and no particular measure should
be considered mandatory. OBTE is not prescriptive; the measures and associated vignettes are not
a script. The user should take ownership of the continuous quality improvement that the measures
enable and adapt them as appropriate for the situation at hand.

2.3 OBTE Performance Measures: Planning for Training

2.3.1 Define Outcomes

Defining training outcomes is the first and most critical step in building any program of
instruction. OBTE emphasizes that instructional system design should reflect and respond to the
most urgent needs of Full Spectrum Operations. OBTE focuses on developing Soldiers over time
by exposing them to experiences designed with respect to attributes such as confidence,
accountability, and initiative as well as associated capabilities such as awareness, discipline,
judgment, and deliberate thought. OBTE does not emphasize meeting standards alone; it does not
dismiss them either. Rather, it seeks to shape instruction to promote continuing development of
Soldiers who are agile overall and proficient in particular skills under unpredictable and changing
conditions (Cornell-dEchert, 2009b; see also, Chapter 1).

An outcome is different from a learning objective but ultimately should be integrated with
learning objectives associated with a near-term sequence of learning events (see Cornell-dEchert,

2009a). Outcomes can be viewed as a broader purpose for training or education. They should
relate to execution of military missions and development of individuals to ensure mission success
(Cornell-dEchert, 2009b). As design goals for learning objectives, longer-term outcomes can
provide a framework for linking of learning events in curriculum level design. If a learning
objective specifies the end state of a learning event, outcomes provide guidance and constraints
on the means to the end. An important implication is that, in principle, learning objectives and
developmental outcomes can be convergent or divergent. Without explicit consideration of
development outcomes that are always being influenced, one way or another in a learning event, a
narrow focus on efficiency and near-term objectives can lead to an unacceptable level of risk of
divergence and intertemporal interference between objectives and outcomes (cf., B. Glaser,
2002).

Designers who understand the developmental role they have with cadre will
refrain from narrowly defining every aspect of training or education and instead
focus on principles. The science of the Designers craft must support the art of
the instructors craft; it must be vigilant in never suffocating the art of
instruction. Designers who desire to create a learning environment for students
will first seek to create a learning environment for cadre. Designers who
recognize the importance of an adaptive and thinking Army will first seek to
create an outline of instruction that fosters adaptation and thinking within the
cadre. [M. Darwin, personal communication, November, 2008]

Example Measures

1. Is the training designed to emphasize the importance of combat applications?

Training focuses on
tasks/events; goal is to
pass training (e.g.
qualify)
Combat applications are
described, but training
focuses on tasks/event
Tasks/Events resemble
combat application and
mission success

2. Do the instructors incorporate development of intangible attributes (Judgment,
Adaptability, Accountability, Problem Solving, Confidence, Initiative, Awareness,
Thinking Skills) into their vision for achieving the Commanders intent?

Vision is focused on
apparently efficient
and correct procedural
accomplishment of the
Vision focuses on correct
Soldier performance of
the task but not
development of the
Vision focuses on
effective development
of the individual and
correct Soldier
25 Marceau et al.

event/TSP individual performance of the task

Example Vignette:

An instructors training objective for a certain day is to practice grouping. An expert instructor
might try to schedule a 25-meter range for this day so that the Soldiers can easily walk down
range, see their groups and attempt to self-diagnose before a Drill instructor helps them. An
immediate result is Soldiers who have a better appreciation of factors affecting their shot groups.
More importantly, progress is made toward longer-term developmental outcomes in that Soldiers
come to understand the value of various forms of feedback in learning. A novice instructor, with a
focus on throughput, might plan only to walk down range and then make the changes to the rifle
themselves. The outcome for the novice instructor, whether intended or not, are Soldiers who
believe that practice without feedback is a reasonable approach to learning and who have little or
no awareness that bad habits can be developed as easily as good habits. Expert instructors
understand that there is no such thing as an event in which no learning occurs, they understand
that learning is either good or bad, they understand that time constraints and priorities such as
throughput can lead to instructional shortcuts that have unintended consequences and
counterproductive outcomes. During planning, expert instructors consider these critical influences
on the quality of instruction.

Outcomes and training events should always focus on learning that is relevant to unpredictable
environments rather than the accomplishment of a scheduled event in more or less expedient
instructional settings. The novice instructional behavior in the vignette above might still result in
the Soldiers meeting the grouping standard but very little Soldier development would occur. The
expert instructor allows the Soldiers to view and evaluate the effect they have on their shot
groups. This increases understanding and encourages deliberate thought on behalf of the Soldier.
Though the Soldier may not be able to accurately diagnose the problem at first, guidance from the
instructor will help them to discover the solution to their problem and lead to increased Soldier
confidence and accountability.

2.3.2 Create a Positive Learning Environment

In OBTE, instructors utilize and exemplify leadership. Arguably, the terms instructor and
leader are synonymous in this approach to learning and development. Every instructor is a
facilitator, advisor, and mentor. This means that some mistakes will be allowed, that students will
learn lessons from mistakes instead of being derogated for them, and the instructors role will be
to guide students toward discovering a solution. Time should be built into the schedule for such
guidance and for students to engage in the self-discovery and collaborative reflection that are
especially important when learning events are not completely scripted and controlled.

Instructors should plan to frequently assess how the training is progressing. This assessment does
not have to take any time away from training; it can be as simple as lagging behind the Soldiers
as everyone walks up range. During this short time, instructors can assess if they are achieving
their intent for the day and make changes as necessary. This way, the instructor can adapt the
training to the evolving needs of the students without sacrificing progress toward long-term
developmental objectives.


Example Measures

1. Do the instructors plan to scale down from an authoritarian approach to a mentoring
leadership style as appropriate for training events?
! Yes ! No

2. Do the Drills design the training events to gradually increase in difficulty?

Unaware of stress
management; Plans to
increase difficulty of
training according to a
set schedule/process

Plans to introduce stress,
but not at the appropriate
level or time

Balances stress and
difficulty to the capability
of the Soldiers

Example Vignette:

Drill instructors are planning out how they will have the Soldiers practice shooting from different
positions. Due to the high Soldier to instructor ratio, there will not be enough instructors to ensure
that all the Soldiers are building proper positions. A novice instructor would likely plan to first
demonstrate the positions to all the Soldiers and then spread instructors evenly across the lanes
and run all the Soldiers through in cycles. The instructors would monitor multiple lanes to ensure
safety and provide assistance when necessary. An expert, in this case, might plan to first split the
Soldiers into small groups, one for each instructor. The instructors will facilitate discussions
among their groups on the how to properly build a stable position, guiding the Soldier towards an
understanding of the fundamentals of a stable position.

The experts decision to break the Soldiers into groups will allow instructors to facilitate the
individual development of the Soldiers. By ensuring that each Soldier becomes acquainted with
the fundamentals of how postural configuration and stability relate to marksmanship, they are
increasing the likelihood that Soldier will be successful in shooting not only from the particular
positions experienced but from other positions as well. As a student of combat-relevant
marksmanship, the Soldier will develop perceptual and motor skills of stabilization in addition to
appreciating the implications of stability (cf., Riccio & Stoffregen, 1988; Riccio, 1993). This will
increase a Soldiers confidence in situations where factors of enemy or terrain, for example,
require nonoptimal or unusual postures. Though the initial small group exercises may delay the
start of the shooting portion of the training, the increased understanding and proficiency will
increase the rate of learning in novel or more difficult situations.

The balance between task difficulty and student capability is critical to student development.
Increasing difficulty or applying stress too quickly will lead to failure and be detrimental to the
students confidence. Going too slowly will result in inefficient training and students becoming
disengaged. A proper balance will result in efficient and effective training that also builds
27 Marceau et al.

students confidence as they learn to learn and perform under nonoptimal conditions. Instructors
should employ a building block approach that increases complexity, for example, as students
learn and adapt new skills (see, e.g., Merrienboer & Kirshner, 2007; Tobias & Duffy, 2009). Such
scaffolding can be planned in terms of starting simpler as well as ending more complex.

2.3.3 Create the Parameters of Learning

It cannot be assumed that the students will understand why a training exercise is relevant to their
mission. It is important to design exercises that aid the student in making these connections.
Understanding the why behind training will contextualize students learning and presumably
will lead to better transfer to the operational environment. In OBTE, instructors plan for
opportunities to provide explanations or otherwise to reveal the broader context of a learning
event.

While large blocks of unused time (e.g., hours or days) are generally nonexistent in a training
program, there is a multitude of brief moments (e.g., seconds or minutes) that are wasted in
almost any instructional situation. Much can be communicated during the frequent interstitial
moments between telling and doing. Value can be added in both the content of communication
(e.g., explanation) and the occurrence of communication (e.g., modeling). Instructors should
prepare for such opportunities by having teaching points that can be expressed concisely at
moments that, while not precisely predictable, are reasonably likely to occur. There are
opportunities for education during training.

The inextricable linkage between training and education, in practice, calls for different
approaches in design and development. While training produces a change in behavior, education
produces a change in thinking. Training Soldiers to be agile requires consideration of cognitive
skills even when the focus of a learning event is on fundamental perceptual and motor skills. In
fact, cognition is separable from perception and action only under the most artificial and
contrived situations (cf., Neisser, 1976; Shaw & Bransford, 1977; Winograd, Fivush, & Hirst,
1999). The linkages can be so basic that little or no explanation may be necessary for students to
come to a better understanding of them. The right kind of experience may be sufficient.

Example Measures

1. Do the instructors plan to discuss the tactical relevance of the task with the Soldiers?
! Yes ! No

2. [Follow-up] As part of the discussion, will the instructors ask the Soldiers to describe the
tactical relevance of the events?

Only plans to lecture to
Soldiers on the task; no
context
Plans to engage the Soldiers
on why the event is
tactically relevant, but
Plans to state the problem,
then guide the Soldiers to
discovery of the tactical
relevance (i.e. problem
solving exercise)



Example Vignette:

Soldiers in Army basic training sometimes use sandbags to support their rifles in the prone-
supported position. However, it is unlikely that there will be sandbags laid out for them in theater.
The skill that is being taught is how to establish a solid, supported position, not how to shoot
using a sandbag. Expert instructors would plan to ask the students to think about or discuss ways
they could create a stable prone-supported position in theater. What would they use for support?
How would this change their position? Why is this important in combat? This discussion could
happen during the down time in between shooting cycles while Soldiers are rotating or waiting
their turn to shoot.

Creating the parameters of learning helps explain the why the experience and skills acquired in a
learning event relate to a mission application of a current task. To the extent possible, the learning
environment should have operational relevance and realism. Situation-specific resource and
safety constraints often make it difficult for the training environment to look like the field or
operational environment. The tendency in Army training and education is to think that, absent
obvious realism, operational relevance can be addressed through vignettes or other supplementary
materials that tell the student about a mission and set a conceptual context for instruction. To be
sure, such a conceptual context can be useful to the student but, by itself, it has many of the
limitations of telling without doing. The connections are not likely to be understood deeply or
retained by the student.

A learning environment need not appear superficially like an operational environment to have
operational relevance or, more specifically, to develop skills that transfer to an operational
environment (cf., Warren & Riccio, 1985). A learning environment can have fidelity with respect
to perception and action if it replicates constraints on what an individual can achieve and how.
Generally this requires a context that is sufficiently rich to include observable cause-effect
relationships such as linkage among tasks and concurrent or downstream consequences of ones
actions (cf., J. Gibson, 1977, 1979). In the context of marksmanship, for example, the relationship
between constraints imposed by ones momentary biomechanics, the environment, and
perceptual-motor tasks can be manipulated and revealed in operationally relevant ways without
the physical realities of combat. Training can be designed to give individuals experience,
sometimes in unusual settings, that sensitizes them to the interrelationships between postural
control, stabilizing and destabilizing elements of the environment, and visibility in general (cf.,
Riccio, McDonald, & Bloomberg, 1999). The broader implication is that basic research in the
behavioral sciences can inform the design of a training environment (cf., Gagne & Gibson, 1945;
Riccio, 1995; Riccio & McDonald, 1998).

2.4 OBTE Performance Indicators: Training Execution

2.4.1 Communicate the Parameters of Learning

As indicated in section 2.3.3, OBTE instruction should seek to lay the foundation of the why at
the beginning of training. Students should be exposed to the problem and guided to discovering
the tactical relevance of a particular training activity. OBTE instruction emphasizes how a
particular activity applies to the overall mission or desired outcome.

29 Marceau et al.

Example Measures

1. Does the instructor articulate the why to the Soldiers?

Relies only on tasks
conditions and
standards; focus is on
completing the event
Explains the why but not
in the context of mission
success/problem solving;
Lays the foundation of
why at the beginning of
training; states the
problem, then guides the
Soldiers to discovery of
the tactical relevance

Example Vignette:

Soldiers typically arrive at the range early in the morning following chow. They get into
formation and wait for instruction from the Drill instructor. Often, the Soldiers do not know or
have very little information regarding training content for that day until they are told by a Drill
instructor or the tower to complete a task. Even at this point, they still do not know the reason or
combat relevance of the task, just that they have to complete it. An expert in OBTE would start
the day by bringing everyone in and discussing the goals for that day. The expert does not simply
recite passages from the Training Support Package for the day but presents the goals in the
context of a combat related problem that the Soldiers must overcome in order to achieve mission
success. The instructor takes opportunities to reveal, not simply to lecture about, the critical
interrelationships among barricades, defilade postures, muzzle awareness, location of other unit
members, and characteristics of enemies and noncombatants in terms of the capabilities and
requirements for postural stability and visibility of targets in a task that otherwise is ostensibly
only about marksmanship.

The vignette above describes one way in which an instructor can increase learning by providing
context around a training task. The impact of training presumably is increased to the extent that
Soldiers are sensitized to contextual relationships not limited to the task at hand (cf., J. Gibson &
E. Gibson, 1955; Bransford & Schwartz, 1999). In this case, the context is not just the learning
objectives for the event, but also their relationship to lethality and survivability in combat and
perhaps even to strategic implications of small unit action in the context of FSO. By discussing
why particular activities and props are included in a learning event, the instructor can educate the
attention (cf., J. Gibson, 1979) of the student and help reduce workload due to an unnecessarily
confusing or inexplicably cluttered setting. Through this deeper understanding, Soldiers become
better equipped to apply the same concepts towards solving problems in different situations they
may encounter in the operational environment.

2.4.2 Training Emphasizes Broad Combat or Mission Success

Training should emphasize problem solving irrespective of environment or resources. Many
training environments do not accurately reflect all aspects of the operational environment. This
need not be the case for operational relevance, and it may be desirable for learning (see section
2.3.3.2). In these situations it is important that the instructor ensure the students are making

connections in the training task that are similar to the connections on which mission success
depends in an operational environment.

In the case of OBTE, effectiveness wont truly be known until the Soldier performs in combat,
but evidence of learning and development of efficacy (the foundation or capacity for
effectiveness) will manifest itself in ways that allow meaningful assessment of both the Soldier in
training and the trainer in execution. It also provides for explicit or implicit self-assessment and
the development of a sense of self-efficacy (cf., Bandura, 1977, 1997; see also, Ryan & Deci,
1985) in both trainers and their Soldiers (I know when I am performing well and I know that I
can perform well). This is a sense of self-efficacy that is grounded in demonstrable competence
and an appreciation of what ones actions and capabilities for action in a particular environment
afford for good or ill (cf., J. Gibson, 1977; Shaw, Turvey, & Mace, 1982; Turvey, 1992; see also,
Endsley, 1995; Neisser & Jopling, 1997).

Example Measures

1. Does the training emphasize broad combat/mission success?

Train to a specific task
only
Train to examples or
experience only; reflects
specific environments or
resources only
Creates training that
emphasizes problem
solving irrespective of
environment or resources

Example Vignette:

A unit who knew they would be deploying to an urban area in Afghanistan trained extensively in
simulated urban environments. They practiced shooting around buildings, cars and other urban
terrain. They were proficient in room clearing tactics and close quarter marksmanship. Upon their
arrival they were sent to an area of operation that consisted of partial urban and wooded terrain.
The Soldiers did not feel confident in their ability to operate in the wooded terrain because they
had trained in urban terrain only. Consequently, they would be hesitant to follow the enemy into
the woods. Subsequently, in the role of an instructor in a simulated urban environment, one of the
Soldiers made sure that his trainees understood general principles of individual movement,
collective maneuvering, and situation awareness in the context of general characteristics of the
environment such as paths, obstacles, barriers, margins, brinks, footing, partial enclosures,
camouflage, concealment, vistas, and vantage points. Some of this was accomplished by exposing
Soldiers to variation in these constraints amid invariant tasks involving individual perception and
action as well as collective coordination. Soldiers also were asked to consider contingencies in
the way various characteristics of the environment could be utilized.

What the Soldiers described in the initial part of the vignette did not realize was that all the
fundamentals upon which the urban tactics were built could be applied to the wooded terrain. In
contrast, being exposed to variation in the environment and in behavior with respect to it can help
them avoid being distracted by the concrete facts of buildings and roads. They can differentiate
31 Marceau et al.

the essential from the incidental; they can see various aspects of environment in a different way
(cf., J. Gibson & E. Gibson, 1955; E. Gibson, 1991). The implication is that learners can free
themselves from old habits and assumptions carried from different occupations and purposes.
They can come to understand the essential interdependence of task and the way the environment
is perceived (cf., J. Gibson, 1979; Riccio & Stoffregen, 1988). They can become better prepared
for a new occupation and a new purpose in an otherwise familiar environment. Thus they become
more adaptable to novel tasks and conditions. The instructor can facilitate this learning through
judicious and well-timed guidance (cf., Bransford & Schwartz, 1999; Tobias & Duffy, 2009).
This is not to imply that the Soldier understands these things in an abstract way. The implication
is that he simply knows what to do and is not frozen in his tracks by superficial novelty.

2.4.3 Customize Instruction When Possible Based on Constraints/Conditions

OBTE is learner-centered (cf., Bransford, Brown & Cocking, 2000; see Chapter 3). Instructors
should try to balance training difficulty with the students momentary capability to learn. It is
important to recognize the point of diminishing return (i.e. student is exhausted or burnt out) and
adapt the training accordingly. People learn differently and at varying rates. Instructors should
coach individual students or address the individual training predicaments of students who may
have unique needs. This should not be taken to imply that instructors must provide every student
with individual attention all the time or even provide this to most students most of the time.
Sometimes a little individual attention is all a student needs to get back in the game. Other times,
there might be an individual difference that is unnecessarily hindering a students ability to learn
(e.g. a small Soldier with a large rifle). Even if the instructor cannot always provide guidance to
all the students that could benefit from it, it is better to help some students that none of them.
Moreover, the visible act of providing guidance to a student provides a model even for
bystanders. Instructors can be influential as role models for students who are not the object of the
instructors immediate attention (cf., Lave & Wenger, 1991).

In OBTE, it is desirable to provide students with the right amount guidance at the right time. An
implication of this principle is that it is important to know when not to intervene or to instruct
explicitly. Encouraging students to take ownership of their own learning can have valuable effects
on student motivation and engagement in learning (cf., Guay, Ratelle, & Chanal, 2008).
Instructors are not passive during these opportunities for student initiative. They are vigilant in
monitoring the situation and ready to show signs of approval when students demonstrate
discipline and awareness in exercising such initiative. Timely approval shows that instructor was
willing to trust the student and not merely inattentive or uncaring. It reinforces accountability.

Example Measures

1. Does the instructor adapt the training to the audience/environment?

Sticks to set schedule;
unaware of diminished
learning (e.g. ignores
indicators of
exhaustion)
Reactively balances
training difficulty to the
capability of the Soldiers
(e.g. reacts after
performance has
diminished significantly
Proactively balances
training difficulty to the
capability of the Soldiers;
recognizes point of
diminishing return


2. Do the instructors successfully address individual training predicaments?

Doesnt recognize
individual problem
Recognizes individual
problem, doesnt help the
Soldier find a suitable
solution
Identifies the issue and
helps the Soldier find a
work around (e.g. shorter
weapon for smaller
person)

Example Vignette:

A Soldier is struggling during a grouping exercise; the Soldiers groups continue to stay large
despite direction from the drill instructor on proper breathing, positioning, and trigger squeezing.
A novice instructor may grow frustrated in this situation and insists that the Soldier is not
listening to instructions. An expert, whose focus is on Soldier development, would look deeper to
examine if something else is going on. Upon further inspection the instructor noticed that this
Soldiers earplugs do not fit correctly. The Soldier is wearing them correctly, but they are not
sealing the ear properly causing to Soldier to flinch every time a shot goes off. The instructor
finds a different style of earplug for the Soldier and the groups start to become remarkably
tighter.

In this situation, an OBTE instructor recognized a Soldiers individual difference that was
preventing him from being successful. The instructor was able to then identify a solution to the
students problem (changed earplugs). Doing so then allowed the student to master the
fundamentals and become more confident in his or her ability as a rifleman. Had this gone
unnoticed, that student might think that he simply was not skilled enough to shoot a solid shot
group, leading to low confidence as a rifleman. Additionally, the instructor acted as a mentor in
this case by demonstrating effective problem solving skills in addressing the Soldiers individual
training predicament.

2.4.4 Facilitates Learning of Concepts

OBTE promises to develop deep understanding and habits of learning that help Soldiers
recognize and exploit similarities between their prior experiences and novel situations. This
competency generally will result in the perception of what a unique situation affords for action
and in the associated readiness for expeditious action that is efficacious and that reveals
additional information that reduces ambiguity (cf., E. Gibson, 1988; J. Gibson, 1979). Such
development of perception and action enables Soldiers and their units to be agile and effective in
Full Spectrum Operations.

Instructors structure the problem and coach the student towards the solution. Not only does this
approach to training increase learning by leading the student to discover the solution, it also
encourages students to take ownership of their own learning. From the perspective of a mentor
and facilitator, the focus of the instructors attention when students exercise initiative is on
whether the students response is adaptive in the sense of being task directed and oriented (i.e.,
33 Marceau et al.

efficacious in principle), not whether it is momentarily effective. Such efficacy is the difference
between adaptability and mere flexibility. Flexibility is better than inflexibility but not much.
Efficacy in general eventually leads to effectiveness in particular situations but not in every
instance. Mistakes will be made. The role of instructors is to ensure that the consequences of
mistakes are bounded (Cornell-dEchert, 2009b). It is to identify a students zone of proximal
development (see Cole, John-Steiner, Scribner, & Souberman, 1978), the margin between their
momentary range of efficacy and their potential range of efficacy. This is the zone within which
students are challenged without being frustrated and within which there is the greatest capacity
for semi-autonomous learning without deleterious consequences.

Example Measures

1. Does the instructor guide Soldiers to self-discovery of how to achieve desired results?

Dictates a specific path
to the solution to a task
Structures the problem; but
still dictates a specific path
to the solution
Structures the problem
and guides Soldier to
efficient self discovery of
solution path

2. Does the instructor use safety as a training enabler?

Focuses on SOP and is
regimented; safety is
disconnected from its
real purpose
Explains safety in the
context of accomplishing
the training events, but not
as a combat and training
enabler (i.e. safety is
restrictive)

Explains safety as a
combat and training
enabler (e.g. weapons
awareness allows for more
independent or complex
scenarios)

Example Vignette:

Marksmanship training is heavily structured and regimented in order to maintain safety.
However, Soldiers do not learning to be safe on their own because the tower is always telling
them exactly what to do. They appear to be afraid of their weapons. A different approach would
be to start off regimented and pull back after Soldiers demonstrate safe behavior. Without
constant direction from the tower, Soldiers can eventually take on the responsibility for being safe
and become more confident in their ability to be safe with a weapon.


Transferring the responsibility for safety to the student increases the students awareness and
accountability. The instructor demonstrates trust in the Soldiers ability to be safe.
Soldiers also become more confident in their ability to be safe with their weapons in future
situations when a drill instructor is not present. In Army training, safety and instructional
methodology (i.e., nature, extent, and timing of guidance provided to students) typically are
viewed as the responsibilities of different groups of people if not wholly incommensurate
considerations. The juxtaposition of instructional guidance and safety in OBTE is unusual. As
with most practices in OBTE, this juxtaposition is due to exigencies and realities of the
operational environment, such as the critical interrelationships between lethality and survivability,
especially amid the ambiguities of Full Spectrum Operations. An assumption of OBTE is that,
instead of finding ways to integrate such fundamental capabilities, they should never be separated
in the first place. Accordingly, there should be a persistent coupling between initiative and
accountability from the beginning and at every stage of a Soldiers training, education, and self-
development.

This does mean expanding our own circles of interaction to include, for example,
the range, ammo, and training developers of the Army who rarely see a Soldier
train, but do influence decisions on how, why, and to what level his training will
be supported by the Army Outcomes are in so many ways influenced by the
availability of inputs - resources, including trainers and leaders - and we should
start capturing the inputs required for training more effectively. [K. McEnery,
personal communication, February, 2009]

2.4.5 Creates a positive learning environment

A positive learning environment is foundational in OBTE. A balance of authoritarian and
collaborative styles of instruction is an important part of creating such an environment. One
manifestation of this balance is to avoid being directive about content (what to think) while
being somewhat directive about process (how to think) (HQDA, 2003; cf., Freeman et al.,
2008; Roberto, 2005, 2009). Instructors understand and embrace their role as mentors. They
serve as guides along the Soldiers path of success by providing clear left and right limits, by
noting milestones and decision points, but also by allowing some exploration (cf., Cole, John-
Steiner, Scribner, & Souberman, 1978; Feldbaum, 1965; Filatov & Unbehauen, 2004, E. Gibson,
1988; Henle, 1971; Thelen & Smith, 1994; Safonov, 2001; Tschacher & Dauwalder, 2003). They
facilitate the developmental process by which Soldiers take ownership of their own pursuit of
success. One strategy in this role is for instructors to provide opportunities for Soldiers to think
critically and be problem solvers through training events that require assessment, judgment,
decision-making and execution. As Soldiers try to solve the problem at hand, instructors guide
them through directive questioning and discussion. In the end, the Soldiers solve the problem.

35 Marceau et al.

Example Measures

1. Does the instructor reinforce the importance of problem solving?

Event is scripted;
training is based on
correct performance of
predetermined task
Training provides
opportunities for Soldiers to
be problem solvers but
instructors give the solution
Training provides
opportunities for Soldiers
to be problem solvers;
events require assessment,
judgment, decision
making and execution

2. Does the instructor ask questions and allow/encourage the Soldiers to answer?

Provides answers for
the Soldiers
immediately after
question is asked
Encourages Soldiers to
answer questions, but
provides an answer if one is
not immediately offered
Allows time for Soldiers
to think of an answer and
guides them to appropriate
responses

Example Vignette:

Instructors, during ARM, have set up a range with different stations every five lanes. Soldiers
start at one end and make their way to the other end in pairs, performing different tasks and
exercises at each lane. These exercises include target discrimination, malfunction drills and other
complex tasks. The goal for the day is to develop Soldiers to perform under high stress, combat
like situations. One lane includes a wounded Soldier drill in which the pair of Soldiers has to
react to an injured Soldier, get him out of the line of fire and treat the injury accordingly. Upon
reaching this scenario, two Soldiers are struggling to lift a very large Soldier on stretcher. They
try twice to get him off the ground, but he is simply too heavy. These Soldiers, recognizing that
they have to get the wounded Soldier out of the line of fire, decide to each lift one end of the
stretcher and drag the Soldier out of harms way. Upon seeing this, the drill instructor makes sure
that the Soldiers see that he has noticed their solution while he lets them carry on. Later he
questions the Soldiers with interest about their decision to deviate from standard procedure.

The vignette describes a situation in which the instructor has a choice to intervene or not. He
might have, for example, yelled at the Soldiers, criticizes them for not carrying stretcher the
correct way, and make them start over and do it again. This would have failed to maintain a
positive learning environment. Moreover, overly focused on completing the task correctly
would have punished the Soldiers for not following the prescribed methods for carrying a
wounded Soldier even though what they did was more combat effective given the situation.
Instead, by letting the Soldiers exercise initiative, the instructor allowed them experience
problem-solving under stress. By engaging in collaborative reflection with them after the event,

the instructor found out that their goal was to get the wounded Soldier out of harms way. He thus
could acknowledge that they found the most effective way to accomplish that goal given the
constraints of the situation. He would be able to recognize the deliberate thought and sound
judgment of the Soldiers and positively reinforce their behavior.

It is important to understand that, in OBTE, the deliberate thought and judgment required in
mission-relevant problem solving does not necessarily imply cognition that is mentally effortful
or temporally protracted (e.g., on the order of tens of minutes to hours) as we might imagine
based on years of experience with taking tests in learning environments. Deliberate thought is as
much about disciplined awareness, attunement to things that matter in ones surroundings and in
the flow of events, appreciating the consequence of ones behavior for others and vice versa, and
about the attendant purposeful coupling of perception and action. Such awareness can unfold and
inform decisions on time scales as brief as seconds to minutes (cf., Salas & Klein, 2001).

2.4.6 Instructors Utilize Measures of Effectiveness & Self-Evaluation

OBTE is assessment-centered in the sense that assessment is a component of learning not merely
an assessment of learning (cf., Bransford et al., 2000; see Chapter 3). It is important that
instructors are constantly reflecting on the progression of learning. Instructors ensure that
students are getting the training that they need and decide when to advance to the next event or to
revisit a learning objective. OBTE is focused on the student instead of the schedule. Schedules
should not dictate how the training progresses; the students momentary capabilities and potential
should. Instructors should make in-stride adjustments to training as necessary. The informational
basis for these decisions should be explicit to students whenever possible and appropriate. The
point is not to justify changes to the students but to reveal that training is principled, that changes
are event-driven, and that there should be constant vigilance about the conditions in light of
assumptions that may be violated and thus necessitate a change. If this is occasionally made
explicit to students, it will become implicit to them in future decisions of their instructors and
leaders. Student will become more likely to look for conditions that may have motivated a
change; they will become more aware of the situation (cf., Endsley, 1995).

Example Measures

1. Are the instructors discussing the effectiveness of the training?

Instructors
automatically advance
different groups
without considering
training progress
Instructors check on
Soldiers progress but do
not effectively adjust the
training
Check on Soldiers
progress and discusses
when to advance to next
event or when to revisit a
learning objective

37 Marceau et al.


2. Do the instructors use unexpected conditions to provide tactically relevant training
events?

Allow the unexpected
conditions to interfere
with training
Make in-stride adjustments
to continue training but do
not leverage unexpected
conditions as a training
opportunity
Make in-stride
adjustments to incorporate
unexpected conditions
into the training design if
appropriate

Example Vignette:

A schedule calls for several complex ARM exercises to occur on a particular day. However, it is
over 100 degrees out today and the Soldiers are getting burnt out fast. Instructors notice that body
positions and movements of the Soldiers are becoming progressively worse, fundamentally, as the
day goes on. Concentration wanes and shooting accuracy deteriorates. The instructors decide that
the students are reaching the point of diminishing returns. Thus, they make an in-stride
adjustment with the Soldiers by rolling back the complexity of the tasks to ensure the Soldiers do
not fail.

The vignette describes a situation in which the instructors put the emphasis on learning and
development and not on task completion or following the schedule. Though Soldiers will have to
perform under high stress situations in combat, they first need to adequately master the basics
before they can be expected to perform well under more demanding conditions. Had the
instructor decided to continue the training at the higher complexity level, the Soldiers would have
pressed on, but their confidence would have declined as they continued to get poor results and
eventually failed (cf., Merrienboer & Kirshner, 2007). By adjusting the training complexity to the
current ability of the Soldiers to perform, the instructor ensured that the Soldiers confidence in
their ability to perform would remain high. The adjustment would not have been considered if the
instructors were not vigilant about the state and progress of the students.

In OBTE, instructors do not depend unduly on explicit communication with Soldiers to assess
their physical, physiological, cognitive, or emotional states. It can be a useful tool, however, and
one with additional benefits. By occasionally checking with students about their self-perception
and their perception of the situation, instructors reveal to students the importance of situation
assessment in principled approach to training and, more generally, in a principled approach to
decision making in the field (cf., Endsley, 1995). Instructional situations can be dominated by
planned events or facts and still allow students to have opportunities to consider possibilities and
alternatives when assumptions or expectations are violated. These opportunities can be brief or
small but if they are common, they encourage a certain vigilance and anticipation that
automatically engage students more deeply in the situation. Seizing such opportunity doesnt
require instructors or developers to throw out the lesson plan.


2.4.7 Uses scenarios to facilitate learning

Scenarios are potentially powerful tools for emphasizing the reasons for a learning event, why the
event is structured the way it is, and what are the essential versus the incidental aspects of the
event with respect to the operational environment. Our use of the term scenario thus is broader
than its connotation in common military usage. In our usage, a scenario is a situation that is
sufficiently rich to reveal the consequences of ones actions beyond the task at hand. If a training
scenario focuses on superficial similarity to its operational analog, it may not be effective and
may even distract students from the key lessons to be learned. There are several contextual
variables that can be manipulated effectively in a scenario-based learning event and that are not
especially dependent on superficial similarity to an operational environment. They include but are
not limited to stressors, linking of basic Soldier tasks, and linking of individual activities with
collective activities (cf., section 2.4.2 above).

Stress can be an excellent facilitator of training through scenarios. Instructors should incorporate
stress (mental and physical) into training in ways that challenge the student but are proportional
to the student capabilities. Stress should not cause the student to fail, but should be challenging.
The goal of stress application is to give the student a sense of accomplishment and confidence in
their ability to perform under adverse conditions. Students should be allowed to discover how
stress affects their performance, and how to mitigate it. Stress can be manipulated easily through
time constraints or by task linkage.

Example Measures

1. Do the instructors group tasks into collective behaviors?

Does not group tasks
into collective
behaviors; results in
incorrect performance
of linked tasks
Does not group tasks into
collective behaviors (no
observable negative
consequences)
Groups task in a way that
simulates the combat
application and reinforces
correct performance of
linked tasks

2. Do the instructors effectively incorporate stress (mental and physical) into training events
to benefit the development of the Soldier?

No stress resulting in
apathy or too much
stress resulting in
chronic failure
Some stress resulting in
some learning; Soldier
unchallenged or overly
challenged
Stress is proportional to
the task and Soldier
capabilities resulting in a
sense of accomplishment

39 Marceau et al.


Example Vignette:

During a training exercise, Soldiers are practicing their fundamentals while shooting from the
prone position. The instructors see that the Soldiers are all doing well so they decide to add stress
by limiting the amount of time the Soldiers have to shoot their ten rounds to one minute. At first
the Soldiers all shoot off their rounds in the first 20 seconds and their shot groups are poor. The
instructors point this out and ask the Soldiers to think about how the time stress affected their
performance. Next cycle, the Soldiers take more time and their shot groups improve. After a
couple cycles the instructors then tell the Soldiers they now have to take ten shots from the prone,
reload their rifle while communicating that they are reloading, and then switch to the kneeling
position and take ten more shots in two minutes total. After a few cycles of this exercise, the
instructors facilitate a discussion with the Soldiers about managing stress in combat.

In the vignette, several simple tasks are gradually combined to create more complex, linked tasks.
Stress is added by incorporating a time constraint. This gradual ramping of stress keeps the
Soldier out of their comfort zone, but not to the point of failure. At the end of the day, the
Soldiers in the vignette will be performing complex, linked tasks that they would have never
thought they were capable of doing at the beginning of the day. More novice instructors may
incorporate the wrong kinds of stress (e.g., verbal intimidation) or may apply too much of one
type of stressor. Consider, for example, an apparently reasonable stressor of having Soldiers run a
mile in full kit or do several strenuous activities prior to participating in a training exercise.
Though this is a kind of stress that is relevant to the operational environment it is applied in a way
that does not allow the Soldier to overcome the effects of stress gradually. Through the gradual
addition of stress and complexity, the Soldiers continuously build up their confidence and become
ready to tackle more complex tasks.

The vignette also shows how the linking of tasks can be relatively simple and straightforward yet
give Soldiers experience with critical interrelationships among basic Soldier skills such as move,
shoot, and communicate. Supplemented with a bit of a backstory provided by the instructor, even
tasks that apparently are purely procedural can be learned in ways that vastly improve a Soldiers
readiness for combat. Consider SPORTS, a procedure for correcting a weapons malfunction.
The mnemonic refers to a process to clear a rifle malfunction by slapping upward on the
magazine, pulling the charging handle back, observing the obstruction, releasing charging handle,
tapping forward assist; and squeezing the trigger. The context in which this procedure would be
performed is not incidental and could potentially have profound implications.

On the range, a weapons malfunction is a benign event. Obviously in a firefight, the situation is
quite different. The speed with which a Soldier can get the weapon back into the fight is a
potentially a life or death matter, and inability to do is not simply a matter of a no-go. Returning
fire is the higher-order task, one that could be accomplished to some degree by others in the unit.
Thus it becomes critically important to communicate to others that the weapons malfunction has
occurred. This could change the priorities of another Soldier or simply just change the direction
of fire or his firing position. At the same time, without an operable weapon, the requirements and
tradeoffs for seeking cover change. The tradeoffs are peculiar to the momentary situation and the
relevant factors of mission, enemy, terrain and weather, troops and support available, time
available, civil considerations. The linkage between move, shoot, communicate, and assessment
of the situation could not be more fundamental and the consequences could not be more
immediate.


In a firefight, an outward orientation and the attendant situation awareness is critical but it is
complicated by the distractions of the unusual, if not unique, psychological and physiological
responses to a lethal threat (cf., Kolditz, 2007; Lerner & Keltner, 2001; Lerner & Tiedens, 2006;
Lukey & Tepe, 2008). Habits become important at such times. It is not the time to perseverate on
a mnemonic and its associated procedure if it isnt working. Good habits are necessary. Habits
require frequent repetition. It is never too early to give Soldiers the opportunities to develop good
habits of linking move, shoot, communicate, and situation assessment, and to give them these
opportunities often. A premise of OBTE is that such basic combat skills can be acquired outside
of combat, that they can be acquired in training. These opportunities will be missed if it is
assumed that there are procedures Soldiers can learn in the absence of deliberate thought and
broader awareness.

2.4.8 Instructors exhibit intangible attributes in own actions

OBTE doesnt require that instructors interact with or even be aware of every student all the time.
The instructor is a role model (cf., Bandura, 1977). Instructors should demonstrate behavior and
attitudes their students can model because, more likely than not they will model the instructors
behavior (cf., Lave & Wenger, 1991). It thus is important for the instructor to be aware that his
behavior may be noticed and influential whether or not the behavior was planned or intentional.
One should be aware, for example, that an authoritarian or intimidating style is not necessarily
perceived as flowing from competence. Similarly, evidence that the instructor does not know it
all, or as much as a student knows in some areas, is not necessarily perceived as incompetence.
The expertise with which the instructor should most be concerned is as a leader, mentor, and
facilitator. Their impact is similarly profound whether they accept this responsibility or not, and
whether they are prepared for it or not. The assumption of OBTE is not that instructors should
have a greater impact; it is that they can have a better impact.

Even when there are gaps in an instructors knowledge or experience, there is an opportunity to
demonstrate expertise in learning. The instructor can help a student learn to learn. In the end, this
may be the most important influence an instructor has on a student given that adaptability of
Soldiers in the operational environment implies a capacity to learn. The point here is that students
are always learning to learn in addition to learning what they should learn. We can choose to
teach students to learn one way or another but we cannot pretend that there is not a choice. In
OBTE, initiative and accountability are underpinnings of confidence. Confidence is grounded in
demonstrable competence, and competence is ensured by the capacity to learn. The capacity to
learn is increased by initiative and taking ownership of ones own learning. Responsible initiative
is grounded in accountability and bounded by it. These intangible attributes are not abstract in the
process of learning and teaching. They are observable in the behavior of highly motivated and
engaged individuals.

Example Measures

1. Do the instructors effectively exhibit intangible attributes in their own behaviors as they
conduct their training?
! Yes ! No

41 Marceau et al.


2. Do the instructors demonstrate openness in changing their training progression?

Training will not
deviate from a set
schedule
Instructors determine when
it is time to move on a new
task
Soldiers have input into
the progression of training

Example Vignette:

A Soldier is shooting well from the prone unsupported position. However, a drill instructor sees
that the Soldier is not resting the magazine on the ground and that doing so would create a more
stable position. The drill instructor tells the Soldier to rest the magazine on the ground even
though this is a controversial practice. After putting the magazine on the ground, the Soldier
begins to shoot poorly and miss his targets. The instructor, realizing that the Soldier is more
effective with the magazine off the ground, tells the Soldier that he can shoot whichever way is
more comfortable and more effective for him. The Soldier then switched back to his original
position and shot more accurately. Later the instructor explained why he was willing to have the
Soldier try a different shooting position.

A novice, in the situation above, might have not been as flexible or confident enough to reverse
his prior guidance to the student (cf., Edmonson, 1999). However, the more expert instructor
recognized that placing the magazine on the ground is only one tactic that can be employed to
create a stable position. He was willing to risk his credibility to show the Soldier that there is no
single right solution in most situations. The instructor created several teaching points through
the guidance he provided and its immediate impact on the Soldiers performance. It created an
opportunity, for example, to talk about the assumptions behind the rule of not resting the
magazine on the ground. This is more than merely interesting trivia; it makes explicit that there
are assumptions associated with any standard procedure and that these assumptions may have
nothing to do with the task at hand (e.g., learning the adaptive skills of marksmanship) and may
not be valid in the contemporary operating environment. Another is that, while there are several
solutions to most problems, the best solution may vary from time to time and from individual to
individual. The instructor also has the responsibility, however, to explain to the Soldier that
neither is the case that anything goes. This provides an opportunity to help the Soldier become
familiar with the notion of left and right limits for most tasks that both allow for and require some
autonomy, and that exploration is a reasonable manifestation of such bounded initiative.


2.4.9 Hotwashes and Mini-AAR

After-Action Reviews and Hotwashes are a persistently high priority in OBTE. These discussions
about lessons learned should be conducted in an open and supportive manner regarding. The
purpose is not to summarize what just happened in the training but to facilitate a discussion
among the student that allows them to evaluate their own behavior and performance in relation to
others. The distinction between self-assessment and peer-assessment evaporates in the AAR (S.
Flanagan, personal communication, April 30, 2009). The duality between self and other is
replaced by an extended sense of self (cf., Neisser & Jopling, 1997; Neisser, 1996).
Accountability is developed through a deep and enduring sense of the relatedness within the unit
in the context of shared objectives (cf., Deci & Ryan, 1985, 2008; Lerner & Tetlock, 1999, 2003).
Through these discussions, instructors ensure that students grasp the reasons behind a learning
event and its connection to a mission application. Additionally, this self-critique serves as a
problem solving activity in itself, boosting student confidence as they collaboratively discover
how to perform more effectively.

Example Measures

1. Do the instructors ensure the Soldiers can articulate the consequences of their actions?

Tells Soldiers what
went wrong and why it
is important in combat,
but does not discuss
how to mitigate
mistakes next time
Asks Soldiers to explain
what went wrong, why its
important in
combat/mission, but does
not discuss how to mitigate
mistakes next time
Guides Soldiers through
explaining what went
wrong, why it is
important in
combat/mission, how they
might improve next time

2. Do the instructors focus the why of training back to the relationship between the
individual and big picture/mission?

Does not go into depth
on why the Soldiers
need the skills that
were trained (e.g. You
need these skills
Facilitates discussion on the
big picture; less focus on
the individuals
contribution
Facilitates discussion on
the big picture; focuses on
the importance of
individuals (i.e. You can
make a difference)

43 Marceau et al.


Example Vignette:

After a long day of training in the hot sun, all the Soldiers in the Platoon are exhausted and
hungry. Their drill instructor asks all the Soldiers to sit down on the shaded bleachers for a few
minutes prior to returning to the barracks. He then begins to tell them a story about a time when
his unit was deployed in Iraq. They were spending 36-hour cycles out in the city as part of an
effort to integrate with the populace. On one occasion, the drill instructors unit received sniper
fire from a neighboring building and one Soldier was wounded. One team broke off to go after
the sniper. They followed the sniper into a neighboring area of operations and eventually detained
him. They decided to secure a building and spend the night there because it was getting late. They
all were all exhausted and did not want to return to their AO in the night without more support.
By the time the Soldiers made it back to the FOB they had spent over 72 hours out in the city on
only 36 hours worth of food and water. The instructor uses this story not only to make
connections with a mission context but also as a point of departure for collaborative reflection
about the distractions of hardship and its effect on individual and collective decision making.

In the vignette, the instructor leverages the circumstances and effects of a long hot day to make
connections between a mission context and momentary conditions of training that otherwise are
distractions and apparently irrelevant to learning. Without his discussion, the Soldiers would
likely have gone to chow and never thought critically about that days training and its lessons for
performance in theater. Instead, the instructor revealed the operational relevance of their shared
experience in training on that day training. He did this not only by providing a narrative, back
story to which tired and hungry students might not even pay attention. He motivated their
collaborative reflection about the effects of such stressors on individuals and the group. He thus
brought them back into the moment as a learning event, even as they rested.

The AAR serves two purposes. It helps guide the activity of remembering in which the meaning
of prior events is actively reexamined with respect ones current capabilities for knowing and
interacting with the world (Neisser & Hyman, 2000). It also helps participants in the discussion
take ownership of their own learning and teaching (Magolda, 1999). In the context of OBTE and
instructor education, these are tightly interwoven cognitive and social activities (Sidman, Riccio,
Semmens, et al., 2009). The principles and practices of OBTE help focus an AAR on interactions
and interrelationships among tasks and among individuals within a task organized unit (see
Chapter 3). The conversation is more likely to be self-referential, actionable, and empowering
than a conversation about the real or imagined external factors that conspire to impede learning.
Changing the conversation is the easiest way to change the culture of training and education
(Cornell-dEchert, 2009b).
2.5 Uses of the Measures

The COMPASS process yielded a set of measures that have meaning within a community of
practice. Disciplined use of the measures provides the opportunity for the user to become
increasingly oriented to the values and best practices of the community. This provides additional
meaning and validity to the endeavors of anyone who uses the measures for formative feedback
(Mislevy, & Riconscente, 2006; Pellegrino, Chudowsky, & R. Glaser, 2001). It thus is an
important source of confidence for instructors. Given that the measures of shared values and best
practices refer to observable behavior, they also tend to ground confidence in a sense of
competence; that is, sources of confidence are verifiable by oneself and others.

2.5.1 Formative Measures for Instructors

There are three aspects of a good approach to using formative feedback for instructors: (a) it
should relate to the intent of instruction, (b) it should inform actions that can be taken to improve
instruction with respect to the intent, and (c) it should reflect the instructors span of control. The
relationship of the formative measures to the intent of OBTE is explored in this chapter. This
analysis is developed further in Chapter 3 in the context of categories of measures derived from
the relationship between the measures and the principles of OBTE (see also, Appendix A). For
instructors, the relationship emphasizes that feedback should not be collected willy-nilly. A good
instructor has a plan for what to observe based on the interpretation of intent in terms of the
events of the day and the situation at hand. Measures that support the plan should be given
priority. An experienced instructor may utilize additional measures as needed. In any case, it is as
important to decide what measures not to use, as it is to decide which ones to use, in a specific
learning event. An example of such planning is provided in Appendix A. An implication of
planning for formative assessment is that the measures also can be helpful in planning the
learning event itself. The measures can help an instructor prioritize opportunities and exigencies
of the moment in the context of student learning and development over time. They can help
connect prior and subsequent learning events, and they can help make connections between
learning objectives of an isolated event with longer-term outcomes for a student.

The very nature of the COMPASS methodology ensures that the resulting measures are
actionable because they are based on actual experience and observable behavior of instructors
that is verifiable in principle and in practice (MacMillan, et al., in press; MacMillan, et al., 2005).
Minimal interpretation is required to go from use of the measures to an appreciation of what an
instructor should do differently to improve. In a sense, the measures educate the attention of the
instructor about what is done and what can be done in a learning event with respect to the intent
of OBTE. The COMPASS methodology also tends to isolate actions and outcomes that are within
an instructors span of control. There are nuances in this use of formative feedback, however, in
that the command climate and organizational culture may, explicitly or implicitly, discourage the
behavior emphasized by the measures (Bandura, 1995; Rasmussen, 1997; Rasmussen, Brehmer,
& Leplat; 1991). In such cases, the measures have broader utility (see section 2.5.4 below).

2.5.2 Quality Assurance and Instructor Education

Measures that are informative and actionable to instructors are ideal for an external observer who
would provide feedback to an instructor for the purpose of assuring or improving quality. They
also facilitate the identification and promulgation of best practices among peers and throughout
an organization. One reason for this is that there are many measures that collectively represent
best practices for realizing the principles of OBTE and achieving its intent. An instructor could
improve by behaving better with respect to a particular measure or by behaving differently as
suggested by other measures. In a sense, the complete set of measures is a menu from which
instructors can select in the implementation and systematic improvement of OBTE. In any
situation, it will be easier and more effective to implement OBTE through some types of behavior
than others. Over time, this will become clear in the feedback obtained with various measures by
various instructors. The measures thus facilitate sharing of lessons learned that are of common
interest and relevance.

An interesting use of the complete measures set would be to design an instructor education course
around them (cf., Pellegrino, et al., 2001; Sidman et al. 2009). Categories of measures (see e.g.,
Chapter 3) could be used to design associated teaching modules ranging, for example, from a
half-day to multiple days in length. In a sense, such a course would be a guided version of what
45 Marceau et al.

instructors do in using the measures to plan, execute, and assess of their own instruction. In a
pedagogical context, it would be extremely valuable for instructor to be exposed to the
application of the principles and practices of OBTE to different skills or knowledge sets. This
would be good use of the collaborative tactical decision games promoted by the Adaptive
Learning Model (Vandergriff, 2006, 2007) that is closely allied with OBTE.

2.5.3 Continuous Improvement of Assessments

While the COMPASS process results in measures that are specific enough to provide formative
feedback to instructors, they are pointedly not as specific as they could be. In general, there are
just too many nuances and appropriate variations in how a particular instructor can interact with a
particular student. It would not be feasible to list them all, and it wouldnt be desirable to do so
because they would be too dependent on a particular course context. Moreover, in OBTE, it
would antithetical to tell instructors exactly what to do. This is especially important for
organizations in which instructors historically are told what to do or in which instructors expect to
be told what to do. This does not require individual instructors to be left completely on their own
in applying the measures to a particular course or learning event. A good organizational practice
would be to let instructors engage in collaboration with peers and their chain of command in
adapting the measures for their particular situations (Appendix C; Haskins, 2009; C. Haskins,
personal communication, June 2009; cf., Freeman, et al., 2008; Roberto, 2005, 2009). Such a
practice would be consistent with the approach to measure development described in this chapter
if similar methods of verification and validation were used. The most straightforward extensions
of these methods would be to modify the behavioral anchors for specific measures and to subset
or prioritize the measures with respect to local relevance. Adding new measures should be done
as a last resort and with some prudence.

An interesting use of the complete set of formative measures would be to assess or transform
other methods of assessment. The measures described in this chapter could be used to assess the
extent to which other measures capture some aspect of OBTE principles and practices. If an
external measure cannot differentiate good instructor behavior from poor instructor behavior as
captured in any of the measures described in this chapter, it would reveal a fundamental flaw in
the external measure. Given that assessments often drive or condition behavior within an
organization, it would simply be a matter of time before there is no systematic relationship
between instructional behavior and OBTE. On the other hand, an assessment of the assessment
could lead an organization to abandon irrelevant measures or to try to modify them in some way
to be consistent with OBTE and to differentiate good instructor behavior from poor instructor
behavior.

Consider, for example, adoption of OBTE in a particular program of instruction where other
methods and mindset of assessment are already in place. There may be resistance to change such
assessments given an assumption that they can or should be independent of the approach to
instruction. An example might be the use of Likert scales without any behavioral anchors, let
alone anchors specified at the right level of detail and grounded in best practices of OBTE.
Alternatively, a different set of measures may be developed for OBTE at a particular site. This
could be problematic if such measures are developed from invalid assumptions about OBTE or an
inadequate understanding of it. It is common, for example, for stakeholders with an interest in
OBTE to request a training support package or a description of the techniques that instructors can
be told to execute. Any assessments developed from this mindset might perseverate on situation-
specific deviations of OBTE from standard practice such as, in the case of marksmanship
training, placing the magazine on the ground, placing the weapon on safe instead of squeeze the
trigger after SPORTS, shooting 5 round groups, and 200 meter zero. To adhere to such techniques

is not necessarily to practice OBTE. It confuses the incidental with the essential. Any measures
based on observation of such techniques, such as merely counting instances, are likely to miss the
mark. Concurrent validation against the measures in this chapter would reveal the flaws in simple
Likert scales or technique counting. Presumably this would stimulate consideration of
improvements in the assessment methodology.

The measures described in this chapter might also be useful in coding narrative accounts or open-
ended comments about instruction and learning to facilitate summary and synthesis across a
variety of comments. This speculative methodology requires further research to understand its
practical viability and theoretical validity. In any case, it could be useful in stimulating and
guiding refinement of the current measures for specific applications.

2.5.4 Program Evaluation and Organizational Change

The application of assessments should be holistic (Pellegrino et al., 2001; Mislevy, 2006). Quality
assurance and improvement can be pursued at multiple levels in an organization (Rasmussen,
1997; Rasmussen et al., 1991). In this respect, it is noteworthy that the most successful
implementations of OBTE to date appear to be at sites in which it was supported by the command
climate (Haskins, 2009; Perry & McEnery, 2009; Schwitters, 2009). A powerful strategy for
creating a positive command climate for OBTE is to include instructors and their chain of
command in decisions about everything from outcomes to the level of specificity about the
structure of courses and associated assessments (see Chapter 14; Appendix C; C. Haskins,
personal communication, June 2009; cf. Roberts, 2009). The measures described in this chapter
also can be used for troubleshooting problems such as unintended or easily remedied resistance to
instructor behavior considered desirable in OBTE. Thus, every improvement does not rest solely
on the shoulders of the instructors. Some improvements can be accomplished at the
organizational level.

Consider, for example, the effect of a training support package for a particular learning event or
of a program of instruction that is specific about the timing and order of events on particular days
over a period of weeks. This level of detail can have a chilling effect on initiative, prioritization,
tradeoffs, judgment, and problem solving of instructors. Perhaps counterintuitively, it also can
have an undesirable effect on accountability and responsibility. While it may seem that such
detail gives an instructor something quite concrete and easily understandable with respect to
which they can be accountable, it makes them accountable for the wrong things. As instructors,
they should primarily be accountable for student learning, not for adherence to a script that may
or may not promote student learning given the situation at hand. The measures in this chapter can
be used to identify the level of detail that is necessary and sufficient to specify in advance for a
course. The result may be a menu of instructional events from which instructors can choose to
achieve the highest priority learning objectives given the conditions, resources, and time that can
be impossible to predict with specificity or certainty.

2.6 References

Darwin, M. (2008a). Asymmetric Warfare Group combat applications training course (CATC)
senior leader discussion (Briefing, January 23, 2008). Fort Meade, MD: Asymmetric Warfare
Group.
Darwin, M. (2008b). Outcomes-based training and education: fostering adaptability in full
spectrum operations (Briefing, December 2008). Fort Meade, MD: Asymmetric Warfare
Group.
47 Marceau et al.

Cornell-dEchert, B. (2009a). An introduction to outcomes-based training and education. Fort
Meade, MD: Asymmetric Warfare Group.
Cornell-dEchert, B. (2009b). Outcomes-based training and education: Implementation guide.
Fort Meade, MD: Asymmetric Warfare Group.
Bandura, A. (1977). Social learning theory. New York: General Learning Press.
Bandura, A. (Ed.) (1995). Self efficacy in changing societies. Cambridge, UK: Cambridge
University Press.
Bandura, A (1997). Self-efficacy: The exercise of control. New York, NY: Freeman and
Company.
Bransford, J. D., Brown, A. L., & Cocking, R. R. (2000). How people learn: Brain, mind,
experience, & school. Washington, DC: National Academy Press.
Bransford, J. D., & Schwartz, D. L. (1999). Rethinking transfer: A simple proposal with multiple
implications. In A. Iran-Nejad & P. D. Pearson (Eds.), Review of Research in Education (24,
pp. 61101). Washington, DC: American Educational Research Association.
Camic, P.M., Rhodes, J.E., & Yardley (Eds.) (2003). Qualitative research in psychology:
Expanding perspectives in methodology and design. Washington DC: APA.
Cole, M., John-Steiner, V., Scribner, S., & Souberman, E. (1978). Mind in society: The
development of higher psychological processes (L.S. Vygotsky). Cambridge, MA: Harvard
University Press.
Deci, E.L. & Ryan, R.M. (1985). Intrinsic motivation and self-determination in human behavior.
New York: Plenum.
Deci. E.L. & Ryan, R.M. (2008). Self determination theory: a macrotheory of human motivation,
development, and health. Canadian Psychology, 49, 182-185.
Denzin, N.K, & Lincoln Y.S. (Eds.) (2003). Strategies for qualitative inquiry. Thousand Oaks,
CA: Sage.
Edmondson, A. (1999). Psychological safety and learning behavior in work teams. Administrative
Science Quarterly, 44, 350-383.
Endsley, M.R. (1995). Toward a theory of situation awareness in dynamic systems. Human
Factors, 37, 32-64.
Feldbaum, A. (1965). Optimal control systems. New York: Academic Press.
Filatov, N.M. & Unbehauen, H. (2004). Adaptive dual control: Theory and applications.
Springer.
Freeman, J., Jason, J., Aten, T., Diedrich, F., Cooke, N., Winner, J., Rowe. L., & Riccio, G.
(2008). Shared Interpretation of Commander's Intent (SICI). Final Report to the Army
Research Institute for the Behavior and Social Sciences, contract number W74V8H-06-C-
0004.
Gagne, R., & Gibson, J. (1947). Research on the recognition of aircraft. In: J. Gibson (Ed.)
Motion picture training and research. Washington, DC: U.S. Government Printing Office.
Gibson, E. (1988). Exploratory behavior in the development of perceiving, acting, and the
acquiring of knowledge. Annual Review of Psychology, 39, Pages 1-42.
Gibson, E. (1991). An odyssey in learning and perception. Cambridge, MA: MIT Press.
Gibson, J. (1977). The theory of affordances. In R. Shaw & J. Bransford (eds.), Perceiving, acting
and knowing: toward an ecological psychology. Hillsdale, NJ: Erlbaum.
Gibson, J. (1979). The ecological approach to visual perception. Boston: Houghton Mifflin.
Gibson, J. & Gibson, E. (1955). Perceptual learning: Differentiation or enrichment. Psychological
Review, 62, 32-51.
Glaser, B. (2002). Efficiency versus sustainability in dynamic decision making: Advances in inter-
temporal compromising. New York: Springer.
Guay, F., Ratelle, C. F., & Chanal, J. (2008). Optimal learning in optimal contexts: The role of
self-determination in education. Canadian Psychology, 49, 233240.

Haskins, C. (2009, March). Development of outcomes based training. Presentation at the US
Army Asymmetric Warfare Group workshop on Outcomes-based Training and Education,
Applied Physics Laboratory, Johns Hopkins University, Laurel MD.
Henle, M. (1971). The selected papers of Wolfgang Kohler. New York, NY: Liveright.
Kolditz, T. (2007). In extremis leadership: Leading as if your life depended on it. New York, NY:
Wiley.
Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation.
Cambridge, UK: Cambridge University Press.
Lerner, J., & Keltner, D., (2001). Fear, anger, and risk. Journal of Personality and Social
Psychology, 81(1), 146-159.
Lerner, J., & Tetlock, P. E. (1999). Accounting for the effects of accountability. Psychological
Bulletin, 125(2), 255-275.
Lerner, J., & Tetlock, P.E. (2003). Bridging individual, interpersonal, and institutional approaches
to judgment and choice: The impact of accountability on cognitive bias. In S. Schneider and
J. Shanteau (Eds.), Emerging Perspectives on Judgment and Decision Making (pp.431-457).
Cambridge: Cambridge University Press.
Lerner, J., & Tiedens, L. (2006). Portrait of the angry decision maker: How appraisal tendencies
shape angers influence on cognition. Journal of Behavioral Decision Making (Special Issue
on Emotion and Decision Making), 19, 115-137.
Lukey, B. & Tepe, V. (Eds.) (2008). Biobehavioral resilience to stress. Boca Raton, FL: CRC
Press.
MacMillan, J., Garrity, M. J., & Wiese, E. W. (2005, November 25). The value of metrics: You
cant train what you cant measure. Mass High Tech. Retrieved June, 2009, from
http://tinyurl.com/km25pw.
MacMillan, J., Entin, E. B., Morley, R., & Bennett, W. (in press). Measuring team performance in
complex dynamic environments: The SPOTLITE method. Military Psychology.
Magolda, M.B.B. (1999). Creating contexts for learning and self-authorship: constructive-
developmental pedagogy. Nashville, TN: Vanderbilt Press.
Mislevy, R. J., & Riconscente, M. M. (2006). Evidence-centered assessment design: Layers,
concepts, and terminology. In S. Downing & T. Haladyna (Eds.), Handbook of Test
Development (pp. 61-90). Mahwah, NJ: Erlbaum.
Neisser, U. (1976). Cognition and reality. San Francisco: W.H. Freeman.
Neisser, U. (Ed.) (1996). The perceived self: ecological and interpersonal sources of self
knowledge. Cambridge, UK: Cambridge University Press.
Neisser, U. & Hyman, I. (2000). Memory observed: Remembering in natural contexts (second
edition). New York, NY: Worth.
Neisser, U., & Jopling, D.A. (Eds.) (1997). The conceptual self in context. New York: Cambridge
University Press.
Pellegrino, J., Chudowsky, N., & Glaser, R. (Eds.) (2001). Knowing what students know: The
science and design of educational assessment. Washington, DC: National Academy Press.
Perry, R. & McEnery, K. (2009). Army reconnaissance course: Defining the aim point for
reconnaissance leader training. Armor, July-August, 14-20.
Rasmussen, J. (1997). Risk management in a dynamic society: a modeling problem. Safety
Science, 27, 183-213.
Rasmussen, J., Brehmer, B. & Leplat (Eds.) (1991). Distributed decision making: Cognitive
models for cooperative work. London, UK: John Wiley and Sons
Riccio, G. (1993). Information in movement variability about the qualitative dynamics of posture
and orientation. In: K. Newell (Ed.). Variability and Motor Control (pp. 317-357),
Champaign, IL: Human Kinetics.
Riccio, G. (1995). Coordination of postural control and vehicular control: Implications for
multimodal perception and simulation. In Hancock, P., Flach, J. Caird, J., and Vicente, K.
49 Marceau et al.

(Eds.), Local applications of the ecological approach to human-machine systems, pp. 122-
181. Hillsdale, NJ: Erlbaum.
Riccio, G., & McDonald, P. (1998). Characteristics of EVA Mass Handling Skill. Society of
Automotive Engineers Paper No. 981625 (also NASA Technical Paper 3684).
Riccio, G., & McDonald, P. & Bloomberg, J. (1999). Multimodal Perception and Multicriterion
Control of Nested Systems: III. A Functional Visual Assessment Test for Human Health
Maintenance and Countermeasures, NASA/TP-1999-3703c, Johnson Space Center, Houston,
TX.
Riccio G., & Stoffregen T. (1988). Affordances as constraints on the control of stance. Human
Movement Science, 7, 265-300.
Roberto, (2005). Managing for conflict and consensus: Why great leader dont take yes for an
answer. Saddle River, NJ: Wharton School Publishing.
Roberto, M. (2009). Know What You Don't Know: How Great Leaders Prevent Problems Before
They Happen. Saddle River, NJ: Wharton School Publishing.
Safonov, M. (2001). Recent advances in robust control, feedback, and learning. In: S.O. Reza
Moheimani, (Ed.). Perspectives in robust control. London, UK: Springer-Verlag.
Salas, E., & Klein, G. (Eds.) (2001). Linking expertise and naturalistic decision making.
Mahwah, NJ: Erlbaum.
Schwitters, J. (2009, March). Command imperative for change. Presentation at the US Army
Asymmetric Warfare Group workshop on Outcomes-based Training and Education, Applied
Physics Laboratory, Johns Hopkins University, Laurel, MD.
Shaw, R. & J. Bransford, J. (eds.) (1977). Perceiving, acting and knowing: toward an ecological
psychology. Hillsdale, NJ: Erlbaum.
Shaw, R., Turvey, M., & Mace,W. (1982). Ecological psychology: The consequence of a
commitment to realism. In W. Weimer & D. Palermo (Eds.), Cognition and the symbolic
processes (pp. 159226). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
Sidman, J., Riccio, G., Semmens, R., Geyer, A., Dean, C., & Diedrich, F. (2009). Reshaping
Army institutional training: Current training. Final Report to the Army Research Institute for
the Behavior and Social Sciences, contract number W74V8H-04-D-0047 DO 0010.
Tobias, S. & Duffy, T. (eds.) (2009). Constructivist instruction: Success or failure. New York:
Routledge:
Tschacher, W., & Dauwalder, J. (Eds.) (2003). The dynamical systems approach to cognition:
Concepts, and empirical paradigms based on self-organization, embodiment, and
coordination dynamics. River Edge, NJ: World Scientific.
Turvey,M. (1992). Affordances and prospective control: An outline of the ontology. Ecological
Psychology,4, 173187.
Warren, R. & Riccio, G. (1985). Visual cue dominance hierarchies: implications for simulator
design. Transactions of the Society for Automotive Engineering, 6, 937-951.
Winograd, E., Fivush, R., & Hirst, W. (1999). Ecological approaches to cognition: Essays in
honor of Ulric Neisser. Mahwah, NJ: Lawrence Erlbaum.
Vandergriff, D. (2006). Adaptive leaders course, Part 1: Old dogs teaching new tricks. Army,
November 2006, 59-66.
Vandergriff, D. (2007). From swift to Swiss: Tactical decision games and their place in military
education and performance improvement. Performance Improvement, 45(2), 30-39.
Van Merrienboer, J., & Kirschner, P. (2007). Ten steps to complex learning: A systematic
approach to four-component instructional design. New York: Routledge.
ix Riccio, Diedrich, & Cortes (Eds.)
TABLE OF CONTENTS
.
page

Prologue: A Programmatic View of the Inquiry into Outcomes-Based Training & Education.......1
Historicity of our Research on OBTE..........................................................................................1
The Approach and Lessons Learned from the Research..............................................................3
Documentation of the Research ...................................................................................................4

Section I. Development of Stakeholder Requirements for OBTE..............................................6

Chapter 1. Preparation for Full Spectrum Operations ......................................................................7
1.1 Requirements of Full Spectrum Operations...........................................................................8
1.2 Outcomes-Based Training and Education (OBTE)..............................................................10
1.2.1 Exemplar of OBTE: Combat Applications Training Course........................................11
1.2.2 OBTE as a Multifaceted Instructional System .............................................................12
1.3 An Appraisal of Instruction with Respect to OBTE ............................................................13
1.3.1 A Systems Engineering Framework for Integration and Development of OBTE........13
1.3.2 Preparation for Validation and Verification .................................................................14
1.4 References ............................................................................................................................17

Chapter 2. Formative Measures for Instructors ..............................................................................20
2.1 Development of Formative Measures ..................................................................................20
2.1.1 The COMPASS Methodology......................................................................................20
2.1.2 Development of Measures for OBTE...........................................................................21
2.2 Description of Formative Measures .....................................................................................21
2.2.1 Results of the COMPASS Process................................................................................21
2.2.2 Elaboration on the Description of Measures.................................................................23
2.3 OBTE Performance Measures: Planning for Training.........................................................23
2.3.1 Define Outcomes ..........................................................................................................23
2.3.2 Create a Positive Learning Environment ......................................................................25
2.3.3 Create the Parameters of Learning................................................................................27
2.4 OBTE Performance Indicators: Training Execution............................................................28
2.4.1 Communicate the Parameters of Learning....................................................................28
2.4.2 Training Emphasizes Broad Combat or Mission Success ............................................29
2.4.3 Customize Instruction When Possible Based on Constraints/Conditions ....................31
2.4.4 Facilitates Learning of Concepts ..................................................................................32
2.4.5 Creates a positive learning environment.......................................................................34
2.4.6 Instructors Utilize Measures of Effectiveness & Self-Evaluation................................36
2.4.7 Uses scenarios to facilitate learning..............................................................................38
2.4.8 Instructors exhibit intangible attributes in own actions ................................................40
2.4.9 Hotwashes and Mini-AAR............................................................................................42
2.5 Uses of the Measures ...........................................................................................................43
2.5.1 Formative Measures for Instructors..............................................................................44
2.5.2 Quality Assurance and Instructor Education ................................................................44
2.5.3 Continuous Improvement of Assessments....................................................................45
2.5.4 Program Evaluation and Organizational Change..........................................................46
2.6 References ............................................................................................................................46

Table of Contents x
Chapter 3. Principles and Practices of Outcomes Based Training & Education............................50
3.1 Multifaceted Inquiry.............................................................................................................50
3.1.1 Interaction with Progenitors of OBTE..........................................................................51
3.1.2 AWG Documents on OBTE .........................................................................................52
3.1.3 Collaborative Reflection on Participant Observation in CATC ...................................52
3.1.4 Interaction with Stakeholders .......................................................................................53
3.2 Essential Characteristics of OBTE.......................................................................................53
3.2.1 The Meaning of Developmental is a Critical Difference..............................................53
3.2.2 The Definition of Outcomes is a Critical Difference....................................................56
3.2.3 The Emphasis on Values and Causally Potent Intangibles is a Critical Difference .....58
3.2.4 The Meaning of Experience is a Critical Difference ....................................................61
3.2.5 The Emphasis on Instructor-Student Interactions is a Critical Difference ...................62
3.2.6 The Emphasis on Learning to Learn is a Critical Difference .......................................63
3.2.7 The Emphasis on Collaborative Design and Development is a Critical Difference.....65
3.3 Toward a Grounded Theory for OBTE................................................................................66
3.3.1 Need for an Integrated Interdisciplinary Framework ...................................................66
3.3.2 Formative Measures of Instructor Behavior as Evolving Best Practices of OBTE......67
3.4 Emerging Best Practices in OBTE for a Community-Centered Environment.....................68
3.4.1 Leadership and Enculturation of Soldiers.....................................................................68
3.4.2 Robust and Adaptable Plan...........................................................................................70
3.4.3 Instructors as Role Models ...........................................................................................70
3.4.4 Collaborative Identification of Outcomes and Measures .............................................71
3.5 Emerging Best Practices in OBTE for a Knowledge-Centered Environment .....................71
3.5.1 Integrated Understanding of Basic Soldier Skills in Full Spectrum Operations ..........72
3.5.2 Task Relevance of Planned Instructional Events..........................................................72
3.5.3 Reveal Operational Relevance of Training...................................................................73
3.5.4 Incorporate Stress into Instructional Events .................................................................73
3.5.5 Identify General Lessons Learned and Extrapolate to New Situations ........................74
3.6 Emerging Best Practices in OBTE for an Assessment-Centered Environment ...................74
3.6.1 Collaborative Reflection and Problem Solving ............................................................75
3.6.2 Communication.............................................................................................................75
3.6.3 Nature and Extent of Guidance.....................................................................................76
3.6.4 Establish a Pervasive Mindset of Collaborative Reflection..........................................76
3.7 Emerging Best Practices in OBTE for a Learner-Centered Environment ...........................77
3.7.1 Soldier Motivation and Development of Intangibles....................................................77
3.7.2 Plan for Development of the Individual .......................................................................78
3.7.3 Get Students to Take Ownership ..................................................................................78
3.7.4 Collaborative Reflection as a Means to Develop Self Efficacy....................................79
3.8 References ............................................................................................................................79

Chapter 4. Grounded Theory for Values-Based Training & Education.........................................86
4.1 Exploration of Holistic and Functionalistic Underpinnings for OBTE ...............................86
4.1.1 Fundamental Units of Analysis.....................................................................................87
4.1.2 Nested Time Scales and Adaptability...........................................................................88
4.1.3 Adaptability and Ambiguity .........................................................................................90
4.1.4 Mechanistic Analogies and Predominant Experimental Paradigms .............................92
4.2 Three Pillars for the Scientific Foundation of OBTE ..........................................................93
4.2.1 Ecological Psychology..................................................................................................93
4.2.2 Self-Efficacy Theory.....................................................................................................97
4.2.3 Positive psychology ......................................................................................................98
4.3 A More Integrated Scientific Infrastructure.......................................................................101
xi Riccio, Diedrich, & Cortes (Eds.)
4.3.1 Self Determination Theory .........................................................................................101
4.3.2 Situated Learning Theory ...........................................................................................103
4.3.3 Existential Psychology................................................................................................105
4.4 Building on the Scientific Infrastructure for OBTE...........................................................109
4.4.1 Triadic Frameworks....................................................................................................109
4.4.2 Further Development ..................................................................................................112
4.5 References ..........................................................................................................................112

Chapter 5. Passion and Reason in Values-Based Learning & Development ...............................118
5.1 The Nested Self ..................................................................................................................118
5.1.1 An Alternative to Individual versus Collective ..........................................................118
5.1.2 Cognition and Reality .................................................................................................119
5.2 Conscious Experience and the Dynamics of Thinking ......................................................122
5.3 Emotion, Information, and Engagement ............................................................................125
5.3.1 Ecological Perspective on Emotion ............................................................................125
5.3.2 Emotion as Engagement .............................................................................................126
5.3.3 Implications for Training and Education....................................................................129
5.4 Emotion, Decision-Making, and Inter-Temporal Choice...................................................129
5.4.1 Toward a More Integrated Theory..............................................................................129
5.4.2 Emotion and Decision-Making...................................................................................130
5.4.3 Emotion and Nested Time Scales ...............................................................................131
5.4.4 Neuroeconomics and Inter-Temporal Reasoning .......................................................132
5.5.5 Inter-Temporal Reasoning and Adaptive Dynamical Systems...................................133
5.5 Beyond Science..................................................................................................................134
5.5.1 Existentialism..............................................................................................................134
5.5.2 The Soldier-Scholar as an Emergent Property of a Collective Pursuit.......................135
5.6 References ..........................................................................................................................137

Section II. Verification and Validation of OBTE as a Service System..................................142

Chapter 6. Initial Impressions of Participation in CATC.............................................................143
6.1 Methods..............................................................................................................................143
6.1.1 Participants..................................................................................................................143
6.1.2 Procedure ....................................................................................................................143
6.1.3 Analyses......................................................................................................................144
6.2 Results ................................................................................................................................144
6.3 Implications for Service System Development: Peer Review ...........................................146
6.4 References ..........................................................................................................................147

Chapter 7. Local Development of Measures of Effectiveness .....................................................149
7.1 What do Instructors Believe Soldiers Should Learn in Initial Entry Training?.................149
7.2 Measure Development Process ..........................................................................................150
7.3 What do OBTE-Trained DS Believe is Important to Assess in BRM/ARM? ...................151
7.4 Implications........................................................................................................................156
7.5 Conclusions ........................................................................................................................158
7.6 References ..........................................................................................................................159

Table of Contents xii
Chapter 8. Observations of Behavior and Communication in Rifle Marksmanship Training .....160
8.1 Methods..............................................................................................................................160
8.1.1 Participants..................................................................................................................160
8.1.2 Procedure ....................................................................................................................160
8.1.3 Analyses......................................................................................................................161
8.2 Results ................................................................................................................................163
8.2.1 Behavior of DS ...........................................................................................................163
8.2.2 Behavior and Performance of Privates .......................................................................165
8.2.3 Patterns of Communication ........................................................................................168
8.2.4 Potential Influence of Instructor Behavior on Performance of Privates.....................170
8.3 Implications for Service System Development..................................................................171
8.3.1 Verification of OBTE .................................................................................................171
8.3.2 Validation of OBTE....................................................................................................172
8.4 References ..........................................................................................................................173

Chapter 9. Impact on Rifle Marksmanship Training....................................................................174
9.1 Behavioral Data Collection During Basic Rifle Marksmanship........................................174
9.1.1 Method........................................................................................................................174
9.1.2 Assessment..................................................................................................................175
9.1.3 Results An Overview...............................................................................................177
9.1.4 Evidence for Influence of OBTE................................................................................178
9.1.5 Behavior of Drill Sergeants after Exposure to OBTE ................................................180
9.1.6 Behavior of Privates....................................................................................................182
9.1.7 Patterns of Communication ........................................................................................186
9.1.8 Summary.....................................................................................................................186
9.2 Attitudes Toward an OBTE in Basic Training...................................................................187
9.2.1 Method........................................................................................................................187
9.2.2 Results.........................................................................................................................187
9.4 References ..........................................................................................................................191

Chapter 10. Influence of CATC in an Operational Setting ..........................................................192
10.1 Methods............................................................................................................................192
10.1.1 Participants................................................................................................................192
10.1.2 Procedure ..................................................................................................................192
10.1.3 Analyses....................................................................................................................193
10.2 Results ..............................................................................................................................193
10.2.1 Downstream Impact on Marksmanship ....................................................................193
10.2.2 Downstream Impact on Training in the Units ..........................................................194
10.2.3 Downstream Impact on Self Efficacy.......................................................................195
10.3 Implications for Service System Development: Validation.............................................196
10.4 References ........................................................................................................................197

Chapter 11. Implications for Service System Development.........................................................198
11.1 Lessons Learned about Transfer of OBTE.......................................................................198
11.2 Implications for Service System Development................................................................199
11.2.1 Further Development and Analysis of Stakeholder Requirements for OBTE..........199
11.2.2 Further Development of OBTE as a Service System ...............................................199
11.2.3 Further Verification and Validation of OBTE..........................................................201
11.3 References ........................................................................................................................203

xiii Riccio, Diedrich, & Cortes (Eds.)
Section III. Further Development of OBTE as a Service System ..........................................206

Chapter 12. Development of General Measures for Students ......................................................207
12.1 Intent ................................................................................................................................207
12.2 Performance Measure Development Process...................................................................207
12.2.1 Phase One: Define Performance Indicators (PI).......................................................207
12.2.2 Phase Two: Translate PI into performance measures...............................................208
12.2.3 Phase Three: Measure refinement.............................................................................208
12.2.4 Phase Four: Retranslation of Measures ....................................................................208
12.3 Product of Measure Development....................................................................................209
12.3.1 Learner Perception of the Instructor and Course ......................................................209
12.3.2 Learner Engagement .................................................................................................211
12.3.3 Student Relationship with Teacher ...........................................................................212
12.3.4 Student Results .........................................................................................................214
12.3.5 Self-Report Measures ...............................................................................................216
12.4 Conclusion........................................................................................................................217
12.5 References ........................................................................................................................217

Chapter 13. Adapting OBTE in a Classroom Environment .........................................................219
13.1 Intent ................................................................................................................................219
13.2 Observing OBTE in the Classroom Environment............................................................219
13.2.1. Participants...............................................................................................................219
13.2.2. Procedure .................................................................................................................220
13.2.3. Measures ..................................................................................................................220
13.3 Utility of OBTE Measures in a Classroom Environment ................................................220
13.3.1 Generality of Measures.............................................................................................220
13.3.2. Implications for Improvement of Measures.............................................................221
13.3.3 Implications for improvement of course design .......................................................222
13.4 Use of 360 Reviews for Collaborative Reflection..........................................................223
13.4.1 The Role of a 360 Review in OBTE.......................................................................223
13.4.2 Narrative of a Participant Observer ..........................................................................225
13.5 Learning, cognitive load and motivation..........................................................................228
13.5.1 The NASA Task Load Index as a subjective measure of student workload.............228
13.5.2 Results.......................................................................................................................229
13.5.3 Implications ..............................................................................................................230
13.6 Conclusions ......................................................................................................................230
13.7 References ........................................................................................................................231

Chapter 14. Organizational Climate and Creation of Durable Change ........................................233
14.1 The Need ..........................................................................................................................233
14.2 Initial Indications of Possible Resistance to Change .......................................................234
14.3 Models and Considerations for Sustainable Change........................................................235
14.3.1 The Change Transition Period ..................................................................................235
14.3.2 Organizational Culture..............................................................................................237
14.3.3 Clarity of Mission and Shared Understanding..........................................................237
14.3.4 Relevant Observations During the Current Investigation.........................................238
14.3.5 Organizational Support and Incentives.....................................................................238
14.4 Conclusions ......................................................................................................................239
14.5 References ........................................................................................................................239
Table of Contents xiv
Chapter 15. Five ways OBTE can enable the Army Leader Development Strategy....................242
15.1 Background ......................................................................................................................242
15.2 An Emerging Consensus ..................................................................................................244
15.2.1 What Part to Balance?...............................................................................................244
15.2.2 Improving Training, by Design ................................................................................245
15.2.3 Increased Use of dL and Dependence on Self-Development ...................................246
15.2.4 Future Orientation, Unknown Requirements............................................................247
15.2.5 The Quality Instructor Challenge .............................................................................247
15.2.6 Purpose and Design are Key.....................................................................................248
15.2.7 A Natural Advantage ................................................................................................249
15.2.8 Task Specialization or Generalized Competency .....................................................249
15.3 Conclusion........................................................................................................................251
15.4 References ........................................................................................................................252

Epilogue. Integration of Leader Development, Education, Training, and Self-Development .....254
Toward Values-Based Standards for Army Doctrinal Requirements ......................................254
Nested Standards and Quality Assurance.................................................................................256
Needs and Opportunities for Staff & Faculty Development ....................................................259
A Role for Science and Measurement .................................................................................259
Toward Best Practices in Instructor Education....................................................................260
Critical Considerations for Further Scientific Investigation ....................................................263
The Necessity of Long-Term Studies ..................................................................................263
False Dichotomy of Objective-Subjective...........................................................................264
Clarity About What Is Evaluated.........................................................................................265
Next Steps ............................................................................................................................266
References ................................................................................................................................268

Section IV. Appendices...............................................................................................................270

Appendix A. OBTE Principles & Practices: Instructor Measures................................................271
A.1 Genesis of Formative Measures for Instructors ................................................................271
A.2 Principles of Outcomes-Based Training & Education ......................................................272
A.3 Guide to Using Measures of Instructor Behavior..............................................................276
A.4 Complete Menu of Instructor Measures............................................................................279

Appendix B. OBTE Principles & Practices: Student Measures ...................................................318
B.1 Guide to Using Measures of Student Behavior .................................................................318
B.2 Complete Menu of Student Measures ...............................................................................319

Appendix C: A Commanders View of Outcomes-Based Training and Education.....................340
Summary ..................................................................................................................................340
Definition.............................................................................................................................340
Description...........................................................................................................................340
Elements of OBTE. ..................................................................................................................341
Developing the Outcomes....................................................................................................341
Developing the Training Plan..............................................................................................341
Conducting Training............................................................................................................342
How Training is Assessed....................................................................................................344
Conclusion................................................................................................................................344
xv Riccio, Diedrich, & Cortes (Eds.)
Appendix D: Warrior Ethos..........................................................................................................345
Analysis of the Concept and Initial Development of Applications..........................................345
Current Understanding of Warrior Ethos.............................................................................345
Purpose.................................................................................................................................348
Approach..............................................................................................................................348
Expansion of the Definition of Warrior Ethos.....................................................................348
The Tenets of Warrior Ethos ...............................................................................................349
Clarifying the Definition of Warrior Ethos..........................................................................351
Warrior Attributes Derived from the Tenets of Warrior Ethos ...........................................353
References ................................................................................................................................355
Supplementary Work Product from Warrior Ethos Project .....................................................355

Appendix E: Indicators of Warrior Ethos.....................................................................................356
Methods....................................................................................................................................356
Participants...........................................................................................................................356
Instruments and Facilities ....................................................................................................356
Procedure .............................................................................................................................356
Results ......................................................................................................................................358
Qualitative Findings.............................................................................................................358
Quantitative Findings...........................................................................................................358
Discussion ................................................................................................................................359

!"#$%&'#( #* &+, '(",-&'./&'#( /- 0,*$,1&,2 '( &+, 1+/3&,0- #* &+'- 4#(#.0/3+5

Toward Formative Measures For Instructors in Outcomes Based Training & Education

Diunggah oleh

Informasi Dokumen

Judul Asli

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Toward Formative Measures For Instructors in Outcomes Based Training & Education

Diunggah oleh

Hak Cipta:

Format Tersedia

Marceau, R., Jackson, C., Dean, C., Diedrich, F., Artis, S., Wiese, E., & Riccio, G. (2010).

Formative Measures for Instructors. In: Riccio, G.,

Anda mungkin juga menyukai