IMPLEMENTATION OF A SCHOOL-WIDE

ADOLESCENT CHARACTER EDUCATION AND

PREVENTION PROGRAM
Evaluating the Relationships Between Principal
Support, Faculty Implementation, and Student
Outcomes

Carol K. Holtzapple, J. Suzy Griswold, Kathleen Cirillo, and Jim Rosebrock

Department of Research, The Flippen Group

Noreen Nouza
Office of Program and Professional Development, Oneida-Herkimer-Madison BOCES

Cami Berry
Safe Schools Unit, Division of Educational Services, Riverside County Office of Education (RCOE)

School-based character education and violence prevention programs focus on improving prosocial competen-
cies and reducing negative behaviors in students. The Capturing Kids Hearts Campus by Design model is a
school-level intervention that impacts student behavior by enhancing school climate through improved rela-
tional and conflict management skills. The purpose of this randomized controlled study is to evaluate the
effect of the intervention model on risk factors and protective factors that impact student behavior. The results
demonstrate that the level of principal support and quality of faculty implementation is related to and predic-
tive of student acquisition of specific prosocial behaviors (protective factors). Using school archival data for
discipline referrals, univariate ANOVA analyses demonstrated that schools implementing Capturing Kids
Hearts Campus by Design experienced on average a 22% decrease in discipline referrals whereas control
schools experienced on average an 11% increase in referrals. In addition, students in intervention schools
exhibited a 26% increase in prosocial behaviors associated with the training whereas students in control
schools exhibited a 15% decrease in these behaviors.

Carol K. Holtzapple, Director of Research, The Flippen Group, 1199 Haywood Dr., College Station, TX 77845. E-mail:
carol.holtzapple@flippengroup.com

Journal of Research in Character Education, 9(1), 2011, pp. 7190 ISSN 1543-1223
Copyright 2011 Information Age Publishing, Inc. All rights of reproduction in any form reserved.
72
INTRODUCTION
Components of the Capturing Kids Hearts
Campus by Design model have been widely
disseminated throughout the United States. By
the end of 2009, these programs had been
implemented in more than 5,500 schools in 47
states. The model is based upon a multifacto-
rial causal model of youth risk behaviors and is
closely aligned with Social Learning Theory
(Bandura, 1986).
Character education programs encourage
schools to create learning environments that
foster the development of ethical, responsible
students who demonstrate caring concern for
others (What Works Clearinghouse, 2007).
Capturing Kids Hearts has been used in con-
junction with other educational programs to
meet the goals of federal initiatives such as
Smaller Learning Communities (Flippen
Group, 2009; U.S. Dept. Ed., 2009) in order to
promote a healthy school climate. A positive
school culture is dependent upon a number of
parameters that include developing a safe, car-
ing environment, establishing rules that hold
all school members to a high standard of
behavior, and building healthy relationships
that create a professional culture and increase
student connectedness (Lickona & Davidson,
2005; Character Education Partnership, 2010).
Resnick et al. (1997) reported that this con-
nectedness to a parent or school-based role
model (teacher) is a protective factor that pro-
tects youth against every measure of health risk
behavior except history of pregnancy. A subse-
quent study (Resnick Ireland, & Borowsky,
2004) reported that the percentage of youth
involved in violence is reduced in the presence
of protective factors, despite the continued
presence of significant risk factors. In addition,
the Office of Juvenile Justice and Delinquency
Prevention (2004) states that programs that
have the ability to increase students attach-
ment to their school are often highlighted as
models for violence prevention. Results from
the National Longitudinal Study on Adolescent
Health demonstrated that student connected-
ness reduced negative behaviors in four
Journal of Research in Character Education Vol. 9, No. 1, 2011
domains of adolescent health and morbidity:
emotional health, violence, substance use, and
sexuality. Thus, the Capturing Kids Hearts
Campus by Design model is considered to be
both a character education and violence pre-
vention intervention because it is designed to
strengthen students connectedness to school
through enhancing protective factors (i.e.,
strong bonds with teachers; clear rules of con-
duct that are consistently enforced) and target-
ing modifiable risk factors (i.e., inappropriate
behavior; poor social coping skills).
Capturing Kids Hearts Campus by Design
is a skill intensive, systemic process designed
to develop high-performing school cultures,
align organizational and individual behaviors
to outcomes, and increase school connected-
ness. As a result of implementing this inte-
grated approach, (1) faculty members build
intentional cultures that emphasize connected-
ness with students and with each other, (2)
negative behaviors are minimized while learn-
ing is maximized, and (3) students acquire
communication and conflict resolution skills
that help them succeed both in school and after
graduation.
Evidence-based longitudinal research stud-
ies and comprehensive literature reviews have
demonstrated that connectedness to an appro-
priate role model (teacher) and effective class-
room management exert a positive impact on
students behavior (Resnick et al., 1997) and
academic achievement (Wang, Haertel, &
Walberg, 1993). In a meta-analysis of more
than 100 studies, Marzano (2003) reported that
the keystone for all aspects of effective class-
room management was the quality of
teacher-student relationships.
The goal of Capturing Kids Hearts Campus
by Design is to develop leaders who have the
knowledge and skill-sets to develop healthy
relationships with students, colleagues, and
members of the community. Through these
relationships, adults model the character traits
that students must acquire in order to reach
their full potential. Capturing Kids Hearts not
only identifies a process designed to build
high-quality relationships, but also provides an
Implementation of a School-Wide Adolescent Character Education and Prevention Program
opportunity for participants to learn and prac-
tice skills that exert a positive impact on class-
room management.
The theoretical framework underlying the
Capturing Kids Hearts Campus by Design
learning approach is the Social Cognitive The-
ory developed by Bandura (1986). A central
premise of Social Cognitive Theory is that
behavior is dynamic, dependent upon both per-
sonal constructs and environmental factors that
influence each other simultaneously (Perry,
Baranowski, & Parcel, 1991). Social Cognitive
Theory asserts that the likelihood that a behav-
ior will be repeated is a function of the individ-
uals expectancy that the behavior will lead to
a particular outcome and of the extent to which
that outcome is valued (Norman, 1991).
In addition, a person must have self-effi-
cacy in order for the behavioral change to
occur: individuals must first have the convic-
tion that they can execute the behavior suc-
cessfully before the desired behavior is
acquired. This conviction is acquired through
observing others and implementing self-regu-
latory mechanisms (Bandura, 1986). Self-reg-
ulation involves self-assessing behavioral
performance, keeping track of actions, com-
paring actions to a standard of behavior, and
self-directing acquired behaviors. An out-
standing summary of the assumptions, state-
ments, and concepts of Social Cognitive
Theory that help explain how people develop
and maintain behaviors was developed by
Glanz, Rimer, and Lewis (2002). The 11 fac-
tors that affect behavioral change, the inter-
vention strategies that promote positive
changes in behavior by addressing these fac-
tors, and the program inputs in the Capturing
Kids Hearts process designed to implement
the intervention strategies are provided in
Table 1. For instance, observational learning
(factor 7) is a factor that impacts behavior by
helping the learner acquire new behavior by
observing the behavior of someone else. The
intervention strategy includes providing credi-
ble role models (teachers and student leaders)
of the targeted behavior. The modules Lead-
ership Skills, EXCEL Model, and Values/
73
Building Healthy Relationships specifically
address how to implement this intervention
strategy. Implementation measures (surveys
and observational tools) are then used to deter-
mine whether or not the program components
within these modules were delivered with
fidelity (i.e., adherence).
Previously reported experimental (Castro
Johnson, & Smith, 2008; Danaher, 2006) and
quasi-experimental (Cirillo-Teverbaugh, 1994;
Cirillo-Teverbaugh & Colwell, 1993; Sher-
wood, 2003) studies, as well as numerous lon-
gitudinal case studies (The Flippen Group,
2006), have highlighted the positive effects
that components of this comprehensive process
exert on behavioral and academic outcomes.
The magnitude of the effect is impacted by a
number of variables, including administrative
support for the process and level of program
implementation by school faculty members.
Comprehensive interventions usually are
not implemented fully in the first year, and
even in subsequent years, not all schools
implement all of the core components of inter-
ventions as prescribed by the developer
(Berends, Bodilly, & Kirby, 2002; Glennan,
1998; St. Pierre & Kaltreider, 2004) due to
conflicts or issues that inevitably arise when
attempting to disseminate a school-wide pro-
gram. As a result of this variability in imple-
mentation, Zhang, Shkolnik, and Fashola,
(2005) report that student outcomes such as
academic achievement may lag years behind
initial program implementation. In addition,
the relationship between faculty implementa-
tion and student outcomes is often weak, and
becomes weaker when implementation stag-
nates or declines over time.
Interventions that impact an entire school
community must be implemented fully and
with fidelity in order for schools to obtain
maximum effect. A number of studies (Domi-
trovich & Greenberg, 2000; Durlack & Wells,
1998; Gresham, Gansle, Noell, & Cohen,
1993) report that few programs provide data
on levels of implementation or consider
whether implementation quality is related to
intervention outcomes. Hallfors and Godette
74 Journal of Research in Character Education Vol. 9, No. 1, 2011

TABLE 1
Relating Selected Capturing Kids Hearts Topics and
Activities to Behavioral Factors and Intervention Strategies Identified by Social Cognitive Theory
# Factors That Definitions Intervention Capturing Kids Hearts
Affect Behavior of Factors Strategies Topics & Activities
1 Environment Factors external to person Provide opportunities for social Name Game Warped Speed
such as social support and support SocialAnxiety/
family DecreasingSocial Anxiety
2 Situation Perception of the Correct misperceptions and First Impressions/What
environment promote healthy and safe People Notice
atmosphere Styles of Interaction
3 Behavioral Knowledge and skill to Promote mastery learning 1 Minute Speeches
Capability perform a given behavior through social skills and skills Leadership Skills
training Steps for Personal Growth
4 Expectations Anticipatory outcomes of Model positive outcomes of EXCEL Model
a behavior healthful behavior Listening Skills
5 Expectancies The values that the person Present outcomes of change that Social Contract
places on given outcomes; have functional meaning Discipline Model
incentives
6 Self-Regulation Personal goal-directed Provide opportunities for EXCEL Model
behavior or performance self-assessing, self-monitoring, Personal Mastery
goal setting, problem solving, Steps for Personal Growth
and self-reward
7 Observational Behavioral acquisition that Include credible role models of Leadership Skills
Learning occurs by watching the the targeted behavior EXCEL Model
actions and outcomes of Values/Building
others behavior Healthy Relationships
8 Reinforcements Responses to a persons Promote self-initiated rewards EXCEL Model
behavior that increase or and incentives Leadership Skills
decrease the likelihood of Steps for Personal Growth
re-occurrence
9 Self-Efficacy The persons confidence in Approach behavioral change in EXCEL Model
performing a particular small steps to ensure success Reframing
behavior
10 Emotional Coping Strategies or tactics that Provide training in problem EXCEL Model
Responses are used by a person to solving and stress management Reframing
deal with emotional Question Process/Discipline
stimuli Model
11 Reciprocal The dynamic interaction of Consider multiple avenues to Every process in the Captur-
Determinism the person, the behavior, behavioral change, including ing Kids Hearts training
and the environment in environment, skill acquisition, addresses this assumption
which the behavior is and personal change
performed

(2002) found that while most school districts
selected research-based programs, many pro-
grams were implemented with low fidelity.
Other studies (Kam, Greenberg, & Walls,
2003, and references therein) demonstrate that
implementation quality impacts student out-
comes; thus, even an empirically-validated,
school-based program does not produce con-
sistent student outcomes if it is not imple-
mented with fidelity.
Greenberg, Domitrovich, Graczyk, and
Zins (2005) identify a number of reasons for
including implementation assessment when
evaluating interventions. For instance, lack of
Implementation of a School-Wide Adolescent Character Education and Prevention Program
implementation data may result in researchers
incorrectly determining that a program is inef-
fective when, in fact, the program was imple-
mented so poorly that no analyses can provide
meaningful conclusions. Additionally, pro-
gram evaluators may be faced with consider-
able unexplained variance in program effects.
Obtaining implementation information can
help researchers with both types of research
difficulties by providing data that allows them
to draw valid conclusions and identify reasons
for variability in program outcomes.
In this study, we report the first-year results
obtained after implementation of a
school-based character education program,
Capturing Kids Hearts Campus by Design.
The primary questions of the study include: (1)
How well do principals in intervention schools
support the program as measured by personally
modeling the skills taught in the process and
providing a systemic mechanism by which pro-
gram progress can be discussed? (2) How well
do teachers in intervention schools implement
Capturing Kids Hearts skills in the classroom?
(3) What is the relationship between principal
support and teacher implementation? (4) How
well do students acquire the skills that are mod-
eled by teachers? (5) What is the relationship
between teacher implementation of the process
and student acquisition of skills? (6) What spe-
cific changes occur in student behaviors (i.e.,
changes in the number of discipline referrals or
increases in specific prosocial behaviors) as a
result of program implementation? The rela-
tionship between fidelity of implementation
and changes in student outcomes is discussed.
METHODS
Background
School leadership teams from the
Oneida-Herkimer-Madison Board of Coopera-
tive Educational Services (BOCES) in New
York participated in a state-wide meeting that
led to the directive for districts to improve stu-
dent outcomes. In response to this directive,
75
the leadership team developed a BOCES-wide
vision specifically addressing student behav-
iors and skills that would prepare students to
be successful in the workplace beyond high
school graduation. The findings from the
National Longitudinal Study on Adolescent
Health (Resnick et al., 1997) indicated that the
leading factor supporting positive outcomes
and preventing youth participation in negative
behaviors is connectedness to an adult role
model.
Oneida-Herkimer-Madison BOCES imple-
mented a strategic plan to increase student
connectedness. A Professional Development
Council was convened and involved a commit-
tee of 20 superintendents, assistant superinten-
dents, and other district leadership members
tasked with examining program options. The
team met for a number of months, reviewing
regional needs assessments, interviewing prin-
cipals and staff, and engaging in extensive
committee discussions.
All members of the Professional Develop-
ment Council were asked to consider profes-
sional development programs that would help
connect administrators, teachers, and students.
Eight research-based programs (as evidenced
by inclusion of the program on a model pro-
gram list or by publication of research articles
in peer-reviewed journals) were identified that
would address relational connectedness. The
Capturing Kids Hearts process, a comprehen-
sive, school-based program designed to
improve prosocial competencies and reduce
negative behaviors in students, was the pro-
gram that was chosen. The committee then
developed a test group of individuals to partic-
ipate in the training and provide feedback.
The test group, consisting of 52 teachers
and administrators, attended a Capturing Kids
Hearts 3-day workshop. A BOCES-wide meet-
ing was then convened to discuss outcomes of
the training, determine the districts options,
and develop next steps. A unanimous vote was
provided by the test group indicating that the
program would be of great benefit for the stu-
dents and would help the schools meet
state-directed, district-wide goals.
76
Before committing a significant amount of
funding to implement the training dis-
trict-wide, the BOCES leadership team
decided to invest time and resources in a pilot
study that would be capable of measuring the
effectiveness of the program in local high
schools. Principals within the region that were
willing to participate in the study were asked to
submit school archival data from the preceding
three years describing demographic, behav-
ioral, and academic trends. Due to limited
funding, BOCES was able to include only six
schools in the research study. Statistical mod-
eling (Holtzapple, 2009; Spybrook Rauden-
bush, Liu, & Congdon, 2006) indicated that for
a six-school study using randomization at the
school level, researchers could encounter sig-
nificant difficulties in measuring outcomes.
Therefore, a partner district was considered in
order to increase the number of schools in the
study, thus improving the likelihood that
changes in measurable outcomes could be
detected if they occur.
The Riverside County Office of Education
(RCOE) in California also sought to document
the effectiveness of this program in local
schools. The leadership had previously devel-
oped a strategic initiative to increase school
connectedness within the county and had
implemented the Capturing Kids Hearts pro-
cess for a number of years, but had not partici-
pated in a randomized, controlled research
trial. The RCOE leadership team asked princi-
pals interested in participating in a study to
submit the same data as that submitted by the
BOCES schools.
General Research Design
In order to meet federal and state guidelines
for using evidence-driven research, an experi-
mental (randomized, controlled), blocked,
research design was used, with schools chosen
as the unit of randomization. Randomization at
the school level was necessary because there
would be a significant spillover effect
between intervention and control participants
within a school that would tend to underesti-
Journal of Research in Character Education Vol. 9, No. 1, 2011
mate the apparent effectiveness of the inter-
vention. Eighteen public high schools served
by BOCES and RCOE provided academic and
behavioral data from the 3 years preceding the
study (SY2005, 2006, and 2007). After deter-
mining the level of variability in the informa-
tion derived from the 3-year discipline referral
data (Holtzapple, 2011), eight schools exhibit-
ing the lowest variability (< 25%) in the 3-year
data were paired according to school type
(grades 7-12 or grades 9-12) and then ran-
domly assigned to intervention or control
groups. Schools with greater levels of variabil-
ity in the data were not eligible for participa-
tion in the study.
Participant Sample
The study sample included 8,350 students
in Grades 7-12 and 469 teachers from 6 high
schools located in districts served by the
Oneida-Herkimer-Madison Board of Coopera-
tive Educational Services (BOCES) in New
York and 2 high schools served by the River-
side County Office of Education (RCOE). In
the intervention group, 4% of the students
were African American, 33% were Hispanic,
and 57% were Caucasian. In the control group,
5% of the students were African American,
26% were Hispanic, and 65% were Caucasian.
Overall, 4% of the students were African
American, 28% were Hispanic, and 62% were
Caucasian, with 30% of the students eligible
for free or reduced lunch.
Program Description
Schools assigned to the treatment condition
received all of the programs and processes in
the school-wide model, Capturing Kids
Hearts Campus by Design, including Captur-
ing Kids Hearts, Process Champions, Execu-
tive Coaching, and Phone Consulting. These
trainings address the mechanisms and pro-
cesses of social skills instruction (SSI) that
lead to improved student behavior by promot-
ing skills acquisition (i.e., modeling, coaching,
and behavioral rehearsal), enhancing skills
Implementation of a School-Wide Adolescent Character Education and Prevention Program
performance, removing competing behavior,
and facilitating maintenance of social skills
(Gresham & OShaughnessy, 2002).
Capturing Kids Hearts is a 3-day teacher
and administrator training program that pro-
vides school faculty with the skills they need to
model and teach relational skills, communica-
tive competencies, problem-solving skills, cit-
izenship, and consequential thinking. This
program provides the foundation on which
teachers build effective classroom manage-
ment strategies including engaging students at
the beginning of class, developing classroom
expectations for behavior, providing effective
feedback, and identifying and addressing con-
flict.
Process Champions is a 2-day teacher and
administrator training program that reinforces
the Capturing Kids Hearts training and devel-
ops teachers and administrators who act as
on-site process mentors, helping their col-
leagues apply and master all aspects of the
training. Process Champions mentors receive
special additional instruction that prepares
them to assist and support their peers in imple-
menting the process with fidelity.
Campus Consulting consists of two, 2-day
sessions that provide the opportunity for cam-
pus administrators and Process Champions
(teacher mentors) to work one-on-one with a
Project Consultant from the Flippen Group
who helps improve implementation of the pro-
cess. One session is offered in each school
term (fall and spring).
Phone Consulting consists of six, 1-hour
phone calls that provide support for campus
administrators and Process Champions as they
implement the Campus by Design process.
During the consultations, the Project Consul-
tant may assist with setting implementation
expectations, dealing with conflict, establish-
ing an effective communication system, or
addressing other issues that require attention.
The consultation phone calls are conducted on
a monthly basis during the school year.
77
Survey and Observational Tool
Development
Survey development included intensive dis-
cussions to identify research objectives and
design survey questions that address specific
outcomes. An eight-elementary school, field
pretest of an initial survey measured responses
to behavioral and psychometric survey items.
Analysis of the survey responses and evalua-
tion of individual teacher comments indicated
that the items measuring specific behaviors
would be most effective in measuring changes
that result from program implementation, and
that psychometric survey items derived from
previously validated surveys were insuffi-
ciently specific given the number of schools
used and the variability in how items can be
interpreted by respondents. For example, in
response to Adults at this school really care
about every student, teachers responded that
they can only report how they personally feel,
and that they didnt know which aspects of
care are involved in the statement.
Although it is preferable to use questions
from previously validated surveys so that
study results can be compared with those
reported in the literature, using poor measures
of the program being evaluated, regardless of
the historical value of the analytical tool, will
not provide an effective means of measuring
outcomes. As noted by Fowler (1995), there is
a large body of research in the social and med-
ical sciences that repeatedly incorporates poor
outcome measures because of their historical
significance; however, he points out that sci-
ence is best served when researchers use sur-
vey questions that have been systematically
evaluated using procedures that ensure that the
questions are effective. In addition, Fowler
comments that although multi-item surveys
can often improve measurement, 20- to 30-
item scales can be effective tools if the items
are chosen carefully.
Therefore, although the Capturing Kids
Hearts survey initially contained approxi-
mately 100 items in the field pretest, in order
to minimize respondent burden, the final sur-
78
vey contained 19 items. When developing sur-
vey tools evaluating factual events
(observable behaviors), key principles
involved in good survey design include (1)
asking questions that respondents can answer,
(2) clearly defining key terms so that respon-
dents are all answering the same question, and
(3) providing adequate context so that respon-
dents will see the value of answering the ques-
tions accurately (Fowler, 1995). In order to
follow these key principles, the final survey
and observational form only included items
measuring specific behavior.
Table 2 demonstrates how key outcomes
taught in the Capturing Kids Hearts process
correspond to items described in the taxon-
omy of variable outcomes developed by
Berkowitz and Bier (2005). The table further
links these outcomes to items within an
established psychometric survey (WestEd,
2008), and provides examples of how psy-
chometric survey items were converted to
measures of observable behaviors in the Cap-
turing Kids Hearts Survey or Observational
Form. For instance, to measure the leader-
ship skill of engagement (first item in Table
2) the more general terms used in the Cali-
fornia Healthy Kids Survey (supportive
and inviting) was converted to a specific
behavior that can be observed (the teacher is
at the door greeting students). Other behav-
ioral measures of prosocial skills are pro-
vided in Table 2.
Both the Capturing Kids Hearts survey
and observational form contain similar ques-
tions, but are specific to the person filling out
the tool (i.e. The teacher [is at the door] is
an example of the beginning of a sentence
used in the observational form whereas I
[am at the door] is an example of the corre-
sponding item used in the self-report sur-
veys). A review of these items by former
school administrators and teachers, current
teachers, and program trainers with knowl-
edge of the subject matter established the
content validity (Litwin, 1995) of the survey
and observational form.
Journal of Research in Character Education Vol. 9, No. 1, 2011
Teacher and Principal Implementation
Measures
The proximal outcomes of the training
(teacher and principal implementation) were
assessed using the Capturing Kids Hearts
Observational Form at the classroom level
through observation of 40 randomly selected
classrooms (using a web-based number ran-
domizer; 5 teachers per school), and at the
individual-level (all teachers) using the Cap-
turing Kids Hearts Survey. Each wave of
classroom-level observational data was aggre-
gated to represent a school-level construct
(Tseng & Seidman, 2007). A data triangulation
approach was used in that each wave of indi-
vidual-level (survey) data was also aggregated
and used to determine whether or not teachers
perspectives of how well they model Captur-
ing Kids Hearts behaviors correspond with
the implementation ratings obtained from
observing teachers in the classroom. School
implementation ratings were generated by
trained members of the program developers
research staff, each of whom had taught the
process and had been part of the evaluative
process for more than 5 years. Implementation
analyses were based upon two, on-site obser-
vational sessions for each school during the
school year. The first wave of observational
data (midyear data point) was obtained
approximately 3 to 5 months after teachers
attended Capturing Kids Hearts training, and
the second wave was obtained at the end of the
school year. The evaluation measured proxi-
mal outcomes in three areas: (1) principal sup-
port for the process (i.e., (s)he personally
modeled Capturing Kids Hearts skills and
established procedures for following up with
on-site mentors), (2) teacher implementation
of the Capturing Kids Hearts relational skills,
and (3) teacher implementation of specific
classroom management techniques. All three
areas were rated based upon a 3-point Lik-
ert-type scale with 0 indicating no implemen-
tation, 1 indicating partial implementation, and
2 indicating full implementation. Fractional
points were not used in the evaluation of indi-
Implementation of a School-Wide Adolescent Character Education and Prevention Program 79

TABLE 2
Relating Capturing Kids Hearts EXCEL Knowledge and Skills to a
Variable Outcome Taxonomy, Items From the California Healthy Kids Survey, and
Items From the Capturing Kids Hearts Observable Behaviors Survey and/or Implementation Form
Example Measures of
Observable Behaviors in the
Capturing Kids Hearts
EXCEL Knowledge Variable Outcome Taxonomy* Example Items From the Survey and Observation
& Skills (*Berkowitz & Bier, 2005) CA Healthy Kids Survey Form
E Learn Meet & Greet 2.2 Personal Morality This school is a supportive The teacher is at the door
skills (Leadership Skills 2.2.8) and inviting place for students greeting students as they
to learn. (6) enter the classroom (E1)
Model social skills2.4 Communicative
Competency Adults at this school treat The teacher uses a
Model professional (Communication Skills 2.4.1) every student with respect. welcoming tone of voice
skills (Attentive Listening 2.4.2) (39) (E2)
X Explore needs of 2.3 Proocial Behaviors This school gives all students For 3-5 minutes at the
others (Caring Concern 2.3.6) equal opportunity to beginning of class, the
(Sharing 2.3.12) participate in classroom teacher asks students to
Implement Good discussions or activities. (17) share (X6)
Things 2.4 Communicative
Competency Adults at this school
(Attentive Listening 2.4.2) acknowledge and pay The teacher asks
attention to students. (34) open-ended questions
(X7)
C Acquire effective 2.3 Prosocial Behaviors & The school encourages The teacher and students
listening skills Attitudes opportunities for students to created a Social Contract
(Teamwork and cooperation decide things like class that outlines (C9)
Build a Social 2.3.7) activities or rules. (16)
Contract
2.4 Communicative
Ask for feedback Competency
(Communication Skills 2.4.1)

2.7 Citizenship
(Democratic Values 2.7.1)
E Implement a Social 2.2 Personal Morality The school handles discipline The teacher reviews the
Contract (Respect 2.2.3) problems fairly. (27) Social Contract (Em15)

Develop 2.3 Prosocial Behaviors & The school effectively The teacher uses a question
self-managing skills Attitudes handles student discipline and approach to redirect
(Helping others 2.3.8) behavioral problems. (28) misbehavior (Em16)

Use nonverbal 4.4 Problem-Solving Skills
signals to manage (Conflict resolution strategies
behavior 4.4.5)
(Behavioral adjustment 4.4.3)
L Use meaningful 2.2 Personal Morality
ending to class (Leadership Skills 2.2.8)
Adults at this school treat all The teacher does not avoid
students fairly. (38) conflictaddresses ALL
misbehavior (Em17)
Adults at this school want The teacher ends class with
every student to do their best. a powerful message such
(35) as (L19)
vidual teachers. An aggregated implementa-
tion score (reported as a percentage of full
implementation) was calculated for each
school.
Student Acquisition of Prosocial Skills
Measures
Observational measures of students acqui-
sition of prosocial skills (distal outcomes) spe-
cific to the intervention were obtained during
the two on-site days for each school in which
teacher and principal implementation mea-
sures were acquired. Selected items from the
California Healthy Kids School Climate Sur-
vey (2008) were matched to specific behav-
iors/skills taught in the Campus by Design
model, and these behaviors/skills were
matched to the taxonomy of outcomes
described by Berkowitz and Bier (2005).
These outcomes were divided into subscales
that measured modeling of specific skills dem-
onstrating personal morality (sense of justice/
fairness, respect), prosocial behaviors (caring
concern for others, teamwork, helping others,
sharing), communicative competency (com-
munication skills, attentive listening), citizen-
ship (democratic values), and problem-solving
skills (consequential thinking, behavioral
adjustment, conflict resolution). An aggre-
gated prosocial skills outcome score was also
calculated, giving equal weight to each sub-
scale.
Data Collection
Schools were provided a prestudy data form
requesting demographic, academic, and
behavioral data for the 3 school years preced-
ing initiation of the study. Principals from each
of the schools provided the data sheet and sub-
mitted it to their respective leadership teams.
For Capturing Kids Hearts Campus by Design
(CKH CBD), a process that impacts school cli-
mate through strengthening teacher-student
connectedness, the BOCES and RCOE leader-
ship teams were primarily interested in mea-
suring differences in behavioral outcomes (i.e.
reduction of risk factors such as discipline
referrals and acquisition of protective factors
such as prosocial relational skills).
Stratification of Schools by Level of
Variance in Discipline Referral Data
Using discipline referral data as a measur-
able outcome of program implementation can
be problematic because the application of
referral rules and policies can vary greatly
between teachers within a school and between
schools within a district. Additionally, new
policies may be implemented in one or more of
the intervention or control schools involved in
a research study such that the number of refer-
rals reported at the end of the study is not
indicative of what actually occurred in the
school. For instance, a principal in a control
school could implement stricter discipline
rules during the year such that it would seem
that referrals increased relative to the previous
years data, but in fact, the increase would only
reflect changes in policies, not true increases in
disruptive student behaviors. Thus, although
discipline referral records continue to be used
to evaluate behavioral changes within schools
(Irvin, Tobin, Sprague, Sugai, & Vincent,
2004), the integrity of the data may be compro-
mised when consistency in the discipline refer-
ral system is not maintained (Wright & Dusek,
1998).
Sprague, Sugai, Horner, and Walker,
(1999) attempted to overcome potential vari-
ability in schools discipline referral systems
by selecting schools that had an established
discipline referral system and that were in the
initial stages of a plan to improve school-wide
discipline procedures. These two strategies
were utilized in the present study. In addition,
a stratification of schools strategy based
upon the variability in the discipline referral
data in the 3 years preceding the study was
implemented so that changes in referrals could
be detected if they occurred.
Stratification of schools to meet statistical
demands of research trials (Holtzapple, 2011)
is particularly critical to the success of studies
Implementation of a School-Wide Adolescent Character Education and Prevention Program
involving fewer than 10 schools. In the present
study, funding was available for only 8
schools; therefore, given the statistical diffi-
culties involved in measuring outcomes when
using fewer than 10 schools, the variability in
the number of discipline referrals in the 3 years
preceding the study was determined for each
school that applied to be a study participant.
The variability in the data ([standard deviation/
3-yr mean number of discipline referrals]
expressed as a percentage) ranged from 1% to
42% (see Appendix). Because previous studies
had demonstrated that the Capturing Kids
Hearts process can reduce discipline referrals
in schools 25% to 75%, and because school
leadership teams wanted to be able to detect
changes in this outcome, the 8 schools whose
school-level variability in discipline referral
data was less than 25% were chosen for the
study.
Statistical Analysis
Statistical power analyses (Holtzapple,
2009) were conducted using the Optimal
Design software (initially version 1.76 and
then the updated version 2.0) for cluster ran-
domized trials (Spybrook et al., 2006; Spy-
brook, Raudenbush, Liu, & Congdon, 2009).
The significance level () was held constant at
0.05 for all calculations. Univariate ANOVA
analyses were conducted using SPSS version
16.0 software.
RESULTS & DISCUSSION
Implementation Analyses
School-level prestudy, midyear, and end of
year implementation outcomes are highlighted
in Figure 1. The prestudy survey data provides
the baseline level of Capturing Kids Hearts
Campus by Design (CKH-CBD) prosocial
behaviors self-reported by school principals
and faculty for each of the eight schools. The
presurveys were used as a surrogate for
prestudy direct observation of behaviors
81
because scheduling conflicts prevented com-
pletion of classroom observations in the inter-
vention schools prior to the training. The pre-
and postsurvey (end of year) means (23.25
5.68 and 25.25 5.44, respectively; p
[two-tailed] = 0.6222) from the control schools
provides a test-retest reliability measure of the
survey instrument. The presurvey and
mid-year observation means (23.25 5.68 and
18.75 5.5, respectively; p [two-tailed] =
0.2803) in the control schools demonstrates
that the results from prestudy self-report sur-
veys are a reliable substitute for the prestudy
observational instrument. A sampling of 5 ran-
domly selected teachers from each school pro-
vides good information about faculty
follow-through and the pervasiveness of
implementation. Therefore, the presurvey
results served as the initial baseline for all
schools, whereas midyear, and end-of-year 1
data sets were obtained using direct observa-
tion of behaviors and skills.
Initially, intervention and control schools
demonstrated similar baseline levels of specific
behaviors and skills associated with the Cap-
turing Kids Hearts training. By the
end-of-year 1 evaluation, control schools
exhibited levels of CKH-CBD skills that were
similar to baseline levels; however, the inter-
vention schools exhibited increased, but vary-
ing levels of CKH-CBD skills, with
implementation ranging from 32% to 72% as
determined by observation of CKH-CBD skills
exhibited by teachers and principals. In order to
determine which factors affect program imple-
mentation and, ultimately, student outcomes,
we investigated: (1) the level of principal sup-
port of the intervention, (2) the relationship
between principal support and teacher imple-
mentation of the intervention in the classroom,
and (3) the relationship between teacher imple-
mentation and student acquisition of specific
prosocial skills.
Figure 2 demonstrates the end-of-year 1
level of principal support for the intervention
for all 8 schools as measured by direct obser-
vation of principal-initiated skills related to
Campus By Design. Note that even in the con-
82 Journal of Research in Character Education Vol. 9, No. 1, 2011

End of School Year:

70 CKH-CBD
Control () and Intervention (, ) Schools Implementation

Intervention
Percent CKH-CBD Implementation

60 schools that did

implement CKH-
CBD well
50

40

Intervention school
30

that did not
implement CKH-
CBD well
20
Control schools that

did not receive
10
CKH-CBD training

Pre-training Mid-year Post-training

FIGURE 1
Assessing Implementation Progress. Teacher and principal implementation of CKH-CBD skills were mea-
sured prior to CKH training, midyear, and at the end of the school year.

trol schools (represented by the bar graphs
located in the small box in Figure 2), the base-
line level of principal support/principal leader-
ship is not zero. Posttest evaluation of
principal support/principal leadership was
based upon evaluator-observed use of specific
behaviors and skills by the principals. Thus,
the levels of Campus By Design skills
observed in principals from control schools
would be skills that may be expected in indi-
viduals as a result of natural skills, college
training, or postcollegiate principal develop-
ment trainings whose objectives overlap those
of Capturing Kids Hearts Campus By Design.
Levels of these skills in principals from control
schools ranged from 10% to 33%.
Initially, the principals of the intervention
schools demonstrated a similar level of pro-
gram-specific skills; however, by the end of
the school year, the principals from 3 of the
intervention schools had implemented and
were modeling for their teaching faculty
between 72% and 87% of the skill set taught
by the intervention. In contrast, the principal
from the fourth intervention school (I4) imple-
mented less than 30% of the skills.
Given the importance of principal leader-
ship when implementing interventions, we
Implementation of a School-Wide Adolescent Character Education and Prevention Program 83

100

90
Percent Principal Implementation
of the CKH-CBD Process 80

70

60

50

40

30

20

10

0
C3 C2 C8 C6 I7 I1 I5 I4
Control () or Intervention () School

FIGURE 2
Posttest level of principal support for the CKH-CBD process. The level of principal support (implementation
of CKH-CBD skills) was determined using observational measures.

investigated how principal support for the
intervention is related to teacher implementa-
tion of CKH-CBD skills in the classroom. As
can be observed in Figure 3, there is a direct,
linear relationship between the level of support
provided by the principal and the degree to
which the process is implemented by teachers.
In the intervention schools, when principal
support is low (school I4), there is a corre-
spondingly low level of teacher implementa-
tion. In schools in which principal modeling of
the intervention skills is high (schools I1, I5,
and I7), there is a correspondingly high level
of teacher implementation of the interventions
skills in the classroom.
The variability in the level of implementa-
tion of the process in the classroom in inter-
vention schools suggests that there will be
similar variability in student outcomes. Figure
4 demonstrates that this is indeed the case. In
intervention school I4, low implementation of
the CKH-CBD skills by teachers resulted in a
low level of student acquisition of specific pro-
social skills.
Analyses of Student Outcomes:
Withdrawal for Nonadherence Versus
Intention-to-Treat
The information in the preceding figures
demonstrate that one of the intervention
schools implemented less than 35% of the
skills taught in the Campus By Design model
and may therefore be considered a nonadher-
ing participant. A withdrawal for nonadher-
ence approach was considered for this
intervention school exhibiting < 60% imple-
mentation of the intervention because doing so
84 Journal of Research in Character Education Vol. 9, No. 1, 2011

80
I7
I7
70
Percent Teacher Implementation

I5
I5
of the CKH-CBD Process

60
I1
I1
50

40

I4
I4
30

20
C8
C8
10 C6 C3
C3
C6 C2
C2
0
0 10 20 30 40 50 60 70 80 90 100

Percent Principal Implementation

of the CKH-CBD Process

FIGURE 3
Teacher implementation versus principal support. The data demonstrates that there is a direct, linear relation-
ship between the percent of teacher implementation of CKH-CBD skills in the classroom versus the percent of
principal support for the intervention (R2=0.9398).

would allow the trial to provide a comparison
of student outcomes resulting from optimal
intervention versus control condition. Includ-
ing the results from the nonadhering school
would contaminate the intervention group
and reduce the observed outcomes of the trial,
thus making the intervention appear to be less
effective. In clinical trials, it has been pro-
posed that if the rules for withdrawing partici-
pants are identified prior to study initiation,
then it is legitimate to remove nonadhering
participants (Sackett & Gent, 1979). However,
Friedman, Furberg, and DeMets (1998) argue
that an intention-to-treat approach should be
used in research studies because adherence, or
lack thereof, is itself a response to the interven-
tion. They state that the potential for bias can-
not be avoided when nonadhering participants
are dropped from the trial. In addition,
Chaudhry and McDermott (2008) point out
that nonadherence is not only a problem in
research studies, but is also a major barrier to
obtaining desirable outcomes in nontrial sce-
narios. Low adherence to the intervention in
trials of educational interventions may indicate
that widespread use of the program may not be
feasible unless developers address the reasons
for non-compliance (i.e., participant knowl-
edge of what is required is lacking or fol-
low-up strategies are ineffective). For the
following evaluation of student outcomes, an
intention-to-treat approach was used; there-
fore, the outcomes of all eight schools were
included in the analyses.
Implementation of a School-Wide Adolescent Character Education and Prevention Program 85

80

of CKH-CBD Pro-Social Skills 70 I7

Percent Student Acquisition

60
I1 I5
50

40

I4
30

20
C6
10 C3
C8
C2
0
0 10 20 30 40 50 60 70 80

Percent Teacher Implementation

of the CKH-CBD Process

FIGURE 4
Student acquisition of prosocial skills versus teacher implementation. The data demonstrates that there is a
direct, linear relationship between the percent of student acquisition of prosocial skills versus the percent of
teacher implementation of CKH-CBD skills in the classroom (R2 = 0.9229).

Student Prosocial Outcomes As can be seen in Figure 5, prosocial out-

Student prosocial outcomes were classified
according to the taxonomy of outcomes pre-
sented by Berkowitz and Bier (2005) in a
research guide for educators. Direct observa-
tion of behaviors was used to detect changes
in personal morality (PM; respect, sense of
justice and fairness), prosocial behaviors
(Pr-S; caring concern for others, teamwork,
helping others, sharing), communicative
competencies (CC; communication skills,
attentive listening), citizenship (CZ; demo-
cratic values), and problem solving (PS; con-
sequential thinking, behavioral adjustment,
conflict resolution).
comes increased from 24% (pretest) to 57%
(posttest) in the intervention schools whereas
they decreased from 23% (prestudy) to 12%
(poststudy) in the control schools. The p - val-
ues for these outcomes were 0.8254 at pretest
(indicating no significant difference between
the control and intervention groups at pretest)
and 0.0070 at posttest (indicating statistically
significant differences between the two
groups). These prosocial outcomes were
divided into subcategories, and the scores for
these subcategory outcomes are also provided
in Figure 5. (P -values: PM = 0.0167; PrS =
0.0276; CC = 0.0103; CZ = 0.0185; PS =
0.0472).
86 Journal of Research in Character Education Vol. 9, No. 1, 2011

Percent Acquisition of Pro-Social Outcomes (Protective Factors)

Student Acquisition of Pro-Social
Pro Outcomes
Pro-Social Outcomes

Composite Sub-Category
100 Scores for Scores for PMPersonal Morality
Pro-Social Pro-Social
sense of justice/fairness
Outcomes Outcomes
90 respect

Pr-SPro-Social Behaviors
80 caring concern for others
teamwork
Control ( ) vs. Intervention ( ) Schools

helping others
70 sharing

60 CCCommunicative Competencies
communication skills
attentive listening
50
CZCitizenship
40 democratic values

PSProblem Solving
30 consequential thinking
behavioral adjustment
20 conflict resolution

10

0
Pre- Post- PM Pr-S CC CZ PS
Test Test Post-Test

FIGURE 5
Pretest and posttest composite scores for prosocial outcomes for students in the control and intervention
schools are provided on the left-hand side of the figure. The posttest subcategory scores for personal morality
(PM), prosocial behavior (Pr-S), communicative competencies (CC), citizenship (CZ), and problem solving
(PS) are provided on the right-hand side of the figure.

Discipline Referral Outcomes CONCLUSIONS

All schools were asked to provide the num- The Centers for Disease Control and Preven-
ber of discipline referrals for the 3 years pre- tion (2009) provides strategies that will help
ceding the study as well as for the 2008-2009 create an environment that facilitates student
school year. Discipline referral data was ana- health and academic achievement. The strate-
lyzed by univariate ANOVA. The effect size, gies addressed by the Capturing Kids
reported as the standardized mean difference HeartsCampus by Design model include (1)
(Hedges g; df = 7) was computed to be -2.1 providing students with social skills necessary
(reduction in discipline referrals) with p = to be actively engaged in school, (2) using
0.0189. This outcome represents an average effective classroom management methods to
11% increase in discipline referrals in the 4 foster a positive learning environment, (3) pro-
control schools; in contrast, a 22% decrease in viding professional development for teachers
referrals occurred in the 4 intervention to enable them to meet the emotional needs of
schools. adolescents, and (4) creating trusting and car-
Implementation of a School-Wide Adolescent Character Education and Prevention Program 87

APPENDIX
Mean Campus Enrollment and Number of Discipline Referrals
(During the 3 Years Preceding the Study) for Schools Originally Interested in Participating in the Study
SD/Mean #
Discipline
Mean Campus Mean # Discipline Referrals Variability
Condition School # Enrollment Referrals ( SD) (expressed as %) Stratum

Intervention 1 1,112 1,200 ( 25) 13%

Control 2 1,347 1,185 ( 21) 11%
Control 3 1,540 1,785 ( 20) 3%
Intervention 4 1,373 1,739 ( 149) 9% < 25%
Intervention 5 1,429 1,647 ( 75) 12%
Control 6 1,248 1,239 ( 13) 1%
Intervention 7 2,106 8,236 ( 1318) 16%
Control 8 3,195 8,378 ( 1214) 14%
9 2,350 2,692 ( 985) 37% 25-40%
10 1,594 1,556 ( 635) 41%
11 2,834 4,028 ( 1707) 42% > 40%

ing relationships that promote open communi-
cation. Implementing the EXCEL Leadership
Model through the Capturing Kids Hearts
Campus by Design process results in a culture
in which students and staff model leadership
qualities, show respect for others, share and
celebrate each others successes, demonstrate
caring concern for others, communicate effec-
tively, listen attentively to others, have a sense
of justice and fairness, help others, resolve
conflicts, and work together as a team.
The limitations of the study are directly
linked to the statistical constraints placed on
school districts when attempting to conduct a
randomized, controlled trial using schools as
the unit of randomization. In order to minimize
costs, (1) a limited number of schools were
used, and as a result of the associated statistical
constraints, (2) principals had to be willing to
participate, (3) only schools with stable base-
line data were eligible for the study, and (4)
schools were recruited from only two states,
New York and California. The results do not
necessarily demonstrate outcomes that would
occur in schools in which principals are disin-
clined to participate or for which the baseline
data is unstable. Next steps in research include
measuring outcomes in elementary and middle
school students, and working with a different
unit of analysis (using both intervention and
control participants within the same school)
despite the possibility of a spillover effect
between participants.
When implemented with fidelity, the Cap-
turing Kids Hearts Campus by Design model
increases prosocial behaviors (protective fac-
tors) in students and decreases the number of
discipline referrals (risk factors) in interven-
tion schools. Continued support for the process
will help sustain the behaviors that enhance the
learning environment and equip students for
success in school and after graduation.
Acknowledgment: The author is grateful
for invaluable research discussions with Drs.
Thomas Lickona, Matt Davidson, and Vlad
Khmelkov (SUNY-Cortland); Vic Battistich
(late) and Marvin Berkowitz (U. Missouri-St.
Louis); Jessaca Spybrook (U. Western Michi-
gan); and Victor Willson (Texas A&M Uni-
versity).
88
REFERENCES
Bandura, A. (1986). Social foundations of thought
and action: A social cognitive theory. Princeton,
NJ: Prentice-Hall.
Berends, M., Bodilly, S., & Kirby, S. N. (2002).
Facing the challenges of whole-school reform:
New American schools after a decade. Santa
Monica, CA: RAND.
Berkowitz, M., & Bier, M. (2005). What works in
character education: A research driven guide
for educators (Variable Outcome Taxonomy,
Table 4). Retrieved from http://education.csuf-
resno.edu/bonnercenter/documents/
What_Works.pdf
Castro, V., Johnson, M. B., & Smith, R. (2008).
Self-reported resilient behaviors of seventh and
eighth grade students enrolled in an emotional
intelligence based program. Journal of School
Counseling, 6(27). Retrieved from http://
www.jsc.montana.edu/articles/v6n27.pdf
Centers for Disease Control and Prevention. (2009).
School connectedness: Strategies for increasing
protective factors among youth. Atlanta, GA:
U.S. Department of Health and Human Services.
Retrieved from http://www.ok.gov/
strongandhealthy/documents/connectedness.pdf
Character Education Partnership. (2010). Develop-
ing and assessing school culture: A new level of
accountability for schools. Washington DC:
Character Education Partnership. Retrieved
from http://www.character.org/uploads/PDFs/
White_Papers/
DevelopingandAssessingSchoolCulture_
Final.pdf
Chaudhry, H. J., & McDermott, B. (2008). Recog-
nizing and improving patient nonadherence to
statin therapy. Current Atheroschlerosis
Reports, 10(1),19-24.
Cirillo-Teverbaugh, K. J., & Colwell, B. (1993).
Effects of a 10-week social-cognitive group
intervention on selected psychosocial attributes
and interpersonal effectiveness of high school
students. Unpublished manuscript, Texas A&M
University, College Station, TX.
Cirillo-Teverbaugh, K. J. (1994). Adolescent loneli-
ness: Implications and intervention strategies.
Eta Sigma Gamma Student Monograph Series,
12(1), 1-10.
Danaher, A. C. (2006). Character education: The
impact of a teen leadership program. Kingsville,
TX: A&M University. Dissertation. Powerpoint
Journal of Research in Character Education Vol. 9, No. 1, 2011
presentation retrieved from http://www
.powershow.com/
view.php?id=P1252428677PgvnI&t=
Character+Educa-
tion%3A+The+Impact+of+a+Teen+
Leadership+Program+on+Student+
Connectedness
Domitrovich, C. E., & Greenberg, M.T. (2000). The
study of implementation: Current findings from
effective programs that prevent mental disorders
in school-aged children. Journal of Educational
and Psychological Consultation, 11(2),
193-221.
Durlack, J. A., & Wells, A. M. (1998). Evaluation of
indicated preventive intervention (secondary
prevention) mental health programs for children
and adolescents. American Journal of Commu-
nity Psychology, 26, 775-802.
Flippen Group. (2006). Report to the Texas Educa-
tion Agency. Retrieved from http://www
.flippengroup.com/pdf/funding/
TEA8CaseStudies.pdf
Flippen Group. (2009). Approved programs.
Retrieved from http://www.flippengroup.com/
education/eduaprovc.html
Fowler, F. J. (1995). Improving survey questions:
Design and evaluation (Applied Social Research
Methods Series, Vol. 38). Thousand Oaks, CA:
SAGE.
Friedman, L. M., Furberg, C. D., & DeMets, D.L.
(1998). Fundamentals of clinical trials (3rd ed.).
New York: Springer.
Glanz, K., Rimer, B. K., & Lewis, F. M. (2002).
Health behavior and health education: Theory,
research and practice. San Francisco: Wiley.
Glennan, T. K., Jr. (1998). New American schools
after six years. Santa Monica, CA: RAND.
Greenberg, M. T., Domitrovich, C. E., Graczyk, P.
A., & Zins, J. E. (2005). The study of implemen-
tation in school-based preventive interventions:
Theory, research, and practice. Draft of final
report for U.S. Department of Health and Human
Services, Substance Abuse and Mental Health
Services Administration, Center for Mental
Health Services.
Gresham, F. M., Gansle, K. A., Noell, G. H., &
Cohen, S. (1993). Treatment integrity of
school-based behavioral intervention studies:
1980-1990. School Psychology Review, 22,
254-272.
Gresham, F. M., & OShaughnessy, T. E. (2002).
Interventions for children with or at risk for
Implementation of a School-Wide Adolescent Character Education and Prevention Program
emotional and behavioral disorders. Boston,
MA: Allya and Bacon.
Hallfors, D., & Godette, D. (2002). Will the Princi-
ples of effectiveness improve prevention prac-
tice? Early findings from a diffusion study.
Health Education Research, 17(4), 461-470.
Holtzapple, C. K. (2009). Controlling for baseline
covariates to improve the statistical power of
efficacy trials involving whole school prevention
programs. 17th Annual Meeting of the Society
for Prevention Research. Retrieved from http://
www.preventionscience.org/
SPR_09_program_COMPLETE.pdf (abstract
#224, pg 63)
Holtzapple, C. K. (2011). Measuring behavioral
outcomes associated with comprehensive char-
acter education programs: A practical approach
to using fewer schools in school randomized
controlled trials. Journal of Research in Charac-
ter Education, 9, 57-69.
Irvin, K. L., Tobin, T. J., Sprague, J. R., Sugai, G.,
& Vincent, C. G. (2004). Using office discipline
referral data to evaluate school-wide discipline
and violence prevention interventions. Oregon
School Study Council (OSSC) Bulletin, 42(2),
1-18.
Kam, C. -M., Greenberg, M. T., & Walls, C. T.
(2003). Examining the role of implementation
quality in school-based prevention using the
PATHS curriculum. Prevention Science, 4(1),
55-63.
Lickona, T., & Davidson, M. (2005). Smart & good
high schools: Integrating excellence and ethics
for success in school, work and beyond. Cort-
land, NY: Center for the 4th and 5th Rs (Respect
& Responsibility). Washington, DC: Character
Education Partnership.
Litwin, M. S. (1995). How to measure survey reli-
ability and validity. San Francisco, CA: SAGE.
Marzano, R. J. (with Marzano, J. S., & Pickering, D.
J.). (2003). Classroom management that works.
Alexandria, VA: ASCD.
Norman, P. (1991) Social learning theory and the
predication of attendance at screening. Psychol-
ogy and Health, 5, 231-239.
Office of Juvenile Justice and Delinquency Preven-
tion. (2004). OJJDP Model Programs Guide,
version 3.0. Retrieved from http://www2
.dsgonline.com/mpg/prevention.aspx?
continuum=prevention
Perry, C. L., Baranowski, T., & Parcel, G. S. &
(1991). How individuals, environments and
health behavior interact: Social Learning The-
89
ory. In K. Glanz, F. Marcus Lewis, & B.Rimer
(Eds.), Health behavior and health education.
San Francisco, CA: Josey-Bass.
Resnick, M. D., Bearman, P. S., Blum, R. W., Bau-
man, K. E., Harris, K .M., Johnes, J., Tabor, J.,
Beuhring, T., Sieving R. E., Shew, M., Ireland,
M., Bearinger, L. H., & Udry, J. R. (1997). Pro-
tecting adolescents from harm. Findings from
the National Longitudinal Study on Adolescent
Health. JAMA, 278(10), 823-832.
Resnick, M. D., Ireland, M., & Borowsky, I. (2004).
Youth violence perpetration: What protects?
What predicts? Findings from the National Lon-
gitudinal Study of Adolescent Health. J. Adoles-
cent Health, 35, 424e1-424e10.
Sackett, D. L., & Gent, M. (1979). Controversy in
counting and attributing events in clinical trials.
N. Engl. J. Med. 301:1410-1412.
Sherwood, R. (2003). It all began with a handshake.
The Effective Schools Project Journal, 9, 6-11.
Sprague, J. R., Sugai, G., Horner, R., & Walker,
H.M. (1999). Using office discipline referral
data to evaluate school-wide discipline and vio-
lence prevention interventions. Oregon School
Study Council (OSSC) Bulletin, 42(2), 1-18.
Spybrook, J., Raudenbush, S. W., Liu, X., & Cong-
don, R. (2006). Optimal design for longitudinal
and multilevel research: Documentation for the
Optimal Design software.
Spybrook, J., Raudenbush, S.W., Liu, X., & Cong-
don, R. (2009). Optimal design for longitudinal
and multilevel research: Documentation for the
Optimal Design(version 2.0) software.
Retrieved from http://www
.wtgrantfoundation.org/resources/overview/
research_tools/research_tools
St. Pierre, T. L., & Kaltreider, D. L. (2004). Tales of
refusal, adoption, and maintenance: Evi-
dence-based substance abuse prevention via
school-extension collaborations. Am. J. Evalua-
tion, 25(4), 479-491.
Tseng, V., & Seidman, E. (2007). A systems frame-
work for understanding social settings. Am. J.
Community Psychol, 39, 217-228.
U.S. Department of Education (2009). Smaller
Learning Communities (SLC) Awards Database.
Retrieved from http://slcprogram.ed.gov/
cgi-bin/mysql/
slcawards.cgi?showds=s&showstructure=
&showstrategy=&searchss=&showdemopercent
=&showdemo=&showstates=%&andor=
AND&searchforschoolname=&andorprofile=
AND&searchprofile=&showcohort=&sortby=
90
state&resultsperpage=200&startrecord=
1401&showlocale=&showrelatedschools=&l=
showlist&sr=
Wang, M. C., Haertel, G. D., & Walberg, H. J.
(1993). Toward a knowledge base for school
learning. Review of Educational Research,
63(3), 249-294.
WestEd. (2008). California Healthy Kids Survey.
Retrieved from http://chks.wested/org/
resources/chks-hs-core.pdf
What Works Clearinghouse. (2007). Character edu-
cation: Overview. Retrieved from http://
ies.ed.gov/ncee/wwc/reports/
character_education/topic/
Wright, J. A., & Dusek, J. B. (1998). Research into
practice; Compiling school base rates for disrup-
Journal of Research in Character Education Vol. 9, No. 1, 2011
tive behaviors from student disciplinary referral
data. School Psychology Review, 27(1),
138-147.
Zhang, Yu., Shkolnik, J., & Fashola, O. (2005).
Evaluating the implementation of comprehen-
sive school reform and its impact on growth in
student achievement. Report retrieved from the
American Educational Research Association
website. Retrieved from http://www.air.org/
news/documents/
AERA2005Implementation%20of%20CSR-
Impact%20on%20Student%20Achievement.pdf
Copyright of Journal of Research in Character Education is the property of Information Age Publishing and its
content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's
express written permission. However, users may print, download, or email articles for individual use.