Background
Evaluating the
health-promoting
school
Alysoun Moon
The author
Alysoun Moon is a Senior Research Fellow, Health Care
Research Unit, University of Southampton, Southampton,
UK.
Keywords
Schools, Health, Curricula
Abstract
Examines some of the constraints on the effective
evaluation of the health-promoting school, particularly
where a whole-school approach is used, and outcomes
are not behaviour-specific. Considers the components of a
successful school health intervention, and identifies some
of the difficulties in achieving comprehensive evaluation.
Explores the advantages of randomisation and controlled
studies and identifies some limitations of these
approaches, when the main aims are to examine process
and non-specific behavioural outcomes. Identifies some
common errors that have been made in school-based
evaluations in the past and argues that the use of
multiple methods of evaluation are necessary for a multi-
factorial whole-school approach to teaching and learning
about health. Discusses the recommendations for future
evaluation by the WHO European Working Group on
Health Promotion Evaluation.
Electronic access
The current issue and full text archive of this journal is
available at
http://www.emerald-library.com
Health Education
Volume 100 . Number 6 . 2000 . pp. 237241
# MCB University Press . ISSN 0965-4283
237
The strong move towards evidence-based
health education and promotion in recent
years, the focus on quality standards, and the
need to justify the use of scarce resources,
have made monitoring and evaluation an
essential part of any health-related
programme in schools. Questions about the
impact of a programme, its effectiveness, and
resultant changes in knowledge, attitudes and
behaviour have to be answered. There are a
number of difficulties, however, particularly
when programmes encompass a whole school
approach and are not focused and behaviour
specific (Moon et al., 1999a).
The components of an effective
intervention
In recent years, the concept of health has been
broadened, from a focus on specific individual
behaviours in a personal and social vacuum,
to a holistic one which incorporates lifestyle
and examines the influences of home, school,
the media, the community and the workplace.
This has meant that the process of evaluation
has become complex. A healthy school
approach includes and involves teachers,
support staff, parents, visitors to the school
and the wider community, as well as the
pupils. There is a need for multiple
interventions and approaches when trying to
bring about behavioural, environmental or
social health-related change.
Allensworth (1993) carried out an
assessment of the state of health education
and promotion in the USA in the early 1990s.
Following this review she noted the increasing
complexity of health education and its many
elements that need to be addressed
(Allensworth, 1994), and identified a number
of components of an effective school-based
intervention. These include:
. the use of multiple theories and models
when planning interventions;
. a focus on priority health behaviours;
. an expanded curriculum, to include, for
example, food services, and the
community;
. the use of multiple strategies in
addressing ``problem'' behaviour;
Received February 2000
Revised version accepted June 2000
 Evaluating the health-promoting school
Alysoun Moon
. co-ordination of school and community
health promotion activities;
. co-ordination of whole school
programme through all subjects;
. promotion of active pupil participation in
lessons, and the use of active learning
methods;
. a focus on the development of life skills;
. a wider view of all aspects of school life,
such as developing a caring, nurturing
environment; and
. close co-operation with parents, carers
and, where possible, whole families.
It is clear that any extensive evaluative studies
of school-based health promotion,
particularly of the health-promoting school
and its effectiveness in changing school
management, policy and practice and pupil
behaviours, are likely to include a range of
methods and approaches (Elder, 1991;
Bremberg 1991; Moon, 1999b).
Some difficulties in evaluating
health-promoting schools
The need to monitor and assess each
component highlights the importance of
employing a variety of research methods and
approaches. Each intervention approach and
its implementation will need to be dealt with
separately and variables excluded where
possible. These variables may include
extraneous influences that will be hard to
identify and measure but which can skew the
results. The media, for example, may be
responsible for having a powerful impact on
attitudes through a programme screened at
the time of the intervention. These attitudes
may then be reinforced by the intervention
but cannot be said to be a specific outcome of
the intervention.
Funding bodies generally focus on
behavioural change as a measure of
effectiveness, and usually over a period which
is short. This demonstrates a lack of
understanding about what is involved in
school-based health promotion and of the
difficulties of changing behaviour through one,
or even many, interventions. Short-term
follow-up is inadequate to detect medium to
long-term, sustained behaviour change. In any
case, the diverse nature of many health-
promoting school projects cannot always be
evaluated by looking only at outcomes, as some
projects will not have measurable outcomes.
238
Health Education
Volume 100 . Number 6 . 2000 . 237241
Randomised controlled trials are still
thought by many to be the best research
design, because they avoid selection bias, and
distribute confounders by chance. There have
been a number of school-based evaluation
studies in which different schools have been
randomised to different conditions. The
studies have been critically appraised
subsequently and judged to be sound
methodologically. Nutbeam and Smith (1991)
suggest that ideally, youth health evaluations
should follow a basic randomised control,
experimental design. This will involve:
. pre-test studies to establish baseline
measurements;
. the use of a representative sample of the
target population;
. the random assignment of subjects to
intervention and control groups;
. the use of a clearly designed intervention;
and
. the use post-test studies to identify
change from baseline measurements.
However, as Nutbeam and Smith (1991)
acknowledge, it is impractical, and possibly
unethical, to allocate children to control and
intervention groups and recognise that school-
based research can be very difficult. There are
so many other influences on the health of
young people that they cannot always be
accounted for. Young people may spend a
considerable part of their time in school, but
they spend much more within their own home
background with family, friends and others.
Even carefully matched schools that are close
together can be contaminated, for example,
where groups from each school mix with each
other socially and share what is happening in
the intervention.
The complex nature of schools means that
randomised controlled trials are not always
appropriate or manageable, logistically and
ethically, in school settings. Random allocation
is not always possible, particularly in a study
where schools volunteer to take part in the
intervention. The fact that intervention schools
are volunteers introduces a bias, and may be
indicative of an immediate difference between
them and any controls before the study starts.
Where randomisation is not possible, an
experimental, non-randomised controlled
design is likely to be the best alternative. But
the use of controls can also be problematic if
they too are volunteers, again introducing a
form of selection bias. Furthermore, the
 Evaluating the health-promoting school
Alysoun Moon
feasibility of the randomisation process can be
a major problem: it requires large samples and
this may be difficult to achieve.
When it is not possible to allocate individual
students to intervention or control groups, for
example where there is a whole school
intervention, the unit of analysis becomes the
school. This can cause problems because of
lack of power to detect change unless there is
a large number of schools in the sample.
Type I, II and III errors
Some of these difficulties have resulted in a
number of statistical errors that commonly
occur in school-based evaluations. They
include Type I and Type II errors (Tones and
Tilford, 1990; Hansen, 1992). A Type I error
happens when a study falsely finds differences
in results or outcomes statistically that do not
in fact exist. A Type II error occurs when a
study fails to find differences statistically that
are actually there. In his review of abuse
prevention programmes, Hansen (1992)
highlights the fact that while most studies now
seek to eliminate Type I errors, not enough
attention has been given to Type II errors. He
suggests that the conditions that predispose to
Type II errors include:
. Insufficient statistical power to detect
differences that may exist (because of
insufficient numbers of subjects by design
or attrition or from insufficiently strong
programme impact). An example of this
might be using the school as the unit of
analysis, where a large number of schools
would be required to ensure validity.
. Control groups that for some reason do
not exhibit sufficient change in behaviour
to make any differences detectable.
. Pre-test non-equivalence that biases
against the intervention group. One
example might be using frequent school
absentees as a control group, a factor that
makes them different from the rest of the
school population from the start and
highly likely to demonstrate different
health-related behaviour than others.
. Contamination of experimental
manipulations through the adoption of
alternative programme strategies by
control groups. Pupils in the drug
education control group, for example,
may be given alternative work to the
intervention that is still related to drug
239
Health Education
Volume 100 . Number 6 . 2000 . 237241
prevention in some way and may result in
changes in their drug related behaviour.
The danger of such errors occurring
emphasises the need to involve a statistician
early at the planning stage of an evaluation to
ensure that the risks of such errors are minimal.
Tones (1996) has drawn attention to the
dangers of Type III errors. A Type III error
occurs when a health promotion intervention is
judged to be ineffective when it is actually the
intervention itself which is grossly deficient.
This emphasises the need to pay attention to
the components of an effective intervention.
Beyond outcome measures
In the 1970s, Kreuter and Green (1978)
recommended that evaluation should be
carried out at three levels, namely process
evaluation, precursor evaluation and then,
where appropriate, outcome evaluation. This
multi-level model is particularly useful in the
context of the health-promoting school, the
holistic nature of which touches on every
aspect of school life, including ethos and
environment, and thus demands that
evaluation should be multi-faceted.
Process or formative evaluation, which
measures the activities of the programme, its
quality and who it is reaching, is vital because,
without it, it is impossible to identify which
parts of a programme contributed towards
any successful outcomes. Precursor
evaluation refers to the evaluation of criteria
which theoretically, or by previous empirical
study, have a high probability of affecting
health outcomes. Precursor evaluations might
include cost effectiveness studies and
evaluation of the immediate impact of the
intervention on specific knowledge and skills.
Kreuter and Green caution that, unless steps
are taken to clarify the specific immediate
function of school health education,
programmes might be judged on outcome
measures that are inappropriate and
unrealistic.
Tones (1996) states that the health-
promoting school ``. . . should be judged
primarily by its contribution to such health
promotion goals as equity and empowerment
or by the success of initiatives contributing
to such long term goals''. He then goes on to
argue that, rather than focusing on outcome
measures to assess output following an
 Evaluating the health-promoting school
Alysoun Moon
intervention, intermediate measures of
effectiveness and efficiency are needed. These
might include the acquisition of knowledge
and understanding, evidence of successful
values-clarification, the efficient performance
of health skills, the enhancement of self-
esteem and changes in self-efficacy beliefs.
Nutbeam and Smith (1991) concluded that
only outcomes-related, empirical,
experimental research is perceived as having
high status, and that the value of process-
related research, so called ``soft'' research, is
devalued. ``The paucity of literature on
process evaluation in health education
research is testimony to the fact that it is not
yet taken seriously by researchers''. Since
these words were written, the recognition of
process evaluation, as well as outcomes
measures, as particularly suited to school
settings, has grown (Baric, 1991; Denman,
1994; Parsons et al., 1996). Where there are
many process measures of effectiveness, it is
vital to assess which parts worked and why. It
is very hard, nonetheless, to isolate input,
process change and outcome as three different
areas for examination when there are multiple
``ins'' and multiple ``outs''. In 1996, Tones
(1996) still felt that there remained a lack of
formative or process evaluation, which is
necessary to determine which aspects of an
intervention actually worked and which had
no impact and were a waste of time and
money. As Tones comments, the simple
application of a ``who'' did ``what'' with
``whom''; ``where'' did it happen and ``why'';
``how did it work?'' checklist may be all that is
required.
In practice, outcome evaluation and process
evaluation are not mutually exclusive. There
are a number of recorded studies in which the
acceptability of the intervention to the study
group has been assessed because of the effect
this can have on the outcome of the
intervention (DiClemente and Wingwood,
1995; Ploem and Byers, 1997). In a recent
evaluation of a healthy school award scheme
in the South of England (Moon, 1999b), the
findings of process evaluation carried out
through focus group interviews with pupils
were triangulated with the results of a
quantitative pupil health-related behaviour
questionnaire to identify similarities between
the two. The use of both process and outcome
measures in evaluating a health education
intervention can enhance the study and give
greater insight and validity to the results.
240
Health Education
Volume 100 . Number 6 . 2000 . 237241
What of the future?
A major concern for the future is the funding
of health promotion evaluations. The growing
awareness amongst teachers and researchers
of the components of an effective intervention
and of the need for a range of research
methods and approaches, particularly when
evaluating issues surrounding the health-
promoting school, is not always shared by
funding bodies, which are still concerned
primarily with behaviour change. Process
evaluation takes time, even when being
carried out by teachers within a school, and
will need to be costed. Without adequate
funding, practitioners will remain uncertain as
to what works or does not, and how best to
move forward. Inadequate and inappropriate
interventions, unsupported by research, will
continue to be used and there will be many
more lost opportunities to influence positively
the health and well being of all members of
the school community, including those in its
environs.
A WHO European Working Group on
Health Promotion Evaluation that was
composed of distinguished, international
health promotion specialists, met in April,
1998. They identified four core features of
approaches appropriate for the evaluation of
health promotion initiatives:
(1) the active participation of all key players;
(2) the use of multiple methods;
(3) evaluations which should enhance the
capacity of individuals, communities,
organisations and governments to address
health promotion concerns; and
(4) evaluations that accommodate the
complex nature of health promotion
interventions and their long-term impact.
They drew up the following recommendations
for policymakers, applicable to schools, stating
that they should:
. encourage the adoption of participatory
approaches to evaluation that provide
meaningful opportunities for involvement
by all of those with a direct interest in
health promotion initiatives;
. require that a minimum of 10 per cent of
the total financial resources for a health
promotion initiative be allocated to
evaluation;
. ensure that a mixture of process and
outcome information is used to evaluate
all health promotion initiatives;
 Evaluating the health-promoting school
Alysoun Moon
. support the use of multiple methods to
evaluate health promotion initiatives;
. support further research into the
development of appropriate approaches
to evaluating health promotion initiatives;
. support the establishment of a training
and education infrastructure to develop
expertise in the evaluation of health
promotion initiatives; and
. create and support opportunities for
sharing information on evaluation
methods used in health promotion
through conferences, workshops,
networks and other means.
These recommendations are particularly
relevant to the health-promoting school in
which the use of participatory approaches and
multiple methods are necessary to ensure that
all aspects of the concept are assessed and
evaluated thoroughly. There is indeed a need
for further research into the development of
appropriate approaches to evaluating health
promotion initiatives. The calls for further
training in evaluation techniques and
opportunities to share expertise, if acted
upon, can only benefit future evaluation
studies of the health-promoting school and
make the whole process easier for those
involved. Finally, the need for dedicated
funding for evaluation, to be a built-in part of
every grant for a new intervention, is not only
essential but will help to ensure that evidence-
based health education and promotion are the
foundation of a health-promoting school.
References and further reading
Allensworth, D.D. (1993), ``Health education: state of
the art'', Journal of School Health, Vol. 63 No. 1,
pp. 14-20.
Allensworth, D.D. (1994), ``The research base for
innovative practice in school health education at
secondary level'', Journal of School Health, Vol. 64
No. 5, pp. 180-7.
Baric, L. (1991), ``Health-promoting schools: evaluation
and auditing'', Journal of the Institute of Health
Education, Vol. 29 No. 4, pp. 114-20.
Bremberg, S. (1991), ``Does school health education affect
the health of students? A literature review'', in
Nutbeam, D., Hagland, B., Farley, P. and Tillgren, P.
(Eds), Youth Health Promotion: From Theory to
Practice in School and Community, Forbes
Publications.
Denman, S. (1994), ``Do schools provide an opportunity
for meeting the Health of the Nation targets?'',
Journal of Public Health Medicine, Vol. 16 No. 2,
pp. 219-24.
241
Health Education
Volume 100 . Number 6 . 2000 . 237241
DiClemente, R.J. and Wingwood, G.M. (1995), ``A
randomised controlled trial of an HIV sexual
risk-reduction intervention for young
African-American women'', Journal of the
American Medical Association, Vol. 274 No. 16.
pp. 1271-6.
Elder, J.P. (1991), ``From experimentation to
dissemination: strategies for maximising the impact
and spread of school health education'', in
Nutbeam, D., Hagland, B., Farley, P. and Tillgren, P.
(Eds), Youth Health Promotion: From Theory to
Practice in School and Community, Forbes
Publications.
Flay, B.R. (1985), ``A review of 27 school-based studies of
psychological approaches to smoking prevention'',
Health Psychology, Vol. 4 No. 5, pp. 449-88.
Goldstein, H., Rasbash, J., Plewis, I., Draper, D.,
Browne, W., Yang, M., Woodhouse, G. and
Healy, M. (1998), A User's Guide to MlwiN, Institute
of Education, University of London, London.
Hansen, W.B. (1992), ``School-based substance abuse
prevention: a review of the state of the art in
curriculum, 1980-1990'', Health Education Research,
Vol. 7 No. 3, pp. 403-30.
Kreuter, M.W. and Green, L.W. (1978), ``Evaluation of
school health education: identifying purpose,
keeping perspective'', Journal of School Health,
April, pp. 228-35.
Moon, A.M., Mullee, M.A., Rogers, L., Thompson, R.L.,
Speller, V. and Roderick, P. (1999a), ``Helping
schools to become health-promoting environments
an evaluation of the Wessex Healthy Schools
Award'', Health Promotion International, Vol. 14
No. 2, pp. 111-22.
Moon, A.M., Mullee, M.A., Thompson, R.L., Speller, V.
and Roderick, P. (1999b), ``Health-related research
and evaluation in schools'', Health Education,
Vol. 1, pp. 27-34.
Nutbeam, D. and Smith, C. (1991), ``Evaluating youth
health education: how can we do it and what
should we measure?'', in Nutbeam, D., Hagland, B.,
Farley, P. and Tillgren, P. (Eds), Youth Health
Promotion: From Theory to Practice in School and
Community, Forbes Publications.
Parsons, C., Stears, D. and Thomas, C. (1996), ``The
health-promoting school in Europe: conceptualising
and evaluating the change'', Health Education
Journal, Vol. 55, pp. 311-21.
Ploem, C. and Byers, E.S. (1997), ``The effects of
two AIDS risk reduction interventions on
heterosexual college women's AIDS-related
knowledge, attitudes and condom use'', Journal
of Psychology and Human Sexuality, Vol. 9 No. 1,
pp. 1-24.
Tones, K. (1996), ``Editorial: the health-promoting school:
some reflections on evaluation'', Health Education
Research, Theory and Practice, Vol. 4 No. 4, pp. 1-8.
Tones, K. and Tilford, S. (1990), Health Education:
Effectiveness, Efficiency and Equity, 2nd ed.,
Chapman & Hall, London.
World Health Organisation (WHO) (1998), Health
Promotion Evaluation: Recommendations to
Policymakers, WHO European Working Group on
Health Promotion Evaluation, World Health
Organisation Regional Office for Europe,
Copenhagen.