The current issue and full text archive of this journal is available at
www.emeraldinsight.com/0968-5227.htm
IMCS
17,5
388
Received 26 June 2009
Revised 3 August 2009
Accepted 4 August 2009
Information Management &
Computer Security
Vol. 17 No. 5, 2009
pp. 388-407
q Emerald Group Publishing Limited
0968-5227
DOI 10.1108/09685220911006687
Effects on employees’
information security abilities
by e-learning
Janne Merete Hagen
Gjøvik University College, Gjøvik, Norway, and
Eirik Albrechtsen
SINTEF Technology and Society, Safety Research, Trondheim, Norway
Abstract
Purpose – The purpose of this paper is to measure and discuss the effects of an e-learning tool
aiming at improving the information security knowledge, awareness, and behaviour of employees.
Design/methodology/approach – The intervention study has a pre- and post-assessment of
knowledge and attitudes among employees. In total, 1,897 employees responded to a survey before and
after the intervention. The population is divided into an intervention group and a control group, where
the only thing that separates the groups is participation in the intervention (i.e. the e-learning tool).
Findings – The study documents significant short-time improvements in security knowledge,
awareness, and behavior of members of the intervention group.
Research limitations/implications – The study looks at short-time effects of the intervention.
The paper has done a follow-up study of the long-term effects, which is also submitted to Information
Management & Computer Security.
Practical implications – The study can document that software that support Information
Security Awareness programs have a short-time effect on employees’ knowledge, behaviour, and
awareness; more interventions studies, following the same principles as presented in this
paper, of other user-directed measures are needed, to test and document the effects of different
measures.
Originality/value – The paper is innovative in the area of information security research as it shows
how the effects of an information security intervention can be measured.
Keywords Data security, E-learning, Individual behaviour, Employees, Training
Paper type Research paper
1. Introduction
Maintaining information security in an organization requires the commitment of
employees at all levels. Without full employee commitment, security mechanisms may
be diminished or bypassed entirely (Ward and Smith, 2002; Schneier, 2004; Schultz,
2005). Thomson and von Solms (2006) argue that employees’ compliance with an
organizational security policy can best be achieved through education and awareness
campaigns. Though training and education are generally considered more effective than
more formalistic measures such as procedures and controls (Hagen et al., 2008a), studies
show that many organizations neglect to provide adequate training (Hagen et al., 2008a,
b). A European Network and Information Security Agency (ENISA) report gives an
overview of training practices in 69 companies in nine European countries.
Approximately, 50 percent of the companies used computer-based training (ENISA, Information
2007). security abilities
While the most common way to evaluate the effect of Information Security
Awareness training is quizzes and/or before and after surveys (ENISA, 2007), our by e-learning
study evaluates and discusses the effect of awareness training using a research design
that includes the use of a control group. Our experiment is carried out in the Wilh.
Wilhelmsen (WW) Group, a leading maritime industry group that delivers logistics 389
solutions and maritime services worldwide. The effect of their computer-based security
training program, named Individual Security Awareness (ISA), was tested. A large
part of the program is dedicated to the various aspects of information security. Our
study addressed three research questions:
RQ1. Did the computer-based program result in improved employee information –
security knowledge, awareness, and behavior?
RQ2. Did the extent of the training (i.e. the number of modules performed in the
program) contribute to the improvement of employees’ information security
knowledge, awareness, and behavior?
RQ3. Why did the program either result in changes or fail to create changes in the
employees’ security knowledge, awareness, and behavior?
Section 2 of this paper presents an introduction to ISA; Section 3 outlines the paper’s
theoretical framework; Section 4 explains our applied methodology; and Section 5
describes the findings. In Section 6, the findings are discussed according to the
theoretical framework. Section 7 discusses the conclusions reached.

2. The intervention project: Individual Security Awareness (ISA)

In the WW Group security (including information security) has recently become a
high priority. A corporate security policy was developed and signed by the chief
executive managers, user guidelines were developed, and a Corporate Security
Forum established. There are no technical restrictions on web surfing, but
some restrictions on downloading executable files. In addition, filters are used to
reduce spam and malware from e-mail. To raise individual awareness, in March
2008, the WW Group’s own academy launched the e-learning program, ISA. ISA
consists of six modules, which introduce the following aspects of security:
.
Module 1: introduction. This module introduces various security and risk issues,
defines the risks, and describes the security organization and security
responsibilities.
.
Module 2: information security. This module focuses solely on information
security. It defines information security as confidentiality, integrity, and
availability, explains the threats that may exist to information security, and
shows how employees should handle different classes of information.
.
Module 3: travel security. This module explains how to deal with the risks that
may occur while traveling, such as mugging, street robbery, kidnapping, hotel
fires, diseases, accidents, etc.
IMCS .
Module 4: personal security. This module is about being able to take care of
17,5 yourself, your colleagues, and your family, and discusses ways to deal with risks
such as fire, burglary, kidnapping, loss of sensitive information, etc.
.
Module 5: security of facilities. This module is concerned with the workplace. It is
about protecting premises and detecting and preventing unauthorized entry to
the premises.
390 .
Module 6: internal/external communication. This module is about being aware of
what and how you communicate, both internally and externally.

Even though only one module focuses solely on information security, information
security is included in the other modules, too.
ISA gives an overall introduction to security and information security, but does not
teach the details on how to for instance separate lure web pages from real web pages,
or how to encrypt e-mail, or detect social engineering attacks. It teaches the employees
about the risks connected to these issues and other issues. Each employee can access
and quit ISA through his/her computer, whenever he or she has time to do so. The ISA
e-learning software begins with a vocal introduction, uses pictures, music, and texts to
illustrate the risks, and then provides exercises that motivate reflection. There are also
multiple-choice tests with immediate feedback, including the correct answers. Users
are free to choose whichever module they want to start with. The estimated time to
complete one module is 10-15 minutes. Figure 1 shows a screen-shot from Module 2,
information security. Here, the employee must choose the appropriate security action to
take when leaving documents or a computer in a public place. If the employee makes
an incorrect choice, he or she receives corresponding feedback, but can then continue to
the subsequent question.

Figure 1.
ISA
Our intervention study includes two evaluations of the ISA e-learning software: Information
(1) Evaluation of ISA’s effect on employees’ information security knowledge, security abilities
awareness, and behavior. by e-learning
(2) Evaluation of the training extent’s effect: is there any difference between the
employees who completed only Module 1, Modules 1 and 2, and those who
completed all six modules?
391
The next section will discuss theories of safety and information security management
and the ways that a variety of useful measures may change employees’ behavior.

3. Theoretical framework: information security measures that influence

user performance
Individuals’ performance with regard to information security may be influenced by a
wide range of formal and informal factors: security technologies; formal organizational
structures; awareness, values, and norms; and social relations and interactions
(Albrechtsen, 2007, 2008; Lund and Aarø, 2004). One potential categorization of
information security measures directed at users is shown in Figure 2.
Figure 2 shows what is potentially the most effective sequential ordering for use
of the categories of intervention measures (based on Rundmo, 1990; Hovden et al.,
1992). First, conditions in the working environment should be changed as needed to
be appropriate for and conducive to good security-observant behavior. If this
improvement proves insufficient, it indicates that the workers need further education.
If additional education proves insufficient, it indicates that employees need the kind of
information that will influence their attitudes toward information security. If the effect
of awareness activities proves unsatisfactory, employers should adopt sanctions and

MEASURE:
Measures improving working conditions:
NO
Are technical and organizational preconditions for - Technological security measures (e.g. access control)
safe and secure behaviour satisfactory? - Physical measures (e.g. door locks)
- Formal administrative measures (e.g. policies and instructions)
YES
NO Measures improving skills and knowledge:
Is employees’ knowledge on safe and secure - Experience-based learning (performed work activities;
working routines satisfactory? experienced incidents; simulators)
- Training and education (e.g. tutorials, e-learning programs)
YES
NO Measures improving attitudes:
Are employees positive to make safe and secure - Information, e.g. newsletters, e-mails, web-pages, posters,
actions? screen-savers; mouse pads; direct communication; dialogue
YES
Measures improving behaviour:
NO - Rewards: praise; competitions; gifts; wage scale
Are the working methods safe and secure?
- Sanctions: cautions; threats; punishment; financial sanctions/
compensation
YES
NO Selection of personel:
Are employees qualified to perform
- Positive: engage qualified personel; security clearance
safe and secure actions?
- Negative: remove persons with unacceptable behaviour
Figure 2.
Information security
YES – OK! measures directed at
IT users
Source: Albrechtsen (2008)
IMCS rewards that may modify the employees’ behavior. In dealing with employees who
17,5 may be undesired security risks, a more careful selection of employees is indicated as a
final solution. Unqualified employees should be transferred or dismissed, and work
tasks should be assigned according to individuals’ qualifications. The measures in
Figure 2 should be regarded as complementary. Once, the more technological and
administrative means have been chosen and changes have been implemented,
392 additional “softer” resources can be used if necessary to modify individuals’
performance. The higher up a chosen measure is shown in Figure 2, the more likely
improvement will occur at the individual level. The following sections provide a
synopsis of how a variety of information security measures may influence employee’s
behavior.

3.1 Improving working conditions

The working conditions or environment, within which employees perform their jobs,
includes typically technological and formal administrative measures and also cultural
aspects, the norms, relations, and interactions that exist between individuals.
Computer security systems should be installed that both preserve security and are
usable by employees. However, since users may take short-cuts or lose their motivation
due to a poor user-interface, it is often difficult to take into consideration all the
requirements that might reduce the security threat or risk level (Furnell, 2005).
In addition to technological measures, there are technical-administrative means –
policies, instructions, and plans that document and specify a required level of
behavior – such documents may provide the basic parameters of desired individual
and organizational behavior. To date, the main emphasis in regard to nontechnical
information security has been to use technical-administrative measures (Dhillon and
Backhouse, 2001). However, we know from organizational theory that a planned or
desired behavior may differ considerably from the behavior that actually occurs
(Braverman, 1974; Brunsson, 1989). It is, therefore, likely that security routines and
technology may not be sufficient as single measures to influence individual awareness
and behavior (Albrechtsen, 2008). Nonetheless, it is necessary for control and for
strategic, systematic information security management (Albrechtsen and Hagen, 2008).

3.2 Improving skills and knowledge

Measures that are designed for the purpose of improving skills and knowledge consist
of either experience-based learning activities or systematic training and education. The
former involve learning by personal on-the-job experience; the latter involve learning
by participating in formal education (Hale and Glendon, 1987). Research shows that
most people learn better by actually being involved and doing a particular job than by
sitting in a classroom listening to a lecturer (Wang and Yestko, 2005; Albrechtsen,
2008).
This paper focuses on both the planned training and education courses initiated by
management and the teaching of security expertise through interactive courseware.
Interactive courseware is computer software and associated materials, usually
multimedia in nature, designed for educational and training purposes that teach a
lesson, many times accompanied by a test or quiz. Well-designed interactive
courseware has proven to be an effective teaching mechanism and one that encourages
interactive learning (Wang and Yestko, 2005).
3.3 Improving awareness and attitudes Information
Measures directed at improving employee awareness and attitudes can be applied in security abilities
four ways, by:
(1) directly changing behavioral patterns;
by e-learning
(2) changing the attitude that the behavior is a result of (affection);
(3) creating attentiveness to security questions; or 393
(4) creating a deterrent effect (Rundmo, 1990).

The measures can be used also to improve employees’ knowledge and their perspective
on security measures. In other words, encouraging and instilling a positive view of
security – technology instructions and training programs will lead to better employee
security performance. The various categories of available Information Security
Awareness measures are outlined by both Voss (2001) and Hubbard (2002) to be
notifications, competitions, arrangements, electronic information, public information,
and physical reminders.
There are basically two kinds of awareness campaigns (Iversen et al., 2005). There
are society-wide campaigns, which are characterized by the use of experts, individual
interventions, and large population groups, in which authority figures communicate
with single individuals. There are also community-based campaigns, which draw on
the resources in local communities (empowerment), focus on individuals and groups,
and emphasize cross-disciplinary cooperation. According to Klinke and Renn (2002),
by its use of balanced-risk communication, a discourse-based approach builds
employee knowledge, awareness, and confidence about risk and risk management. Its
emphasis is on convincing employees, not persuading them as is the case with the
top-down approach.

3.4 Improving behavior

Rewards and punishment are used as a means to control the form and frequency of
appropriate/inappropriate security behavior. They become part of the consequences of
employees’ behavior, either positive or negative. Rewards may include social or
personal interaction tools such as competitions, positive feedback, and praise, or more
material and economic things, like gifts and awards. Punishment methods may also
include personal interactions, such as warnings, withholding of certain benefits, and
even dismissal, and more material or economic tools, such as fines and other
economical sanctions and penalties.
While a system of rewards and bonuses is commonly used in the business
world to motivate a high level of performance and productivity, the approach
that predominates at present in the information security field is one of threats and
punishment (Wiant, 2005; Hagen and Spilling, 2009). One problem with the
punishment approach is that people cannot be punished for problems out of their
control. It is not possible to achieve 100 percent protection against the malware
propagating on the internet and information technology (IT) users will be at risk
even though they are careful in surfing the web, opening e-mails, and using
applications, etc. Another problem with the punishment approach is that employees
are protective of one another and thus are reluctant to report their colleagues’ minor
security violations (Hagen, 2009).
IMCS 3.5 Selection of personnel
17,5 The categories of information security-awareness measures listed above are designed,
intended, and presumed to improve employees’ qualifications for achieving adequate
security performance. But the process of selecting personnel is the opposite: people are
selected to do jobs based on their qualifications (i.e. positive selection). Traditionally,
this strategy has long been enforced in the security field by the requirement for
394 personnel security clearances. In Norway, security clearances are legally permitted
under the Norwegian Security Act (Forsvarsdepartementet, 1998). As opposed to
“positive selection,” negative selection means removing personnel from jobs they
cannot handle in a safe and secure way. It typically means relocation or dismissal. If an
employee loses his security clearance, he will automatically be dismissed.

3.6 Combination of measures

To ensure that an organization’s information system is fully secure, a combined
approach including a wide range of measures is needed (Albrechtsen and Hagen, 2008).
The best approach is to combine and implement all the measures discussed in previous
sections to create synergies. For example, a technical administrative system must be in
place before a system of training and education is adopted, because the formal system
provides a framework for the content of the training program. Technology must also
be in place and is an important contributor to the overall effectiveness (Hagen et al.,
2008a). The same is true in regard to our focus here: ISA should be a part of
management’s overall information security efforts.

4. Methodology
This section describes and discusses the research method we used for the intervention
experiment at WW Group.

4.1 Research design, data collection, and statistical analysis

Although there have been few attempts to systematically evaluate the effects of
different ISA programs (Albrechtsen, 2008), intervention studies have long been used
as part of occupational health and safety systems (Goldenhar and Schulte, 1994;
Kristensen, 2005; Robson et al., 2001). The existing studies explore the effects of
planned activities at business worksites that aim at improving the working conditions
and/or the health of workers (Kristensen, 2005). The research design of our study
was inspired by the intervention study literature that focuses on the safety research
area.
To analyze the effects on employees’ security awareness and behavior, we designed
an experiment to measure individual awareness and behavior both before and after the
intervention (shown in Figure 3). In cooperation with WW Group’s security
management, web-based survey questions were developed. The 3,994 employees were
divided randomly into an intervention group that would use the ISA e-learning
software and a control group that would not use the e-learning software.
First, the WW Group published a newsletter on the intranet, signed by the top
manager, announcing ISA’s and the scientific experiment that would measure ISA’s
effectiveness. All 3,994 employees of the WW Group were encouraged to follow the
instructions given in upcoming e-mails and to participate in the experiment.
 Test group Test group Information
security abilities
A A
by e-learning
I
S
B
A
B 395

C C
Study
pop.

Control Control
group group

Figure 3.
The research design and
use of the control group
t1 t2

A week before ISA was launched; an initial survey (t1) was distributed to all
employees. Three weeks after the launch of ISA to the intervention group, a second
survey (t2) was launched. Both surveys included the same questions about knowledge,
awareness, and behavior. The knowledge questions were in the form of a
multiple-choice test, with three possible answers per question. The awareness
questions were answered at five-point Likert (1932) scales; behavior questions were
answered at a five-point scale measuring frequencies of actions (from always to
seldom). The second survey included additional questions about which of the ISA
modules the respondents had completed and about any changes in their awareness or
behavior that had occurred since the launching of the first survey.
The response rate was 68 percent (2,709 answers) for the first survey and 65 percent
(2,587 answers) for the second survey, which is considered a good response. A total of
2,456 respondents answered both surveys. This sample size was reduced to 1,897
respondents after screening both the intervention group and the control group and
excluding those who did not follow the recommendations for participating in the
experiment. The final sample of 1,897 employees constituted the study population, of
which, 1,208 were in the intervention group, and 689 were in the control group. All of
the 1,208 respondents in the intervention group had completed module 1. From those
1,208 employees, three distinct subgroups were formed (Figure 4):
(1) Subgroup A. About 631 respondents who completed Module 1.
(2) Subgroup B. Almost 115 respondents who completed Modules 1 and 2.
(3) Subgroup C. About 356 respondents who completed all modules.

A residual of 106 respondents who completed all or some of Modules 3-5 are not
included in Step 2 analyses.
IMCS
17,5 Subgroup A Completed
(n = 631) module 1

Subgroup B
(n = 115)
396
Completed
module 2
Subgroup C
(n = 356)

Completed module
Figure 4. 3, 4, 5 and 6
The three subgroups

In answering two basic research questions, this research design enabled us to test
several corresponding hypotheses.
RQ1. Did implementation of ISA result in improved employee information security
knowledge, awareness, and behavior?
H01. There was no improvement of information security knowledge, awareness,
and behavior at t2 compared with t1 among members of the intervention
group.
H02. There was no change of information security knowledge, awareness, and
behavior at t2 compared with t1 among members of the control group.
RQ2. Did the extent of the training (i.e. the number of modules performed in the program)
contribute to the improvement of employees’ information security knowledge,
awareness, and behavior?
H03. There was no improvement of information security knowledge, awareness,
and behavior at t2 compared with t1 among members of Subgroup A.
H04. There was no improvement of information security knowledge, awareness,
and behavior at t2 compared with t1 among members of Subgroup B.
H05. There was no improvement of information security knowledge, awareness,
and behavior at t2 compared with t1 among members of Subgroup C.
First, an independent-samples t-test procedure was used to compare means for both the
intervention group and the control group. Since we rely on large randomized numbers
(Ayres, 2007), the respondents were randomly assigned to the two groups, and only the
intervention group got the training (Robson et al., 2001), any difference in response
between the groups can be assumed to be due to training, or a lack of training, not other
factors.
 Anonymous, unique identification numbers were assigned to each respondent. Information
This made it possible to perform time-series analyses and strengthens the validity of security abilities
the analysis. A paired-samples t-test procedure computed the differences in values
between the two variables for each group and tested whether the average differed by e-learning
from 0. A paired-sample t-test was applied to test H01-H05. Cases were excluded
list-wise.
397
4.2 Indexes
The following single items measuring security knowledge were responded to on
a three-point multiple-choice scale. They measured employee knowledge according
to the learning objectives in the e-learning software. To make the scale equal to
the scales used to measure awareness and behavior, the scale was transformed to
a binary scale of 1 and 5 so that a wrong answer equals 1 while a correct answer
equals 5.
Knowledge indexes
.
definition of risk;
.
definition of security policy;
.
definition of integrity; and
.
definition of physical security.

Based on factor analyses, the following awareness indexes were constructed to analyze
the intervention outcomes for employee information security knowledge, awareness,
and behavior. Cronbach’s a is used to measure the reliability of the indexes, and a
value above 0.7 is usually considered as satisfactorily (Ringdal, 2001).
Awareness indexes
.
Security versus functionality. Information security perceived as not being an
obstacle and not only being a technological challenge, consisting of five items,
Cronbach’s a ¼ 0.67.
.
Reporting. Willingness to report a colleague or a superior who breached security
to the security management, consisting of two items, Cronbach’s a ¼ 0.76.
.
Importance of generic security and safety means. Perceived importance of
following security guidelines, of health, environment, and safety management,
and of fire protection, consisting of four items, Cronbach’s a ¼ 0.80. The
reliability of all awareness indexes for the first data set was satisfactory.

It is a concern that people can claim that they understand the vulnerability of writing
down passwords and locking PCs, but not actually do it. Therefore, the questionnaire
also had several questions regarding the frequency of certain security behavior.
We examined how often the respondents performed different information security
tasks such as keeping passwords secret. The items were responded to on a five-point
scale from “always” to “seldom.” Some items had skewed distributions; for instance,
for the questions on how to treat sensitive information, where more than 90 percent
agreed with the correct statements. As it would be difficult to get significant changes
for these items, they were excluded from the analyses. Only the following items
measuring security behavior were included in the analysis.
IMCS Behavior indexes
17,5 .
write down passwords on paper;
.
lock the PC whenever leaving it; and
.
reporting incidents when detected.

398 4.3 Limitations of the study

One limitation of the study was that, due to time constraints, a pre-test was not
conducted outside the project group. A trade-off had to be made to meet the WW Group
management’s objective in launching ISA on time. Some of the employees who
participated had difficulty interpreting some of the questions in the survey. That kind
of feedback could have been sorted out in a pre-test.
The major weakness of the study was the short-time distance between the ISA
intervention and the two surveys, the first survey at t1, a week before ISA was launched
and the second survey at t2 three weeks after ISA was launched. Because of certain
practical arrangements, no longer time schedule was possible at this time. Therefore,
only short-term effects were measured. It would be interesting, however, to determine
the long-term effects as well, so as to study how knowledge, awareness, and behavior
change over time. The research design worked for the measurement of short-time
effects, and it will probably work for the measurement for long-term effects as well.
Another weakness with the study is the meaning of the test scores obtained. They
are a matter of discussion because the ability to answer the questions does not
necessarily correlate with critical security-related changes in job behavior (Schultz,
2004). An independent experiment on, for example, social engineering could have
strengthened the internal validity, but was not possible to arrange within the limited
time constraints.
There are, of course, differences between a training context and real-life
decision-making and actions. A security awareness training program cannot cover
all possible security problems and dilemmas an employee faces in his/her working day.
However, any training program serves as a preparation for employees’ decisions and
actions by training the ability to anticipate monitor and respond to any security
challenges, independent of the topics covered by the training. Our research design,
based on pre- and post-surveys, also address a few concrete security issues. We must
thus assume, if the quantities surveys indicate changes, that the training program
influence the way employees think and act regarding information security on a general
basis. Furthermore, we cannot validate whether the employees actually behave in a
different way when it comes to practical problems in the real world. This is a general
problem for all quantitative research on individual attitudes and behavior – what
people state regarding their attitudes and behavior may differ from how they actually
think and behave. Nevertheless, we must assume that indications of changes in
attitudes, knowledge or behavior provided by quantitative surveys, also reflect changes
in real-life. In the long-term run, it is possible to study whether the training program has
materialized into changed awareness and behavior in the organization, by studying if
there has been a decrease in the number of security incidents, which could be explained
by the training program.
The next section presents the results of the experiment.
5. Results Information
This section first describes the demographic characteristics of the participants and security abilities
then presents the results of the experiment.
by e-learning
5.1 Participants
Table I gives details about the demographic characteristics of the two groups and their
background variables. The WW Group consists of several companies, but the majority 399
of the employees (80 percent) that participated in the experiment came from WMS Ship
Services and WMS Ship Management. A majority of the participants were
well-educated: half had more than 15 years’ of education; an additional one-third
had 13-15 years of education. Every fourth participant was also a manager. x 2-tests at
the 5 percent level confirm that there were no significant differences between the test
group and the control group in regard to gender, age, education, experience, or
responsibility. Significant differences were identified, however, between the groups

Control group in percent Intervention group in percent

Variable (n ¼ 689) (n ¼ 1,208)

Gender
Male 67.9 65.0
Female 32.1 35.0
Age
18-25 7.7 7.9
26-30 13.8 15.1
31-35 12.8 15.6
36-40 16.3 18.8
41-50 28.3 25.5
51-60 17.1 17.4
61þ 4.1 3.8
Formal education (years)
Up to 7 0.6 0.7
8-12 15.2 16.2
13-15 32.8 33.3
More than 15 49.9 47.4
Other 1.5 2.4
Employment period (years)
0-1 19.2 20.2
2-5 25.1 28.1
6-10 19.8 20.0
11-24 28.9 23.3
25þ 7.8 8.4
Level of position
Top manager 7.4 5.3
Middle manager 33.8 36.8
Employee 58.8 57.9 Table I.
Dedicated security responsibility Demographic
Yes 32.2 40.6 characteristics of
No 67.8 59.4 participants in the
intervention and control
Note: n ¼ 1,897 groups at t1
IMCS with respect to which of the two companies they worked for and whether they had any
17,5 dedicated security responsibilities.
The next section presents the result of the pre- and post-tests.

5.2 Pre- and post-tests

Independent-sample t-tests of the intervention group and the control group show no
400 significant differences in knowledge, awareness, or behavior between them at t1.
Together with the results of the x 2-tests, these results show that, except for the
difference between the companies the respondents represented, and their sense of
security responsibility, there was no initial skewing between the two groups. Table II
shows the results of the pre- and post-survey paired-sample t-tests.

Index t1 (mean (SD)) t2 (mean (SD)) t (df)

Knowledge
Definition of risk
Intervention group 2.15 (1.81) 2.01 (1.74) 1.99 (1,016) * *
Control group 2.14 (1.80) 1.98 (1.72) 2 1.66 (517)
Definition of security policy
Intervention group 1.78 (1.58) 1.90 (1.67) 2.07 (1,016) * *
Control group 2.03 (1.75) 2.06 (1.77) 0.37 (517) *
Definition of integrity
Intervention group 1.93 (1.69) 2.27 (1.85) 4.67 (1,016) * *
Control group 2.03 (1.75) 2.03 (1.75) 0.00 (517)
Definition of physical security
Intervention group 3.21 (1.99) 3.48 (1.94) 3.72 (1,016) * * *
Control group 3.27 (1.98) 3.29 (1.98) 0.25 (517)
Awareness
Security versus functionality
Intervention group 3.46 (0.61) 3.56 (0.61) 6.12 (1,016) * * *
Control group 3.40 (0.61) 3.49 (0.57) 3.92 (517) * * *
Reporting
Intervention group 3.80 (0.71) 3.85 (0.72) 2.32 (1,016) * *
Control group 3.76 (0.71) 3.81 (0.67) 2.08 (517) * *
Importance of generic security and safety means
Intervention group 4.04 (0.58) 4.06 (0.62) 1.50 (1,016)
Control group 3.99 (0.61) 3.98 (0.58) 0.72 (517)
Behavior
Write down passwords on paper
Intervention group 4.19 (1.07) 4.29 (0.95) 3.64 (1,016) * * *
Control group 4.14 (1.06) 4.16 (0.99) 0.30 (517)
Lock the PC
Intervention group 3.84 (1.22) 3.99 (1.14) 4.24 (1,016) * * *
Control group 3.85 (1.17) 3.82 (1.13) 0.75 (517)
Report incidents
Table II. Intervention group 3.90 (1.29) 4.09 (1.12) 4.67 (1,016) * * *
Results of the Control group 3.87 (1.23) 3.82 (1.21) 2 1.14 (517)
paired-sample t-tests of
the pre- and post-survey Notes: *p , 0.10, * *p , 0.05, * * *p , 0.001; SD – standard deviation; t ¼ t-value; df ¼ degrees of
for the intervention group freedom; scales: the indexes ranges from 5 – best to 1 – poorest; the test for the intervention group and
and the control group control group are two tailed
The paired-sample t-test results indicate an improvement in the information security Information
knowledge, awareness, and behavior of the employees in the intervention group as security abilities
compared with those in the control group, although the awareness of members of the
control group also changed to some extent. The knowledge indexes show significant by e-learning
improvements for the intervention group for all indexes except risk, revealing a greater
understanding of security policy, physical security, and integrity of information.
The results of the paired-sample t-tests show that both the intervention group and 401
the control group improved their awareness of the necessity to reporting incidents
and their view on security versus functionality. After the intervention took place,
employees in the intervention group showed improved behavior in protecting access to
their computers. They reported security violations and incidents more often, and
locked their PCs more often whenever they left them. In addition, they did not write
down passwords as often as they had before. All these aspects are a focus of the ISA
program.
To validate our findings, we asked the participants in the intervention group why
they thought their attitudes had changed. Of the 736 employees who answered this
question, 66.3 percent reported that it was due to their use of ISA. We also asked the
respondents in the intervention group their opinion of the learning effects of ISA. Of
the 1,206 employees in the intervention group, 49 percent reported that they had
changed in their use of the internet, 45.8 percent had changed the way they kept their
user names and passwords secret, and 55 percent noted a change in their awareness of
how to treat internal and sensitive information. About 55 percent reported an increase
in their attention to security incidents, 31.4 percent reported a change in how they
manage visitors to the site, 20 percent noted a change in their willingness to report
security incidents and weaknesses, while 40.4 percent reported they had had a change
in attitude toward the importance of information security versus productivity.
These results correspond to the findings of the paired-sample t-tests and confirm
that the short-term effect of ISA in the WW Group was improvement in the employees’
information security knowledge, awareness, and behavior even though far from all
employees completed the program.
Nonetheless, when 23 top managers representing different parts of the WW Group
were questioned five months after ISA was launched, they reported a diversified view
of the observed effects of ISA. None of them had noticed any increase in reported
security violations, but some of the managers found that they personally had become
more aware of their own security behavior and also noticed more discussions about
security in their organizations.
Altogether, given these findings, we reject H02 of no effect of ISA. The control
group showed three significant changes in knowledge and awareness regarding
security policy, reporting, and their attitudes towards security versus functionality.
These findings may be explained by the effect that participating in the experiments
influenced their awareness. Therefore, we can partly reject H03 that there were no
change in the control group. In the next section, we continue with our analysis of the
effects of extent of training variations on employees.

5.3 Post-test results: RQ2

The intervention group was divided into three subgroups according to how much
of ISA they had completed. Table III shows that completing all six modules of ISA
IMCS
Index t1 (mean (SD)) t2 (mean (SD)) t (df)
17,5
Knowledge
Definition of risk
Subgroup A 2.18 (1.83) 2.03 (1.75) 1.57 (542)
Subgroup B 2.08 (1.78) 2.42 (1.92) 1.38 (92)
Subgroup C 2.08 (1.78) 1.86 (1.64) 21.83 (298) *
402 Definition of security policy
Subgroup A 1.72 (1.54) 1.81 (1.61) 1.21 (542) *
Subgroup B 1.77 (1.59) 1.82 (1.62) 0.23 (92)
Subgroup C 1.90 (1.67) 2.10 (1.68) 1.74 (298) *
Definition of integrity
Subgroup A 1.86 (1.64) 2.04 (1.76) 1.98 (542) *
Subgroup B 1.90 (1.68) 2.33 (1.89) 1.99 (92) * *
Subgroup C 2.08 (1.78) 2.66 (1.97) 4.75 (298) * * * *
Definition of physical security
Subgroup A 3.14 (2.00) 3.48 (1.95) 3.29 (542) * *
Subgroup B 3.32 (1.99) 3.15 (2.00) 0.67 (92)
Subgroup C 3.29 (1.98) 3.52 (1.94) 1.81 (298) *
Awareness
Security versus functionality
Subgroup A 3.43 (0.58) 3.53 (0.58) 4.43 (542) * * * *
Subgroup B 3.34 (0.61) 3.46 (0.59) 1.98 (92) *
Subgroup C 3.53 (0.62) 3.63 (0.67) 4.00 (298) * * *
Reporting
Subgroup A 3.80 (0.70) 3.79 (0.71) 20.25 (542)
Subgroup B 3.72 (0.82) 3.81 (0.76) 1.32 (92)
Subgroup C 3.78 (0.70) 3.93 (0.74) 4.04 (298) * * * *
Importance of generic security and safety means
Subgroup A 4.04 (0.58) 4.05 (0.62) 0.37 (542)
Subgroup B 3.99 (0.58) 3.95 (0.72) 20.52 (92)
Subgroup C 4.04 (0.59) 4.12 (0.59) 2.23 (298) * *
Behavior
Write down passwords on paper
Subgroup A 4.12 (1.07) 4.24 (0.95) 2.98 (542) * * *
Subgroup B 4.17 (1.16) 4.28 (0.93) 1.23 (92)
Subgroup C 4.28 (1.00) 4.36 (1.90) 1.74 (298) *
Lock the PC
Subgroup A 3.85 (1.20) 3.92 (0.16) 1.45 (542)
Subgroup B 3.81 (1.34) 4.05 (1.15) 2.49 (92) * *
Subgroup C 3.91 (1.17) 4.07 (1.11) 22.56 (298) * *
Report incidents
Subgroup A 3.91 (1.28) 4.06 (1.13) 22.59 (542) *
Subgroup B 3.98 (1.26) 4.06 (1.09) 20.70 (92)
Table III. Subgroup C 3.92 (1.32) 4.19 (1.11) 24.02 (298) * * * *
Independent sample
t-tests when two ISA Notes: *p , 0.10, * *p , 0.05, * * *p , 0.005, * * * *p , 0.001; SD – standard deviation; t ¼ t-value;
modules were completed df ¼ degrees of freedom; scales: the indexes ranges from 5 – best to 1 – poorest; the tests for the
versus more intervention group and control group are two tailed

proved to have the greatest effect on employee information security knowledge,

awareness, and behavior. In Subgroup C, nine indexes showed significant
improvements, while in Subgroups A and B, respectively, six and three indexes
improved significantly. At t2, the mean values of the nine indexes of Subgroup C
are higher than the mean values of Subgroups A and B. Subgroup B members show Information
only three significant improvements, which is a bit surprising because their modules security abilities
focused on information risk and security, and we had expected that responses to the
typical computer security questions would show employee development. Based on by e-learning
these results, H03-H05 can be rejected.

6. Discussion 403
6.1 Did ISA change employees’ information security knowledge, awareness, and
behavior in the short-term?
The theoretical model discussed in Section 2.1 describes how security measures can be
directed at employees so as to influence their behavior. The WW Group had the
technical and organizational measures in place before the ISA experiment started, but
according to the security management, there was room for improvement to achieve
compliance with the security policy and guidelines. Also, the use of sanctions and
rewards (Step 4) and selection of personnel for security reasons (Step 5), were not
applied.
The intervention study documents that ISA managed to significantly change the
security knowledge, awareness, and behavior of employees in the intervention group.
These findings are well in line with Wang and Yestko (2005) who found that
well-designed interactive courseware show effects in improving teaching effectiveness
and encouraging active learning. The statistical results were confirmed by answers
given by the participants. However, the top managers reported diversified views of the
effects of the ISA experiment and did not notice any change in reported incidents. One
hypothesis for this finding is that reported incidents are filtered out on their way up
through the hierarchical structure of the organization, so top management does not
perceive any change.
There were significant changes in both the intervention and the control group in
respect to improved attitudes toward security versus functionality and reporting of
security violations. This may be explained by the Hawthorne effect (Olson et al., 2004),
that the employees were influenced by the experiment itself, and adjusting their
behavior towards what was expected. The Hawthorne effect may be caused by the
promoting activities of the ISA program before and under the experiment.
The many skewed answers in our study indicate that many employees initially
already had a high level of awareness of some information security issues. The skewed
answers correspond well with the finding that many of the employees who participated
in the study were not only well-educated, but many also working with security issues.
This fits well with the findings of Albrechtsen (2008) that user involvement is the best
method to get employees security conscious. The relatively limited potential for
improvement may explain why the measured improvements themselves, though
significant, were not extreme.
One main effect of ISA is recognition of the necessity for employees to report every
security incident that is detected, whether it involves a superior or a colleague at the
same level. This finding is important for three reasons. First, it is expected that
employees will confront a superior or a colleague regarding security and report any
lapse to the security manager. Second, according to the findings of Wiant (2005) and
Hagen and Spilling (2009), an increase in reporting will have a deterrent effect,
preventing future incidents. Third, it will also give the security management a more
IMCS up-to-date picture of the company’s overall security status, providing them with better
17,5 security management information. Hagen (2009) found in her study a correlation
coefficient of 0.6 between detecting and reporting of security incidents. Employees
were reluctant to report a colleague or a superior, they lacked sufficient security
knowledge, or considered an incident insignificant, or they did not consider security
their responsibility. The results of our study show that computer-based training, like
404 ISA, where the aim is to create greater security knowledge and awareness among
employees, can influence some of these undesired attitudes among employees, at least
in the short term. A study of long-term effect can provide more advice regarding the
necessity to frequently repeat training.

6.2 Were there differences between the employees that completed Modules 1 and 2 and
those that completed all modules?
Employees are required and expected to comply with their organization’s security
policies. However, while the security management works as a counter-balance to move
away from the boundaries of unacceptable risk, employees work under pressure from
management to move toward optimum efficiency and their own goal of exerting the
least effort (Rasmussen, 1997). We observed exactly this phenomenon in our
experiment when the managers asked: what is all this security issue about anyway?
Besides, while most of the people in the test group completed Module 1, fewer
completed Module 2, and there was a significant drop with Module 3, and remaining
modules. Moreover, the qualitative answers given by the respondents in the two
surveys indicate a conflict in goals – participating in the security training versus
doing their daily jobs – as one of their reasons for not completing ISA.
The results show varying significant improvements in employee information
security knowledge, awareness, and behavior among those in Subgroups A-C, as
documented in Section 5.3. One possible explanation for this variation is that the people
participating in the experiment had a relatively high level of competence and thus there
was only limited room for improvement, from good to even better, if just a part of
ISA was completed. Another possible explanation may lay in the way the questions
were raised: many of them focused on general security and risk management issues.
The findings confirm that, in comparison to completing only parts of ISA, completing
the entire program results in an increase in employee security knowledge, awareness,
and behavior. Employees should therefore be encouraged to complete all the ISA
modules.

6.3 Why did the intervention result in changes in employee awareness and behavior?
The results indicate that ISA was shown to be an effective method for training
employees, which is well in line with existing theory on interactive courseware (Wang
and Yestko, 2005). However, according to the experiment indicators, the control group,
which was exposed to the promotion and questionnaires, also showed a significant
change over time in three indexes, but not in knowledge or behavior. The following
discussion aims to clarify how this might happen.
While the intervention consisted basically of the ISA e-learning software, news
about the launching of the program and the scientific experiment was first published
on the intranet. In an effort to promote employee participation, before the experiment
began, a trial module test was also made available on the intranet. Therefore, everyone
in the organization received some information about what was going to happen and Information
that the focus was on security. This activity may have had two impacts: a large security abilities
response rate and engagement in the experiment, but also a psychological side effect
similar to the Hawthorne effect, influencing the awareness also in the control group. by e-learning
Through the given answers in the surveys it became clear that the participants learned
not just only from the ISA e-learning software, but also from the surveys they
participated in and their interactions and contacts with colleagues. 405
7. Conclusions
The implementation of ISA in the WW Group provides a large-scale, computer-based
and standardized security training that can facilitate employee compliance with the
organizations’ security policies by raising individual security knowledge and
awareness. We conducted an experiment to evaluate ISA’s effectiveness in those areas.
Our results show that the program had a significant short-term effect on employee
security knowledge, awareness and behavior. There were significant differences
between the intervention subgroups and that, in order to get the full benefit of the
training, all employees should be encouraged to complete the entire program. ISA alone
was shown to have a significant effect on improving employee security knowledge and
behavior. The combination of the ISA e-learning software with surrounding activities,
ISA promotion, and surveys may all have contributed to the observed change in all
employees’ security awareness, as seen in the changes in the awareness of the control
group. Good promotion contributed probably to a high-response rate at the cost of the
Hawthorne effect. One lesson learned from the experiment is to discuss research design
and questions with physiologists to eliminate any psychological side effects that might
occur during such experiments.
Finally, it should be noted that the long-term effect has not yet been analyzed, and
that individual learning is not the same as organizational learning, where the latter
results in a change in common understanding, relations, and interactions. Our
intervention study did not use a group-based approach in which employees could share
knowledge and experience. Rather, ISA is a tool for raising individual employees’
security awareness and, as such, is a good starting point for building a corporate
security culture based on common values and attitudes. The experiment showed that
ISA itself started some knowledge-sharing processes in the organization.
This study has focused on the short-term effects of ISA. We intend to continue with
a follow-up study on the long-term effects of the program. In this follow-up study, we
will discuss computer-based training compared with human intervention and action
research and their effects on organizational learning.

References
Albrechtsen, E. (2007), “A qualitative study of users’ view on information security”, Computers
& Security, Vol. 26 No. 4, pp. 276-89.
Albrechtsen, E. (2008), “Friend or foe? Information security management of employees”,
Thesis No. 2008:101, Norwegian University of Science and Technology, Trondheim.
Albrechtsen, E. and Hagen, J. (2008), “Information security measures influencing user
performance”, in Martorell, S., Soares, C.G. and Barnett, J. (Eds), Proceedings of Safety,
Reliability, and Risk Analysis: Theory, Methods, and Applications, CRC Press, London,
pp. 2649-56.
IMCS Ayres, I. (2007), Super Crunches: How Thinking by Numbers is the New Way to be Smart, Bentam
Books, New York, NY.
17,5
Braverman, H. (1974), Labor and Monopoly Capital: The Degradation of Work in the Twentieth
Century, Monthly Review Press, New York, NY.
Brunsson, N. (1989), The Organization of Hypocrisy: Talk, Decisions, and Actions in Organizations,
Wiley, Chichester.
406 Dhillon, G. and Backhouse, J. (2001), “Current directions in IS security research: towards
socio-organizational perspectives”, Information Systems Journal, Vol. 11 No. 2, pp. 127-53.
ENISA (2007), Information Security Awareness Initiatives: Current Practice and the
Measurement of Success, European Network and Information Security Agency, Heraklion.
Forsvarsdepartementet (1998), Lov om forebyggende sikkerhetstjeneste (Sikkerhetsloven). The
Norwegian Security Act, Forsvarsdepartementet, Oslo.
Furnell, S. (2005), “Why users cannot use security”, Computers & Security, Vol. 24 No. 4, pp. 274-9.
Goldenhar, L.M. and Schulte, P.A. (1994), “Intervention research in occupational health and
safety”, Journal of Occupational Medicine, Vol. 36 No. 7, pp. 763-75.
Hagen, J.M. (2009), “How do employees comply with security policy? A comparative case study
of four organizations under the Norwegian Security Act”, The Human Factor behind the
Security Perimeter. Evaluating the Effectiveness of Organizational Information Security
Measures and Employees’ Contributions to Security, doctoral dissertation, The Faculty of
Mathematics and Natural Sciences, University of Oslo, Oslo.
Hagen, J.M. and Spilling, P. (2009), “Do organizational security measures contribute to the
detection and deterrence of IT-system abuses?”, Proceedings of the 3rd International
Conference on Human Aspects of Information Security and Assurance (HAISA 2009).
Hagen, J.M., Albrechtsen, E. and Hovden, J. (2008a), “Implementation and effectiveness of
organizational information security measures”, Information Management & Computer
Security, Vol. 16 No. 4, pp. 377-97.
Hagen, J.M., Kalberg-Sivertsen, T. and Rong, C. (2008b), “Protection against unauthorized access
and computer crime in Norwegian enterprises”, Journal of Computer Security, Vol. 16,
pp. 341-66.
Hale, A.I. and Glendon, A.I. (1987), Individual Behavior in the Control of Danger, Elsevier,
Amsterdam.
Hovden, J., Ingstad, O., Mostue, B.A., Rosness, R., Rundmo, T. and Tinnmansvik, R.K. (1992),
Ulykkesforebyggende arbeid (Accident Prevention), Yrkeslitteratur, Oslo (in Norwegian).
Hubbard, W. (2002), “Methods and techniques of implementing a security awareness program”,
SANS Institute White Paper, SANS Institute, Bethesda, MD.
Iversen, H., Rundmo, T. and Klempe, H. (2005), “Risk attitudes and behavior among Norwegian
adolescents: the effects of a behavior modification program and a traffic safety campaign”,
European Psychologist, Vol. 10 No. 1, pp. 25-38.
Klinke, A. and Renn, O. (2002), “A new approach to risk evaluation and management: risk-based,
precaution-based, and discourse-based strategies”, Risk Analysis, Vol. 22 No. 6, pp. 1071-94.
Kristensen, T.S. (2005), “Intervention studies in occupational epidemiology”, Occupational and
Environmental Medicine, Vol. 62 No. 3, pp. 205-10.
Likert, R. (1932), “A technique for the measurement of attitudes”, Archives of Psychology, Vol. 140,
pp. 1-55.
Lund, J. and Aarø, L.E. (2004), “Accident prevention: presentation of a model placing emphasis
on human, structural, and cultural factors”, Safety Science, Vol. 42 No. 4, pp. 271-324.
Olson, R., Verley, J., Santos, L. and Salas, C. (2004), “What we teach students about the Information
Hawthorne studies: a review of content within a sample of introductory I-O and OB
textbooks”, The Industrial-organizational Psychologist, Vol. 41 No. 3. security abilities
Rasmussen, J. (1997), “Risk management in a dynamic society: a modeling problem”, Safety by e-learning
Science, Vol. 17 Nos 2/3, pp. 183-213.
Ringdal, K. (2001), Enhet og mangfold: samfunnsvitenskapelig forskning og kvantitativ metode
(Unity and Diversity: Social Science and Quantitative Methods), Fagbokforlaget, Bergen 407
(in Norwegian).
Robson, L.S., Shannon, H.S., Goldenhar, L.M. and Hale, A.R. (2001), “Guide to evaluating the
effectiveness of strategies for preventing work injuries: how to show whether a safety
intervention really works”, NIOSH Publication No. 2001-119, NIOSH, Cincinnati, OH.
Rundmo, T. (1990), Atferdsvitenskaplig sikkerhetsforskning (Safety Research on Behavior),
SINTEF Report STF75A9007, SINTEF, Trondheim (in Norwegian).
Schneier, B. (2004), Secrets and Lies: Digital Security in a Networked World, Wiley, Indianapolis, IN.
Schultz, E. (2004), “Security training and awareness: fitting a square peg in a round hole”,
Computers & Security, Vol. 23 No. 1, pp. 1-2.
Schultz, E. (2005), “The human factor in security”, Computers & Security, Vol. 24 No. 6, pp. 425-6.
Thomson, K.-L. and von Solms, R. (2006), “Towards an information security competence
maturity model”, Computer Fraud & Security, No. 5, pp. 11-15.
Voss, B.D. (2001), “The ultimate defense of depth: security awareness in your company”,
SANS Institute White Paper, SANS Institute, Bethesda, MD.
Wang, A.J.A. and Yestko, K. (2005), “Building reusable information security courseware”,
paper presented at the 2005 Information Security Curriculum Development Conference.
Ward, P. and Smith, C.L. (2002), “The development of access control policies for information
technology systems”, Computers & Security, Vol. 21 No. 4, pp. 365-71.
Wiant, T.L. (2005), “Information security policy’s impact on reporting security incidents”,
Computers & Security, Vol. 24 No. 6, pp. 448-59.

Corresponding author
Janne Merete Hagen can be contacted at: janne.hagen@hig.no

To purchase reprints of this article please e-mail: reprints@emeraldinsight.com

Or visit our web site for further details: www.emeraldinsight.com/reprints