Applied Ergonomics 40 (2009) 379–395

Contents lists available at ScienceDirect

Applied Ergonomics
journal homepage: www.elsevier.com/locate/apergo

Towards a framework to select techniques for error prediction:

Supporting novice users in the healthcare sector
Melinda Lyons*
Engineering Design Centre, University of Cambridge, c/o Department of Engineering, Trumpington Street, Cambridge, CB2 1PZ, UK

a r t i c l e i n f o a b s t r a c t

Article history: Whilst healthcare has increased its awareness of the retrospective safety assessment techniques, such as
Received 8 February 2008 root cause analysis, adoption of the corresponding predictive safety assessment techniques has been
Accepted 15 November 2008 slow and sporadic. Reasons for this may include lack of support in technique choice and practical
knowledge in the published literature. Whilst there have been many publications on these techniques,
Keywords: few have aimed to support the novice user in selecting a technique from the broad array of choice to
Human reliability
facilitate targeting of education in techniques for specific purposes.
Novice users
This paper aims to address this through collecting an evidence base towards developing a bottom-up
Healthcare
(resources and constraints) and top-down (requirements) approach to technique selection.
Conclusions indicate there is a lack of practical experiences described in the literature to conclusively
define a technique for selection and a need for a dedicated research in this area to make it accessible for
healthcare and other novice users.
Ó 2008 Elsevier Ltd. All rights reserved.

1. Introduction
Incident investigation techniques have been increasingly used
in healthcare and are supported by literature specifically written to
help novices in choosing techniques (Johnson, 2003). However,
a similar pattern has not occurred for predictive safety techniques.
Despite many decades of acceptance of the predictive safety tech-
niques in other industries, Lyons et al. (2004a) found only seven
techniques had been published as being used for healthcare
application (change analysis, FMEA, HAZOP, influence diagrams,
SHERPA, event trees and fault trees). This was particularly note-
worthy when task analysis – the precursory step for many human
reliability analysis techniques – has been applied to several areas
within healthcare.
The reasons for its narrow application may be the lack of
awareness that there are so many usable techniques or due to the
challenge of choosing between the overwhelming number of
techniques – with 520 safety assessment methodologies identified
for supporting air traffic management (Everdij, 2004). Although
awareness and understanding of the practical application of such
a great number of techniques may appear impossible, it should be
emphasised that not all the techniques are discrete, with many
variants evolving for a subset of the techniques. One idiosyncrasy of
safety assessment techniques is that there is a popular trend to give
the techniques acronymic (HEART) or initialistic (HTA) names.
* Tel.: þ44 0790 449 7655; fax: þ44 01223 760 567.
E-mail address: melinda_lyons@hotmail.com
Therefore, often techniques that are identical in form have been
given different names due to application in different domains or
have minor changes made by authors. Conversely identical tech-
niques have evolved with slightly different names – e.g. Safety
barrier function analysis (Kecklund et al., 1996), accident evolution
barrier analysis (Svenson, 1991, 2001), energy barrier analysis
(Rahimi, 1986) and barrier analysis (Hollnagel, 2004). Kirwan
(1998a) outlines an approximate evolution relationship of many of
these techniques (e.g. FMEA, HAZOP and event tree based tech-
niques) including cross-links between these.
Even in industries more familiar with reliability engineering
techniques, it has been speculated that these techniques were
used so scarcely due to the unavailability of the means and/or
techniques, technique complexity (Paz Barroso and Wilson,
2000) or lack of information on resources required (Ainsworth
and Marshall, 1998). Pradhan et al. (2001) suggest the cultural
norms of healthcare also add to the challenges – perhaps
resulting in demands for tailor-made instead of the industry
standard techniques.
The continuous professional development required by health-
care professionals is broader than the remits of safety assessment.
Even with governmental support, this may limit opportunities to
learn a range of safety techniques. It may be preferable for
healthcare users unfamiliar with the field to choose a technique to
solve a specific problem and then target the technique education
accordingly.
Otherwise, there is a danger that novices are using techniques to
guide safety-related decisions without the training appropriate to

0003-6870/$ – see front matter Ó 2008 Elsevier Ltd. All rights reserved.
doi:10.1016/j.apergo.2008.11.004
380 M. Lyons / Applied Ergonomics 40 (2009) 379–395
gain the validity and reliability assumed from the technique
(Stanton and Young, 2003).
Whilst there have been a number of reviews of the human
reliability analysis and similar techniques (Everdij, 2004; Hum-
phreys, 1988; Suokas, 1988; Kirwan, 1992a,b, 1994, 1998; Stanton
et al., 2005; Wreathall and Nemeth, 2004), most have focussed
more on the accuracy and reliability of the techniques rather than
the requirements of the problem to be analysed. Together with the
issues of a lack of technique awareness and expertise, the pressures
on healthcare personnel will tend to make healthcare professionals
more reluctant to invest in the rigorous education requirements for
the more complex techniques, designed to produce more accurate
results. This may lead to personnel choosing the ‘‘quick and dirty’’
techniques even if they are not the most appropriate for the
problem. Furthermore, the previous reviews have always been
written in a technique-by-technique format thus demanding time
commitment in investigating techniques that would not be chosen,
simply to identify the technique that best matches the features of
the chosen problem. In many cases, this assumes the readers to
have some knowledge or experience in the field and already have
a mental model of many or all the techniques. In short, what is
required is a user-focussed approach for selecting a technique –
namely a bottom-up method of selection based on the resources,
constraints and requirements of the user.
As yet, there has been no dedicated study to research the
usability of the techniques, the limits of requirements for each
technique in terms of time or human resources or any efforts to
support the novice in selecting a technique to analyse their
problem. Therefore, this paper provides a first step in providing
a literature-based framework for selection of techniques.
2. Terms and definitions
Sources of confusion are the terms and definitions used when
describing the techniques. Therefore this section provides
a cautionary word about the ‘‘idiosyncratic’’ use of acronyms and
initialisms within the field. (A full list is shown in Lyons et al.,
2004b.) Tools, techniques, processes and methodologies are used
interchangeably within the literature. However, for the benefit of
this paper, the term ‘‘process’’ will be reserved to mean the ‘‘clinical
process’’ under assessment for its risks, i.e. ‘‘the process of inserting
a cannula’’, ‘‘the process of prescribing drugs’’, ‘‘the process of
resuscitating a patient’’ or ‘‘the process of organising a referral’’. The
‘‘system’’ is defined as ‘‘a set of connected items or devices which
operate together’’. This is the term used to represent the ‘‘actors’’
carrying out the processes or influencing upon it, including the
work environment, people and equipment.
3. Method
This paper aims to cluster the information in the literature to
provide an initial framework to support novice users in selecting
a technique for use.
The findings of a previous review of the literature (Lyons et al.,
2004b) were used to structure the framework. These had been
reviewed by the author – an individual with over 10 years experi-
ence in human reliability and safety. Acknowledging the findings of
Everdij (2004), Suokas (1988) and Kirwan (1998b), this included
a search using 168 generic and specific search terms on Embase,
Ergonomics abstracts and Medline to identify HRA techniques. The
generic search terms included ‘‘error identification’’, ‘‘human reli-
ability’’ and ‘‘patient safety’’; whereas the specific searches
included the technique names and their acronyms. For each result
identifying more than 100 references, only the first 100 were
included. From this, the cumulative searches, performed in
November 2003, found in excess of 8000 abstracts which revealed
134 techniques for use in predictive safety/human reliability anal-
ysis. These were screened according to the following criteria:
 Conceptual models that were not demonstrable in practice
were eliminated from the list (SCFM).
 Generic descriptions or toolboxes were eliminated – e.g. root
cause analysis or probabilistic risk assessment were broken
down to the original techniques.
 Highly developed domain specific techniques were excluded if
it was deemed unlikely that they could be converted for use in
healthcare (or if conversion would mean building the tech-
nique from first principles or from a more generic HRA tech-
nique that had already been included in the list).
 Computer-based or simulated versions of methods were
included in terms of one base method – not as individual
methods – where methods were duplicated in different
versions, the version showing the greatest potential for use in
healthcare was recorded (e.g. THERP rather than ASEP, SHERPA
rather than PHEA).
 Methods that were only data collection (e.g. questionnaires,
interviews) were eliminated from the list.
 The technique must appear in more than one paper (thus
demonstrating at least presentation of a technique and ongoing
use beyond this). Some leniency was given for methods high-
lighted in recent publications that did not fail totally on the
other criteria and could be representing an emergence of
a popular new method (these were EOCA and TraceR).
This resulted in a shortlist of 35 techniques. On a technique-
by-technique basis, all the relevant papers identified were
collated. From each paper describing or reviewing the method in
theory or practice, any data that would guide the selection in
terms of resources, constraints and outputs were recorded. These
were broken down to clusters of information on time and nature
of expertise. Also, being a critical aspect of the method, the
nature of the information was analysed to identify generic cate-
gories. These included consideration of which aspects of the
process were required – for example, data on the physical or
cognitive aspects of the task. This also concerned the nature of
the situation under assessment; whether, at the start of the
analysis, there was only information on the ‘‘normal’’ running of
the process, whether this was focussed entirely on an abnormal
situation or whether both aspects were required. Because this
work was aimed at novice users, where a technique could
incorporate many of these pieces of information or run
adequately with minimal input, each technique was interpreted
and categorised in accordance with its simplest form. For all of
these features, All variance or discrepancies were noted and
recorded – resolving where necessary through checking consis-
tency with the original ‘‘first published’’ version of the technique.
The data for the technique were then positioned in each part of
the framework as shown in Tables 2–8.
Because of the delay between this initial review and the decision
to publish the framework, this review was updated (performed
June 2007) to ensure that this had included more recent techniques
and method/review papers. Therefore this review used 176 search
terms – due to the author’s increased awareness of additional
techniques. This time the focus was moved to focus on the open
source databases that were well used and trusted within healthcare
– Ovid incorporating Embase and Medline. Over 8000 references
were identified in this search – shortened to 2516 for initial rele-
vance to the topic area, finally identifying 80 papers highlighted as
having potential relevance to providing information that would
support novice users in healthcare to use the techniques (either
techniques reviews or descriptions, or example applications of the
techniques).

Table 1
Brief description of techniques included in the framework.
Technique (acronym and full
name)
AEB (accident evolution and
barrier function model)
APJ (absolute probability
judgement)
ATHEANA (a technique for
human error analysis)
Barrier analysis
CDM (critical decision method)
Change analysis
COGENT (cognitive event tree)
CREAM (cognitive reliability
and error analysis method)
CTA (cognitive task analysis)
ETA (event tree analysis)
FRANCIE (framework assessing
notorious contributing
influences for error)
FMEA/HFMEA (failure/fault
modes effects analysis/
healthcare failure modes and
effects analysis)
FTA (fault tree analysis)
GEMS (generic error modelling
system)
HACCP (hazard analysis and
critical control point)
HAZOP (hazard and operability
analysis/method)
HCR (human cognitive
reliability)
HEART (human error
assessment and reduction
technique)
HRMS (human reliability
management system)
HTA (hierarchical task analysis)
IDA (influence diagram
approach)
ITA (integrated task analysis)
MORT chart analysis
(management and oversight
risk tree)
OCHRA (observational clinical
human reliability
assessment)
M. Lyons / Applied Ergonomics 40 (2009) 379–395 381
Simplified description References
Using a narrative description of an incident, the chain of failure events (errors and failures) and Svenson, 1991, 2001; Svenson et al.,
barriers (technical and human-organisational) that failed are identified. The strength of barriers 1999
and factors impinging on this are analysed to identify weak areas requiring reinforcement
Individual subject matter experts estimate probabilities for a range of specific errors. These are Humphreys, 1988; Kirwan, 1988; Seaver
calibrated through collation of results in a consensus meeting or through mathematical and Stillwell, 1983
aggregation
A team use process information to identify likely human failure events and unsafe actions, Forester et al., 2004; Thompson et al.,
identify the causes of unsafe actions and quantify the human failure events. These values and 1997
incorporated into probabilistic risk analysis
A facilitated group analyse the type of the barriers protecting vulnerable objects from harm Kirwan and Ainsworth, 1992; Lyons
(physical, natural, human action and administrative); assessing their effectiveness and the et al., 2004b; Kecklund et al., 1996
means of improving them
Retrospective interviews with process experts are used to probe non-routine incidents that they Stanton et al., 2005; O’Hare et al., 1998;
had experienced where their expertise was applied; A timeline is created to describe and aid Crandall and Getchell-Reiter, 1993;
system improvement Wong et al., 2000; Hahn et al., 2003
An accident situation is compared with a similar but accident-free situation. The differences Spath, 2000; Kepner and Tregoe, 1976
between the situations are analysed to identify causal factors
Accident scenarios are structured in terms of an event tree analysis of crew decisions and errors, Kirwan, 1998b; Gertman, 1993;
incorporating context through performance shaping factors Gertman et al., 1996
Construction of an event sequence in a specific scenario is used to support description of actions Stanton et al., 2005; Shorrock et al.,
and cognitive activities – the resulting cognitive functions are used to identify error modes 2003; Hollnagel, 1998
A group of process experts are interviewed to collate a description of the actions, critical cues Grunwald et al., 2004; Latorella et al.,
and decisions involved in the process 2001; Militello and Hutton, 1998; Roth
et al., 2001
The events following a single critical initiating event are plotted in a binary tree structure Stanton et al., 2005; Stoykova et al.,
according to different criteria impacting upon the situation 2004; Kirwan and Ainsworth, 1992
Using taxonomies for generic errors, error types and performance shaping factor, a task analysis Ostrom et al., 1997; Haney, 1999
is used to build a specific model of the process and is then analysed through an error event tree
Facilitated group discussed focussed on a graphical description of the product or process is used Kirwan and Ainsworth, 1992; DeRosier
to guide identification of failure modes, listing causes, and assessing the severity of et al., 2002; McDermott et al., 1996;
consequences and probability of occurrence. A HFMEA decision tree supports prioritisation of Duwe et al., 2005; Adachi and Lodolce,
actions 2005; Apkon et al., 2004
An undesirable top-level event is analysed for its causes and/or immediate events and Kirwan and Ainsworth, 1992; Kirwan,
constructed in a tree-like structure connected through AND/OR gates. The process is continued 1994; Stanton et al., 2005
until the events and causes have known probabilities to achieve a quantitative value for the
probability of the event
Analysis of the cognitive behaviour demanded by the goals and intentions of a process is used to Luczak et al., 2003; Reason, 1990
identify errors through mismatches between behaviour required and the behaviour exhibited
The process is described in step form to determine ‘‘critical control points’’ that are necessary for USDHHS, 2006; DeRosier et al., 2002
preventing the hazard. The critical limits for parameters under control are established and
procedure for monitoring these and taking action are established
Through a facilitated group discussion, a set of guide-words (e.g. no, less, late) are applied Humphreys, 1988; Kirwan, 1992a;
systematically to parts of a process diagrams to test their impact on the outcome. For each Kirwan and Ainsworth, 1992; Kletz,
outcome, the cause, consequences and required actions are established 1986; Redmill et al., 1999; Swann and
Preston, 1995; Swuste et al., 1997;
Kennedy et al., 2000; Kennedy and
Slater, 2005; Shorrock et al., 2003;
Stanton et al., 2005
The probability of diagnostic non-response over time is established based on three sets of time- Humphreys, 1988; Kirwan, 1994;
response curves corresponding to the skill, rule and knowledge behaviours required in the Hannaman et al., 1984
specific scenarios
Error probabilities associated with generic tasks are weighted against error producing Humphreys, 1988; Kirwan, 1994;
conditions and their judged effect to produce error probabilities Kirwan and Ainsworth, 1992; Williams,
1986; Farmery and Dean, 2006; Stanton
et al., 2005
Using expert judgement, error descriptors and performance shaping factors are used to modify Kirwan, 1997
the human error probability
Tasks are broken down into progressively more detailed subtasks and structured in Shepherd, 2001; Stanton et al., 2005
a hierarchical manner. Plans for carrying out the subtasks to achieve the higher-level tasks are
established
Through a group discussion, the effects of a number of contributory factors and human actions Humphreys, 1988; Kirwan, 1994
are modelled and quantified in terms of their impact on the outcome of a situation. The
influencing factors can include unsafe acts, performance shaping factors, organisational and
policy factors
Information on both physical and cognitive processes in a task is structured into a goal-oriented Kallus et al., 1998
task description
Using a toolbox of techniques, a fault tree type structure is applied to a scenario to identify the Whitaker-Sheppard and Wendel, 1996
causes in terms of system failures, organisational failures and precursor events
Error modes are used to guide task analysis and classify observational identification of errors in Tang et al., 2005
clinical tasks
(continued on next page)
382 M. Lyons / Applied Ergonomics 40 (2009) 379–395

Table 1 (continued )

Technique (acronym and full Simplified description References

name)
PC (paired comparisons)
Petri-nets
SHERPA (systematic human
error reduction and
prediction approach)
SLIM (success likelihood index
method)
Sneak analysis
SRK-approach (skill, rule,
knowledge)
Task analysis
TESEO (tecnica empirica stima
errori operatori)
THEA (technique for human
error assessment)
THERP (technique for human
error rate prediction)
Timeline analysis
TraceR (technique for the
retrospective analysis of
cognitive errors)
WSA (work safety analysis)
Expert judgement is used to generate estimated probabilities of errors which are then calibrated Humphreys, 1988; Kirwan, 1994
using real data
All the states and transitions occurring when a system changed from a normal to abnormal Kontogiannis et al., 2000; Johnson, 1995
situation are modelled graphically to produce a causal pathway with error consequences and
recovery mechanisms
A facilitated group uses a task analysis, then classifies the tasks and applies error modes (not, Humphreys, 1988; Kirwan, 1992a;
done, too little) to identify errors. Consequences and means of recovery are then established Harris et al., 2005; Stanton and
Stevenage, 1998; Stanton, 1997
Expert judgement is used to rate importance to derive an index of error probability based on Humphreys, 1988; Kirwan, 1988;
performance shaping factors to quantify human error probabilities Kirwan, 1994
A description of the system is used to identify ‘‘sneak conditions’’ which are conditions other Kirwan, 1998b; Hahn and deVries, 1991
than those intended. Barriers to prevent these sneak conditions from occurring are identified
A situation is analysed using a flowchart to establish psychological error mechanisms and Kirwan, 1992a
specifically relate these to error modes
Task analysis includes a range of techniques that provide a description of the actions involved in Kirwan and Ainsworth, 1992; Pines and
carrying out a process Goldberg, 1992; Slagle et al., 2002
A model is constructed based on task description, time available and operator and Humphreys, 1988; Bello and Colombari,
environmental characteristics to establish the probability of human failure in the task 1980
A hierarchical task analysis is carried out for a set of scenarios. Using this, error analysis is driven Stanton et al., 2005; Pocock et al., 2005
by a set of questions to reveal weaknesses to cognitive failures that are inherent in the process
design
Task analysis is used to develop event trees. Error quantification is achieved through the use of Kirwan, 1988; Swain and Guttman,
nominal human error probabilities that can be inter-related and shaped by the relative effects of 1983; Luczak et al., 2003
performance shaping factors
The times of tasks are added to a task analysis to establish contingencies and overlaps between Kirwan and Ainsworth, 1992
tasks and their associated risks
External error modes and performance shaping factors are applied to a task analysis to generate Stanton et al., 2005; Shorrock et al.,
credible error models. Internal error models and psychological error mechanisms are classified 2003
leading to the identification of error recovery steps
A multidisciplinary team analyse a task description to identify known hazard types, possible Kirwan and Ainsworth, 1992
causes and the factors that contribute to them – the focus is on physical and human subsystems
in order to reduce risk probability and consequences

Several further techniques were identified during this phase:
integrated task analysis (Kallus et al., 1998) is a useful tool that
could be incorporated within other techniques to provide both
cognitive and physical information on a task; HACCP, developed
from the food industry and rapidly being applied in healthcare
(Palmer, 1997), to clinical engineering (Hyman, 2003) and to food
hygiene in clinical settings (Sneed et al., 2007; Bas et al., 2005;
Askarian et al., 2004); HFMEA (Linkin et al., 2005) applied to ster-
ilisation and use of surgical instruments) and OCHRA (based on
SHERPA; Tang et al., 2005) applied to laparoscopic cholecystectomy
(Tang et al., 2004). However, this time, error of commission analysis
(EOCA) was removed from the list – due to a continued lack of
presence in the published literature (Kirwan, 2008).
Thus the information on resources, constraints and outputs was
extracted in the same way and the tables were updated accordingly.
The techniques are shown in Table 1 (including the acronym
interpretations as well as the references that were used to populate
the tables for each technique respectively).
4. Results
These results aim to provide a literature-based framework
towards the development of a protocol for a novice user in the
healthcare sector in selecting a predictive safety technique. This
framework aims to limit the choice of feasible techniques to
a shortlist, so it is easier to identify a final technique. The overall
protocol is shown in Fig. 1 with reference to the details of the
sections required in the text.
5. Resources and constraints
5.1. Personnel and expertise
A number of personnel are required to carry out an analysis of
any type. To be more specific, the two types of expert can be
categorised as safety or human factors experts and subject matter
or process experts, as defined in the sections below.
5.1.1. Safety or human factors experts
In this instance, the expert is likely to have a more detailed
understanding and experience of the technique being undertaken,
its purpose, method and expected results, how this compares to
other techniques and its strengths and weaknesses.
This knowledge should be used alongside an appreciation of
scientific concepts such as research methodology, statistics, validity
and reliability and the requirements for good psychological and
human factors expertise such as objectivity, biases and heuristics,
ethics and confidentiality as well as understanding of mathematical
concepts such as probability and Boolean logic.
It is essential that the team is lead by one who is carrying out
an open-minded evaluation of the system and its processes
rather than one who would use the analysis as an opportunity for
blame or manipulating the technique to promote their own
agendas. For this reason, the experts are often independent of the
system. If this is not the case, it must be stressed that the
individual makes significant efforts to maintain an objective
flexible view of the system as well as a willingness to challenge
the status quo.
5.1.2. Subject matter experts
In the healthcare sector, the experts on the technical process
under evaluation could be nurses, doctors, radiographers, phar-
macists, receptionists, cleaners, porters and all aspects of regula-
tory, quality or other management roles. That is, anyone who may
be involved or affected by the process that is being studied for its
human error potential. In some cases, it may not just be experts
who are required to participate – in some cases, it may be desirable
to involve novices in these roles to highlight the problems they
experience with a process that an expert may no longer perceive. In
all cases, it is useful to have more than one representative of each
 M. Lyons / Applied Ergonomics 40 (2009) 379–395 383

Table 2
Personnel required to lead the techniques.

People leading the technique – Technique (acronyms, full Notes from literature
numbers and expertise definition and references used
to populate table)
One leader – no predefined
experience or specific
training
Change analysis A facilitator is required
ETA This requires very little training
Timeline analysis Minimal training is required
HEART Personnel/expertise requirements are low – though the assessor may require training on the HEART
technique
HACCP This is simple enough to be covered in a 1 day/12 h training courses. Generally this is performed in an
inspection type role. No criteria have been given for expertise in use outside the food industry
TESEO One analyst is required with only minimal training needed
HCR Training requirements are low – experts are not required
One leader with independent
view of the process/task
analysis provided
HTA This is simple to learn and carry out, requires only the ability to independently appraise the task
FRANCIE This is either used by a task expert/error analyst with input from a task expert or analysts who need to
perform a task analysis
WSA An independent analyst is required
One leader with some
understanding of safety
terminology
THEA HRA expertise is not necessary for using THEA though it is suggested the terminology in the error analysis
aspect could pose problems. This is designed for systems engineers
THERP Whilst experts aren’t required as the technique can be learnt from the handbook, some training is desirable.
For Kirwan (1998b), all of the assessors were safety assessors with several years experience
Two leaders – no predefined
experience or specific
training
Task analysis This is very easy for the novice, requiring little training. Usually one to two observers are used for inter-
reliability
FMEA/HFMEA It is simple and little training is required. One or two leaders are suggested, for example, a team leader and
scribe. It may be desirable to have quality manager as one of the leaders
HAZOP Generally two independent people are needed to lead the team, chairperson and secretary/ study leader and
human factors expert. It is desirable to have at least one human factors expert – in some case two safety
engineers or HF experts
One leader with safety or
human factors expertise
recommended
ATHEANA HRA analyst is recommended as leader though human factors experience not specified though facilitator-led
quantification is recommended
CREAM This requires knowledge of human factors and cognitive ergonomics and may appear complicated and
daunting for a novice user – usually one analyst leads
HRMS This requires a HRA expert to implement and manage
MORT chart analysis Team leader of a safety expert is suggested
SHERPA This can be done on individual basis, no previous experience required. It is easy to learn as undergraduates
have been used in reliability studies with minimal (unspecified length of time) training though the assessor
must be familiar with psychological error mechanisms. It has been learnt in 3 h
OCHRA Whilst a relatively new technique and not yet tested in different applications, being like SHERPA, it should
require little HRA training
Sneak analysis This requires human factors expertise
TraceR The lead user should have a sound understanding of psychology. A single analyst is usual
Two leaders – one with safety
or human factors expertise
recommended
APJ A group is recommended including an ergonomist and led by a facilitator though could also be done by
a single expert assessor with normative expertise
Barrier analysis For the prospective technique, there is a need for safety and HRA experts. Kecklund et al. (1996) showed
safety barrier function analysis required a process engineer and two HF specialists. Lyons et al. (2004b)
showed this was feasible with one leader with HRA experience
ITA Two observer/interviewers were required with training in the technique
Leaders experienced in the
technique recommended
AEB There is expectation that users would require considerable training in the technique – human factors
specialists are desirable though psychology and engineering students have also done this as part of research
project highlighting a need for both to contribute to the final analysis
CDM Analyst requires great skill
COGENT This requires significant analytical judgement
CTA One expert in CTA and with subject matter expertise leads the study, supported by two observers
FTA A high level of training required if it is to be used quantitatively
GEMS An appreciation for the psychological contribution to error would be recommended in the leader
Petri-nets This is expected to be used by analysts with safety expertise – this may also require complex understanding
of logic and mathematical formula
(continued on next page)
384
Table 2 (continued )
People leading the technique – Technique (acronyms, full
numbers and expertise
Table 3
Personnel required to participate in the techniques.
Numbers of participants
Potentially none other than
the analyst – data via
other means
Participants of specified
numbers
May be involved
individually
Must co-locate as a team
Team is required – numbers
unspecified in the
literature
M. Lyons / Applied Ergonomics 40 (2009) 379–395
Notes from literature
definition and references used
to populate table)
SLIM The minimum requirement is one human factors expert and one PRA expert. It is recommended to have an
experienced (10 yearsþ) human factors professional, safety professional and an independent facilitator for
the process
SRK-approach This is defined as resource-intensive
PC Four reliability analysts and two ergonomists were used in one study. Another study suggested only minimal
training requirements were needed and there is no requirement for understanding of probability theory or
mathematical/statistical concepts
IDA Roles included group consultant and facilitator, technical moderator, probabilistic risk analyst, reliability and
systems analyst, and a human reliability specialist
Technique Notes from literature
HCR This is not required – but simulation data is required
HRMS This is not required once a system has adequate human error data
TESEO No experts are required
HEART There is no requirement for judges
SHERPA The assessor needs to be familiar with the task or be provided with a HTA prior to analysis
FRANCIE This is either used by a task expert, an error analyst with input from a task expert, or analysts who additionally need to
perform a task analysis
GEMS Either an analyst with insight into the tasks or a subject matter expert is required
HACCP Domain experts are required – though the analyst could be a subject matter expert. A team is preferred
Sneak analysis This is an analyst driven approach
Tracer This is an analyst driven approach
THERP This is an analyst driven approach
CREAM This is an analyst driven approach
COGENT This is an analyst driven approach
SRK-approach This is an analyst driven approach
Task analysis Three operational experts of varying experience are required as a minimum
CTA Numbers of people used in other studies include five, eight and 12. In one example, two or more subject matter experts were
interviewed and observed
CDM Four to 17 subjects are shown in the literature. One study had six subject matter experts with 12–20 years experience and
one novice
ITA Six subject matter experts was found to be adequate to get convergence in this study
APJ Numbers used in other studies are three, four, six and 13 – a group is required. One study used two personnel with 5–35 years
domain experience and a safety assessor with knowledge of the plant
SLIM Group constituents vary but they must co-locate for the process. Minimum requirement is one operational expert with
a minimum of 10 years experience, it is recommended to use two personnel with plant experience (10–35 years)
HAZOP Groups of three, four, six people lead by a chairperson have been suggested through previous studies. These recommend
a team of safety, design and operations personnel. For design studies, representatives from the design team and the eventual
users of the system are recommended
ATHEANA A multidisciplinary team is required – literature describes this as similar to a HAZOP team. There are strong
recommendations for domain expertise (training and operational)
Barrier analysis This requires a hybrid team including personnel with operational experience with numbers from 3 to 27
Change analysis This is not specified – but it is likely to be similar to the requirements for barrier analysis
FMEA/HFMEA This recommends four to eight people who are familiar with the product or process. Six to 10 people are recommended for
a HFMEA
IDA Four to eight experts have been used – four should be considered a minimum
PC Eight were used in one study, Preferably more than 10 assessors are used
WSA This recommends a large number of technical personnel but should include management, safety professional and other
workers. No numbers are specified but a team approach is recommended
HTA Numbers depend on the task – this requires expertise of the task to provide a description
Timeline analysis Domain experts may be required for data on task and timing
ETA There is a need for subject matter experts but the numbers are not specified
FTA Consultation with subject matter experts are required though numbers are not specified
AEB Numbers are not specified though a group discussion is desirable
MORT chart Subject matter experts are required and a team approach is recommended
analysis
OCHRA OCHRA requires significant subject matter expertise but there are too few published articles to define numbers
Petri-nets Numbers are not specified though this is expected to be used by analysts working in parallel
THEA This recommends input from subject matter experts but no numbers are specified
 M. Lyons / Applied Ergonomics 40 (2009) 379–395 385

Table 4
Time required for techniques.

Technique Notes from literature

Task analysis This is considered quick and easy to apply
Timeline analysis This is considered quick and easy to apply
TESEO This is considered quick and easy to apply
FMEA/HFMEA This is not specified in the literature though due to its popularity in healthcare, this implies it is both low for training and low for application on tightly
defined problems
HCR This is quick and easy though simulation data must be supplied
WSA This can be carried out as a quick approach
Barrier analysis Time is not indicated and can be considered to probably be low
Change analysis Time is not indicated and can be considered to probably be low
HACCP Time is not indicated and can be considered to probably be low
THEA Stanton et al. (2005) defined the time requirements as low for training and medium for application
ETA Stanton et al. (2005) defined the time requirements as low for training and medium for application
SHERPA This was defined the time requirements as low for training and medium for application though can be long and tedious for complex tasks
OCHRA Whilst a relatively new technique and not yet tested in different applications, based on Tang et al. (2005), it appears that OCHRA will require minimal
training to use as with its origins of SHERPA but significant time may be taken in reviewing the observational recordings
FTA Stanton et al. (2005) suggest no data available on training or application times and timing would depend on the complexity of the problem though they
conclude this as low for training and medium for application
AEB Training can be achieved relatively easily – through self-learning over less than a week. Carrying out the analysis can take as little as 40 min depending on
the task
HAZOP Swann and Preston (1995) estimated each of the 17 deviation guidelines would take 10 min to discuss – so the number of process operations to be
examined will determine the timing
Kirwan and Ainsworth (1992) suggest that studies of small systems take several days whereas for a large system, this can take months
For Swuste et al. (1997), this took 6 h in total
For Kennedy et al. (2000), 16 h were taken to look at a future Air Traffic Management system
For Kennedy and Slater (2005), the process took 2 days
For Shorrock et al. (2003), it took 1 day per task to be analysed
Stanton et al. (2005) defined the time requirements as low for training and medium for application
ITA In Kallus et al. (1998), this took 5–6 h to collect data from each person (air traffic controllers)
HEART For Kirwan (1988), five scenarios took 3–5 days
Kirwan (1994) considers this a ‘‘quick’’ technique to use.
Humphreys (1988) suggests time resources required are low.
Stanton et al. (2005) defined the time requirements as low for training and medium for application
HTA Stanton et al. (2005) defined the time requirements as medium for training and medium for application
FRANCIE Time is not indicated and can be considered to probably be medium
APJ In one study, 39 conditions (scenarios) were evaluated in a 1 day
Five scenarios involving five people took 15 days in the study for Kirwan (1988)
CDM Stanton et al. (2005) specified the time requirements as medium–high for training and high for application (suggested interviews take 1–2 h and
transcription take 1–2 h though training of experts is time-consuming)
CTA One hour of focussed expertise is believed to require 30 h of effort
Training 2 h minimum, 6 days of data collection – no indication of analysis time (Grunwald et al., 2004)
IDA Five events took 4 days to analyse (Kirwan, 1988)
Kirwan (1994) suggests it will take between one and several days
Humphreys (1988) suggested a high level of time resources are required
One example in a NPP case study, eight domain experts took 4 days to quantify five target events using one generic influence diagram (Kirwan and
Ainsworth, 1992)
PC 63 paired comparisons (21 tasks) were carried out in half a day (ref 5), five scenarios took 10 days in the study for Kirwan (1988)
Tracer Stanton et al. (2005) defined the time requirements as medium for training and high for application, Shorrock et al. (2003) found six to 10 tasks of HRA
took 10 person days
SLIM/SLIM-MAUD 25 complex tasks/60 simple tasks analysed in 1 day
(15 complex and 15 simple tasks <7 h on average) (Humphreys, 1988)
Analysis of 26 human error probabilities took 1 day and in a separate study, five scenarios involving five people took 20 days (Kirwan, 1988)
THERP Five scenarios took 8–30 days in the study for Kirwan (1988)
HRMS This can take 2 weeks to assess a scenario for a design project, though the human error identification, representation and quantification phase is said to
take 1 or 2 days – JHEDI developed as quicker technique (Kirwan, 1994)
CREAM Stanton et al. (2005) defined the time requirements as high for training and high for application
ATHEANA This is time-intensive due to comprehensiveness
MORT chart This is considered to be time consuming
analysis
Petri-nets Higher training requirements are required for their use
Sneak analysis This is resource intensive
SRK-approach This is resource-intensive
COGENT Time is not indicated and can be considered to probably be high
GEMS Time is not indicated and can be considered to probably be high

role to ensure the opinion is as representative of the staff as much
as possible – and not merely representative of an individual.
In some cases, the subject matter expert will be called upon to
give ‘‘expert judgement’’ on the probabilities of failure/error for
a particular process. Forester et al. (2004) highlights the biases
associated with expert elicitation of quantitative values that will be
of value to the novice user.
Humphreys (1988) distinguishes the requirements as follows:
Substantive – the ‘‘expert’’ must know the problem sufficiently
well.
Normative – the ‘‘expert’’ (alone or with the aid of a facilitator)
must be able to accurately translate this expertise into
probabilities.
If there is a need to consider the safety of a totally new role
within a system, there may be no substantive experts to consult. For
386 M. Lyons / Applied Ergonomics 40 (2009) 379–395

Table 5
Selection of techniques based on nature and aspects of information.

Information KNOWN to the analyst can be described as: The NORMAL/expected Normal/expected One or more
operations of the operations and abnormal ABNORMAL
process (X) situations and the link situation(s) (Y)
between them (X and Y)
The physical contribution of the human to the working of the system – what APJa APJa APJa
the worker DOES (A)
HTA PCa PCa
PCa
Timeline analysis
SHERPA/OCHRA
The cognitive contribution of the human to the working of the system – COGENT CDM
what the worker THINKS (B)
CTA
CDM
The physical and cognitive contribution of the human to the working of the HCRa GEMS
system – what the worker THINKS and DOES (A & B)
ITA HCRa
SRK-approach SRK-approach
The physical contribution of the human and the hardware to the working of APJa APJa AEB
the system – what the worker and non-human components DO (A & C)
Pro-active barrier analysis Change analysis APJa
Event tree analysis MORT chart analysis FTAa
FMEA PCa IDAa
HACCP Petri-nets PCa
HAZOP Reactive barrier
analysis
HTA
PC3
SHERPA
Sneak analysis
Timeline analysis
WSA
Both non-human and total human (physical & cognitive) contributions to APJa APJa APJa
the working of the system – what the worker THINKS and what the
worker and non-human components DO (A, B & C)
CREAMa PCa ATHEANAa,b
Proactive FRANCIE THEA CREAMa
HEARTa,c THERPa,c Reactive FRANCIE
HFMEA HEARTa,c
HRMSa,b PCa
ITA SLIMa
PCa Tracer retrospectiveb
SLIMa
TESEOc
Tracer predictiveb
a
Indicates that statistical data are required as part of the process but is not already an integral part of the technique – thus there must be simulation data, data from real
operations or sufficient expert judgement to provide this. Further to this, Kirwan (1994) suggests calibration data are required for paired comparisons and SLIM (though no
requirement is specified for APJ, THERP, HEART, IDA and HCR).
b
Is used for techniques that are not generic and have been published only as being applied in one technical non-healthcare domain (and by this fact, may imply a degree of
domain-specificity that is too difficult to replicate in healthcare).
c
Refers to techniques that have statistical data, timings or weighting factors as an integral part of the technique therefore may represent a domain-specificity in terms of the
quantitative values – therefore whilst the technique could be used in healthcare, the probability values and weighting may have to be re-calibrated, (e.g. TESEO uses timing
data from nuclear power plants).

example, if the new system requires a team where nurses fulfil
a new role by carrying out surgery, there may be no nurses available
to feed back on the risks associated with this or other members
of the team who can report on changes to their role as a response
to this.
An issue related to the choice of experts is the need for technique
training. Some techniques are believed to require ‘‘expertise’’ – though
defining what an expert is often as difficult as determining when one
has ‘‘mastered’’ a surgical technique. Some techniques are simple
enough to allow novice users to act in the role of the safety expert with
only a limited amount of training.
The numbers of each type of person may determine which
technique is available for use – some examples of previous research
and recommendations for personnel demands of each technique
(type and number) are shown in Tables 2 and 3.
In general, the requirements for numbers of personnel vary from
technique to technique as well as what investment of time they are
required to commit. As a general rule, the more experts who are
required to be involved, the longer it may take to commit to an
agreement. One consideration that should be taken into account
that some techniques are computer-based and to adapt them for
the healthcare context will also require the time and effort of
programmers and software developers.
5.2. Time
Whilst one would hope that any technique used is provided
with adequate time to complete the study, knowing how much
time is required for a process is one possible means of determining
the chosen technique.
For each technique, time is required to a different degree in
different parts of the study. If each technique is split into the stages:
preparation, implementation and analysis of results – some tech-
niques require considerable preparation for a study but then the
 M. Lyons / Applied Ergonomics 40 (2009) 379–395 387

Table 6
Additional equipment and computer software required for techniques.

Technique Equipment necessary Optional equipment Software necessary Optional software

CDM (Stanton et al., 2005) Yes – audio recording No No No
ETA (Stoykova et al., 2004) No May require video/ No Yes – SMLtreeÔ
audio recording for
observation (Stanton
et al., 2005)
FRANCIE No No No Yes – software available
FTA (Zeng and Okrent, 1991; Dehlinger and Lutz, 2006) No No No Yes – IRRAS – fault tree
quantification software
PLFaultCat
HAZOP (Swann and Preston, 1995) No No No Yes – for recording the data
HTA (Kirwan and Ainsworth, 1992) No No No Yes
HRMS (Kirwan, 1994) No No Yes Yes – JHEDI
IDA (Kirwan and Ainsworth, 1992) No Calculator may be No No
useful for calculating
probabilities
OCHRA (Tang et al., 2005) Yes (video recording) No No No
PC (Humphreys, 1988) No No No Yes
Petri-nets (Kontogiannis et al., 2000) No No No Yes – ARTIFEX
SLIM (Humphreys, 1988; Kirwan, 1994, 1988) No No No Yes – SLIM-MAUD
SNEAK analysis (Hahn and deVries, 1991) No No No Yes
TESEO (Bello and Colombari, 1980) No No No Yes
THEA (Pocock et al., 2001) No No No Yes – ProtoTHEA
WSA (Kirwan, 1994) No Yes (video-recording) No No

rest of the study is relatively quick to complete. For example,
consider computer simulation techniques that test design changes
on outcome – the structure of the simulation would require effort
in programming, whereas running the simulation with different
parameters could be very quick. In contrast, brainstorming may
take very little preparation other than arranging for a group of
experts to work together but making sense of the results may take
considerably longer.
In the literature, there seem to be two ‘‘schools of thought’’ on
how to measure the time required by techniques. The first is
essentially based on ‘‘ordinal measures’’ of time required – where
technique A is more time intensive than technique B for any process
studied though it would not be possible to determine exactly how
long either technique would take. In this instance, actual time
required is contingent on the process under assessment, the level of
detail required and the expertise of the users and therefore can not
easily be specified in exact quantitative terms (Stanton, 2008). One
could consider scenarios where a process was assessed using only
the highest level of its task analysis and another where it had been
analysed to a fine level of granularity with six layers of hierarchical
structure – even with this being the same process to be analysed,
any error analyses of these two scenarios would have significantly
different time requirements. The second school of thought
describes time requirements in terms of actual quantitative terms –
suggesting that, despite the differences, many techniques generally
fit within a specific time-frame and can be categorised accordingly.
Where the literature has outlined the time taken for use of
techniques, whether ordinal and relative or specific and quantita-
tive, they are shown in Table 4.
Assuming adequate timing will be provided for whichever
technique is used, it is the nature of the task (and in many cases, the
people involved) that will determine which technique is optimal.
Timings shown are usually to determine application of completed
techniques as they stand so do not take into account the timing
required to develop or adapt these techniques for healthcare or the
collection of additional information.
5.3. Information
‘‘What you have’’ can be a key factor introducing opportunities
supporting the use of some techniques and limitations that bar the
use of others. The techniques themselves will often provide guid-
ance towards collecting more information as required and thus will
generate more data, perhaps involving interim phases that struc-
ture the data in an alternate form, for example, task analysis. This
may make the technique seem more daunting but often these
initial phases of the technique give a great insight into the under-
standing (or lack of understanding) of the technical process and the
risks associated with it. These include observation, verbal protocol,
structured interviews, questionnaires, activity sampling, talk-
through, walk-through, table-top analysis and critical incident
analysis which will require time, personnel and equipment above
and beyond those required by the techniques themselves. Some of
these are reviewed in Kirwan and Ainsworth (1992). Table 5 is
a simple ‘‘rule of thumb’’ for choosing the most appropriate tech-
nique based on the types of information available to the analyst.
There is a degree of fuzziness to this table and technique placement
is based on the closest category of information to the description of
the technique. Where techniques are sufficiently broad in their
scope of use to cover several categories, they are placed in the
category representing the ‘‘lowest common denominator’’, speci-
fying the lowest complexity of information required (i.e. higher up
the Table 5). Where techniques are idiosyncratic and cover only
specific parts of the table (e.g. APJ and PC), they are included in only
the relevant cells.
To identify whether the information on the task available is
physical/cognitive/non-human or includes a good level of detail on
all aspects is key to deciding which technique is appropriate.
Naturally, if all information is available, it greatly increases the
scope of techniques available to the analyst.
The columns describe the nature of the information available.
For example X: ‘‘do we know more about the normal and expected
running of a process and want to know what potentially can fail
with the process?’’ OR Y: ‘‘do we know more about an incident –
real or imaginary – and want to speculate on how it could be
prevented?’’.
In many cases, the technique will start with one of these types of
information and aim to capture the nature of the other – in some
cases, both types of information will be required at the beginning of
the method. Note that the techniques in Y are typically retrospec-
tive incident analysis techniques that may be used prospectively by
imagining accident scenarios.
388 M. Lyons / Applied Ergonomics 40 (2009) 379–395

Table 7
Outputs from interim phases of analysis techniques.

Technique Output

Task Context description Cognitive model Scenario Task analysis Timeline/ Event tree Fault tree System state
description (e.g. PSF/EPC) description timeline representation
analysis

Description of output:

Description Description of the Description of how Description A specific type A task/ A tree-like A tree-like structure Diagram showing
of the tasks context in which the person/people of of task process/ structure showing a (root) how the system
involved in the task is taking involved is thinking a scenario description scenario showing how top event with state changes (e.g.
the process place (may use through a process on which to description events could multiple branches from operational
a taxonomy to apply the carried out progress from to signify possible to non-
describe this) technique with respect other events causes operational)
to timings
AEB No Yes No Yes No No Concept No Yes (outcome)
similar to one
chain of an
event tree
APJ Yes Yes No Optional Optional No No No Optional
ATHEANA No Yes (error-forcing Yes (generic) Yes No No No No Yes
contexts)
Barrier analysis No Yes (environmental No Yes No No No No – could support No
conditions and (sequence FT construction
vulnerable objects) of events
leading to
accident)
CDM Yes (as part Yes but not in terms Description of Yes Yes (cognitive Yes No No No
of scenario) of categories cognitive processes task analysis) (sequences of
– not one explicit events in
model timeline – not
full analysis)
Change analysis Yes Yes No Yes No No No No No
(minimal
description
required)
COGENT No No Yes Yes No No Yes No No
CREAM Yes Yes (CPC) Yes (COCOM) Yes Yes (HTA) No Yes No No
CTA Yes No – No – No No No No
(cognitive)
Event tree Yes (as list No No Yes (chain Not explicitly – No – No Yes (outcome in
of tasks) of events list of tasks terms of
leading to sufficient consequences)
outcome)
FMEA/HFMEA Yes Optional No No Could be the No No No Yes (in terms of
form of task failure modes – as
description outcome)
FRANCIE Yes Yes (PSFs) Not inclusive No Yes No No No No
Fault tree No No No Yes (chain No No No – Yes (outcome)
analysis of events
leading to
outcome)
GEMS May be Yes (cognitive error Yes May be May be used to No No No No
used to shaping factors) used to guide process
guide guide
process process
HACCP Yes Implicit in No No Not explicit – No Decision tree No No
description prefers flow used to
diagram establish
description critical control
points
HAZOP Yes No No Yes Optional HTA No No No Yes (as
(through (Stanton et al., consequences)
application 2005
of error recommends)
modes)
HCR Not explicit Not explicit No Not explicit No No but uses No No No
time
information
HEART Yes Yes (through EPCs) No Not explicit May support No No No No
identification
of critical task
HRMS Yes Yes (PSFs) Yes (GEMS) Yes Yes (HTA, No but uses Yes (to describe Yes (in No
sequential task tabular task accident quantification
analysis and analysis sequence) module)
TTA)
HTA Yes No No No – No No No No
 M. Lyons / Applied Ergonomics 40 (2009) 379–395 389

Table 7 (continued )

Technique Output

Task Context description Cognitive model Scenario Task analysis Timeline/ Event tree Fault tree System state
description (e.g. PSF/EPC) description timeline representation
analysis

Description of output:

Description Description of the Description of how Description A specific type A task/ A tree-like A tree-like structure Diagram showing
of the tasks context in which the person/people of of task process/ structure showing a (root) how the system
involved in the task is taking involved is thinking a scenario description scenario showing how top event with state changes (e.g.
the process place (may use through a process on which to description events could multiple branches from operational
a taxonomy to apply the carried out progress from to signify possible to non-
describe this) technique with respect other events causes operational)
to timings
IDA No Yes (influencing No Yes No No No No No
factors)
ITA Yes No Yes No – No No No No
MORT chart No Yes No Yes No No No Yes Yes (as for fault
analysis tree)
OCHRA Yes (HTA) No No Yes Yes (HTA) No No No No
PC Yes No No No Optional No No No No
Petri-nets Yes (only as No (except when No Yes No No A sequence No Yes
actions/ part of scenario) similar to this
messages)
SHERPA Yes (HTA) No No Yes Yes (HTA) No No No No
SLIM Yes Yes (PSFs) No Yes Yes No No No No
Sneak analysis Yes Not explicit No Not explicit No No Yes – stepwise No No
flowchart of
task sequence
SRK-approach Not explicit Not explicit Yes (acquired Yes No No No No No
(normal/abnormal through question
situation?) approach)
Task analysis Yes No No No – No No No No
TESEO Not explicit Not explicit No Not explicit No No No No No
(environment
parameter)
THEA Yes As part of system Yes (acquired Yes HTA No No No No
description through question recommended
approach)
THERP Yes Yes No Yes Yes No Yes No No
Timeline Yes No No Yes Yes – No No No
analysis
Tracer Yes (HTA) Yes (PSFs) Yes (through No Yes (HTA) No No No No
predictive Wickens
information
processing model –
Stanton et al., 2005;
Shorrock et al.,
2003)
Tracer No Yes (PSFs) Yes (through Yes No No No No No
retrospective Wickens
information
processing model –
Stanton et al., 2005;
Shorrock et al.,
2003)
WSA Yes Not explicit No Yes Yes (HTA No No No No
recommended)

The rows describe the aspects of the information available at the
start of the method use, which are categorised as follows:
A. The aspects of the system that are concerned with the physical
actions of the workers – i.e. what they DO and how they DO it –
this could involve ‘‘inserting a cannula’’, ‘‘moving a patient
from ward to ward’’, ‘‘opening the drugs cupboard’’ or
‘‘suturing a wound’’.
B. The aspects of the system that are concerned with the cognitive
actions of the workers – i.e. what they THINK and how this
thinking changes or develops – this could involve deciding
whether a patient needed a different dose of a particular drug,
deciding whether the surgical procedure should be changed
from laparoscopic to open or deciding how and when to discuss
sensitive issues with a patient or their family. If there is an
appreciation of how one task may be considered more complex
for a worker than another task or the worker may be affected
by being in a more stressed state, this type of technique is
appropriate.
C. The aspects of the system that are concerned with the non-
human aspects of the system – e.g. ‘‘how an infusion pump
works once it has been set running’’, ‘‘how the telecommuni-
cations systems work in the hospital’’ to the very simple ‘‘how
a bed works’’. This is applicable if the analysis does not require
understanding of the human aspects of the process but does
require knowledge or understanding of how the physical
system works to achieve the overall system goal.
Based on the annotations in Table 5, another feature of the
information that is important may seem apparent. Some
Table 8

390
Outputs from analysis techniques.

Technique Error modes System/barrier Prioritisation Consequences of Psychological Performance Human error probabilities Process failure/success or Error recovery Error prevention/error
identified (errors of failure identified of errors/ error/failure considerations shaping factors/ risk probabilities opportunities barriers/error probability
commission/ e.g. physical sensitivity (e.g. PEMs) error producing reduction strategies/
violations) e.g. task failure initiating analysis conditions recommendations
not done event
AEB Yes Yes No Yes No Not explicit No No No Yes (Barriers)
APJ (as input) No No No No (as input) Yes (expert judgment) Yes (calculated) No Not explicit
ATHEANA Yes (including errors Usually as input Prioritisation No Yes Yes but not pre- Yes Quantification of Human Not explicitly Not explicitly
of commission) for initiators specified failure events incorporated
and event taxonomy in to PRA
trees (error-forcing
contexts)
Barrier analysis No Yes No Yes (as input) No Not explicit in No No No Not explicit
technique –
explanations for
failure of
barriers
CDM No No No No Yes – guided by Some – No No Identification of Identification of
CDM probes psychological psychologically- psychological vulnerabilities
ones through the cued

M. Lyons / Applied Ergonomics 40 (2009) 379–395

CDM probes alternatives
Change analysis No Yes (difference No Yes Not explicit Yes No No No Not explicit
between accident
and accident-free)
COGENT Yes No No Yes Yes No No No No No
CREAM Yes (including No Yes Yes Yes Yes (CPC) Yes No Not explicit No remedial measures are
intentional unsafe made explicit due to
acts) comprehensiveness of
technique
CTA No No No No Only where Only where No No No No
integrated into integrated into
model model
Event tree Not explicit but No No Yes No No No No Not explicit Not explicit
possible actions for
each task are
identified
FMEA HFMEA Not explicitly unless Yes (failure Yes Yes No No Yes (usually through Yes (calculated from Yes Brainstorm actions to limit
using failure modes modes) (prioritisation expert judgement) expert judgement ratings) (consideration the high-risk failure modes.
creatively to analyse of failures) for detectability HFMEA has decision tree to
human tasks of errors) rationalise next actions
FRANCIE Yes No Not explicit No No Yes (PSFs) No No Yes Yes
Fault tree No As part of scenario Not explicit Yes (as input to No Not explicit in No Yes (when quantifying) No Not explicit
analysis but can be top level event) technique
used
following
analysis
GEMS Yes (including rule No No Yes (slip, lapse) Yes Yes (cognitive) No No Yes No
violations)
HACCP No Yes Yes (through Yes No No No No Yes (through Not explicit
CCPs limits) monitoring
CCPs)
HAZOP Not obligatory Yes (failure modes No Yes No Not explicit No No No (recovery Remedies identified through
(unless as input for through pathways may team discussion
human HAZOP) guidewords) be incorporated
into HTA)
HCR Focus on non- No Yes No No Yes (PSFs) Yes No No No
response
HEART No No Yes No Yes Yes Yes (calculated) No No Yes (remedial measures)
(Screening)
HRMS Yes – question/ No Yes – No Yes Yes Yes No Yes (in Yes
answer based sensitivity representation
concept analysis phase)
HTA No No No No No No No No No No
IDA (as input) ‘‘unsafe (as input) ‘‘failure Yes (through (as input) No Yes (through No Yes (quantitative risk and Not explicit Not explicit
acts’’ modes’’ expert expert influences upon it)
judgement) judgement)
ITA No No No No No No No No No No
MORT chart Yes (oversights and Yes (using barrier Yes (expert (as input) Minimal Some No Only as part of fault tree Not explicit Highlights managerial
analysis omissions) concept) judgment) consideration consideration aspect aspects for potential solutions
for (maintenance,
motivational supervisory
aspects control)
OCHRA Yes No Yes – by Yes No No Probability data may be Probability data may be No Yes – strong focus on training
criticality (consequential/ obtained through obtained through
inconsequential successive observations to successive observations to
errors) be applied in the be applied in the
technique technique
PC Obtained prior to the No No No No No Yes Where these are the same No No
analysis as HEP

M. Lyons / Applied Ergonomics 40 (2009) 379–395

Petri-nets No Yes (system stage No (though Yes No Only where part No No Yes (error Preventative measures
changes) highlights of scenario recovery paths) emerge but not explicit
dependency)
SHERPA Yes (through No Yes – by Yes No No Ordinal probability only No Yes Yes (remedy analysis)
taxonomy) criticality
assessment
SLIM No No Yes No No Yes Yes No No Yes
Sneak analysis Yes (including errors Yes No Yes No Yes No No Yes Yes (design deficiencies)
of commission)
SRK-approach Yes No No Yes Yes Question based No No Yes No
Task analysis No No No No No No No No No No
TESEO No No Yes No Quantification Limited Yes No No No
of time stress
only
THEA Yes (including No No Yes Only in As part of the No No No Yes (design remedies)
cognitive errors) relation to system
system description
feedback
THERP (as input) Yes (through Yes No No Yes Yes Yes Yes Yes – through PSFs
dependent
actions)
Timeline No No No No No No No No No No
analysis
Tracer Yes No No No Yes Yes No No Yes Not explicit
predictive
Tracer Yes No No No Yes Yes No No Yes Not explicit
retrospective
WSA Yes (Yes (causative Relative risk Yes (hazard) No No No Yes No Yes (corrective actions)
factors) screening

391
392 M. Lyons / Applied Ergonomics 40 (2009) 379–395
1. Identify need for HRA
2. Identify Resources and Constraints
(what we have & what we do not have)
Personnel and training
HRA HRA Time Information
“experts”, “participants”
HRA Subject matter
process experts
leaders
No
Reduce constraints, techniques
Limit requirements, identified?
Add resources
and reanalyse selection protocol
or
Indicate it is not possible to carry
out HRA in this scenario
Fig. 1. Selection protocol of error prediction techniques.
techniques can be carried out on future systems just by using
speculation and expectation, whereas some techniques require the
timing, failure statistics or other data that can only be obtained
from a current system. Similarly, CREAM, HCR, TESEO, THERP and
timeline analysis specifically require some information on the real
timing involved in tasks or time limits by which point a task should
be completed. Event tree analysis and ATHEANA may incorporate
specific times but this is optional.
In Table 5, it should also be mentioned that some of the tech-
niques have been developed to quite a high level of detail for
application in a particular domain, for example, the nuclear
industry. Therefore, the adaptation required to use these tech-
niques in healthcare must be taken into consideration before their
application.
5.4. Software and equipment
Some require data that are captured using special equipment –
for example, observational techniques that require video-camera to
record the analysis. Some techniques require the use of software –
either for analysis of complex data generated outside of the soft-
ware package or for simulation of the whole technical process to
produce error probabilities independently of expert judgement.
Naturally, if such equipment were unavailable, these techniques
would be considered off-limits for analysing the problem.
However, whether such software is a useful investment for the
current as well as future studies is often a valid consideration.
Whether the software is either useful or feasible for the healthcare
is something that must be considered – either in its current
manifestation or for future developments – i.e. adaptation of soft-
ware from other industries to the health sector. These are shown in
Table 6.
In general, the requirement of software/equipment occurs
mostly for those techniques that are so highly developed, that they
are also quite domain specific and would require considerable
adaptation before they can be applied to healthcare.
3. Identify Requirements
(what we want)
Interim Output
requirements requirements
HRA
software
and
equipment
4. Shortlist
HRA
Techniques
Too many techniques
identified? Tighten constraints;
Add requirements;
Limit resources
and reanalyse selection protocol
or
Refer to literature resources specifying
5. Carry out quality, reliability and other criteria (eg.
HRA technique Kirwan 1992a,1992b, 1997a, 1997b,
identified Humphreys 1988)
6. Requirements
If the resources and constraints have not sufficiently narrowed
the selection, the objective of the analysis should support this goal.
It may be a case that the goal is not clearly defined and there is
merely and intention to understand a bit more about the errors in
the process. For this, safety techniques often have common parts
whereby certain techniques fulfil all or part of the requirements to
a greater or lesser degree.
Knowing whether it is satisfactory to have simply the potential
errors in the system identified or whether quantification is required
is an important consideration in the type of technique to be chosen.
Therefore, Table 7 shows a simple representation of the range of
outputs that will result from the earlier phases of the techniques –
thus allowing a change in direction or the potential for other
techniques to be carried out following the completion of the
technique. A brief explanation of these outputs is shown at the top
of the table. Following this, Table 8 shows the outputs from the
latter parts of the techniques.
There is some flexibility in this that will be undoubtedly
exploited by experts with the techniques. For example, experts may
stretch the interpretation of the more physical failure modes to
indicate the psychological states more consistent with error modes.
However, the table is framed to represented the outputs and
demands of the basic versions of the techniques – and experts may
of course expand them to incorporate other issues if they choose,
those less experienced who only require particular results can use
this to identify whether their chosen technique will fit the bill.
However, the novice should be warned not to aim for the most
complicated technique available to them – and to take the other
factors of time, personnel, etc., into account. A high quality of
information inputted to a highly structured technique does not
always yield as useful results as if part of the same high quality
information was used in a simpler technique.
There are often compromises between these issues – for
example, quality, quantity and time. In this case, may be a trade-off
 M. Lyons / Applied Ergonomics 40 (2009) 379–395
within the resources (e.g. fewer people, more time/more people,
less time) and between the resources and the requirements (e.g.
more breadth, depth or rigor ¼ more time/expertise).
Thus selection using the following method may require an
iterative approach – either limiting or specifying resources or
requirements and tightening the constraints to shorten the list – or
stretching the resources and relaxing the constraints to lengthen
the list.
At this point, the list of techniques best suited to the purpose
should have been reduced. If no techniques remain, it is clear that
the constraints have been too severe for any technique to be
implemented and there will be a need to release them – either by
employing additional expertise, allowing more time or collating
further information. Using Fig. 1, it is recommended that either the
descriptions of the short-listed techniques should then be exam-
ined to select the final choice (using the brief descriptions in Table
1, supported by the references) and ideally referring to the addi-
tional references for relative indications on accuracy, validity and
reliability (Kirwan, 1992a,b, 1997a,b; Humphreys, 1988). These
publications constitute some of the broadest reviews and largest
practical validation exercises of the techniques available. Whilst
few of the short-listed techniques have been tested for validity and
reliability, these papers illustrate comparisons of several tech-
niques in order to support a final decision.
7. Conclusions
Every safety expert started as a novice and usually relied on
mentors to support them in learning the techniques, learning an
extensive number of techniques to ensure breadth and flexibility of
skill. From this point, they limit their toolbox of techniques through
personal experience with (and tailoring of) the techniques, gaining
expertise that is not made explicit in the literature. Healthcare
professionals have limited time and support to devote to education
in predictive safety analysis. Therefore interested parties would
find their progress hindered in carrying out an analysis simply
because they are unable to select an appropriate technique or
invest in the time and effort in learning several techniques to
achieve a toolbox comparable with a professional safety expert.
Through a literature-based review and clustering the constraints
and requirements, this paper has provided an initial framework to
support selection of predictive safety techniques for the novice user
in the healthcare sector. This provides a first step towards
producing a valid and useful means for novice users to choose the
appropriate techniques for solving their problems. In some cases,
the circumstances would still dictate the involvement of a trained
safety professional to achieve a valid outcome; others will allow
knowledge to be gained by a novice user through training or
mentorship with safety professionals.
This review has identified and acknowledged several challenges
for constructing this framework. The literature frequently shows
inconsistencies or lack of specificity for supporting selection. In
many cases, the literature does not even include the information to
make the techniques directly reproducible in a scientific manner.
One of the reasons for this is that many of these techniques are
applied in the commercial sector and therefore documentation of
issues such as the number of people involved, time taken or
information involved are not available in the public domain. More
specifically, it is rare to find publications describing how the tech-
niques vary in practice from the ‘‘pure’’ validated textbook versions.
These factors all contribute to making the field of predictive safety
opaque to novice users as well as review or validation by other
safety analysts.
One example of this issue is the key issues of ‘‘time required’’.
Whether the time required can be represented robustly in terms of
specific time periods or whether it is simply ‘‘the length of a piece
393
of string’’ relating to the complexity of the process analysed is an
important issue for novice users. Without inclusion of this infor-
mation in publications, this adds unnecessary risk for novices who
may choose to replicate the technique – either of non-completion
due to lack of time, or simply weakening a convincing evidence
base that could be used to convince senior management that this is
a valid and cost-efficient approach to improve safety. There is
a notable paucity of data on time taken in analysis and the next step
that should be taken is to focus research in this area – essentially
carrying out a timeline analysis on the techniques for a range of
processes and examining the factors that impinge on these times.
These data could therefore be integrated to form an improved
version of the framework that includes time (means and standard
deviations) required for each type of technique, process and indi-
vidual (for example, analyst or subject matter expert) involved in
the analysis.
Similarly, this paper has not focussed on the next stage of
selection – the consideration of scientific validity and reliability of
the techniques. Again, in many cases, there are methods which are
widely used that have never been validated. There is a similarity
between these issues and that of the concept of an ordinal measure
of ‘‘time requirements’’: the validation studies (for example Hum-
phreys, 1988; Kirwan, 1997a,b) compare a limited number of
techniques to produce comparisons which cannot be extrapolated
to provide an all-encompassing ‘‘interval’’ measure of validity.
Therefore, this paper highlights a need to carry out dedicated
research investigating the resources required (human, informa-
tional and time) and a large-scale validation study for the broad
range of techniques. Whilst there are still areas of knowledge that
rarely permeates the literature, only by taking this first step of
clustering the knowledge in the literature into a framework do we
have an ‘‘artefact’’ that can be then more easily scrutinised, tested
and validated in practice. This leads to a necessary second stage for
the research – to provide a refined framework that is not only based
on the literature – but also based on the knowledge in practice.
This must be accepted as a first step and it is necessary for the
framework to be refined through collation of expert experience and
practical case studies. Acknowledging this challenge as for novice
users within the healthcare sector, this paper could further guide
novice users in other sectors and bring a new characteristic of
transparency and usability to techniques in error prediction.
Acknowledgements
Thanks to Charles Vincent and Sally Adams and The Nuffield
Trust for supporting the literature review phase of the work and to
Professor P. John Clarkson and the Patient Safety Research Portfolio
for supporting the later analysis. Thanks to Professor Neville
Stanton and Dr Barry Kirwan for their advice.
References
Adachi, W., Lodolce, A.E., 2005. Use of failure modes effects analysis in improving
the safety of i.v. drug administration. American Journal of Health-System
Pharmacy 62, 917–920.
Ainsworth, L., Marshall, E., 1998. Issues of quality and practicability in task analysis:
preliminary results from two surveys. Ergonomics 41 (11), 1607–1617.
Apkon, M., Leonard, J., Probst, L., DeLizio, L., Vitale, R., 2004. Design of a safer
approach to intravenous drug infusions: failure modes effects analysis. Quality
and Safety in Healthcare 13, 265–271.
Askarian, M., Kabir, G., Aminbaig, M., Memish, Z.A., Jafari, P., 2004 Jan. Knowledge,
attitudes, and practices of food service staff regarding food hygiene in Shiraz,
Iran. Infection Control & Hospital Epidemiology 25 (1), 16–20.
Bas, M., Temel, M.A., Ersun, A.S., Kivanc, G., 2005. Apr. Prerequisite programs and
food hygiene in hospitals: food safety knowledge and practices of food service
staff in Ankara, Turkey. Infection Control & Hospital Epidemiology 26 (4),
420–424.
Bello, G.C., Colombari, V., 1980. The human factors in risk analyses of process plants:
the control room operator model ‘‘TESEO’’. Reliability Engineering 1, 3–14.
394 M. Lyons / Applied Ergonomics 40 (2009) 379–395
Crandall, B., Getchell-Reiter, K., 1993 Sep. Critical decision method: a technique for
eliciting concrete assessment indicators from the intuition of NICU nurses.
Advances in Nursing Science 16 (1), 42–51.
Dehlinger, J., Lutz, R.R., 2006 Jan. PLFaultCAT: a product-line software fault tree
analysis tool. Automated Software Engineering 13 (1), 169–193.
DeRosier, J., Stalhandske, E., Bagian, B.P., Nudell, T., 2002. Using Health Care Failure
Mode and Effect AnalysisÔ: The VA National Centre for Patient Safety’s Prospective
Risk Analysis System. Journal on Quality Improvement 28 (5), 248–267.
Duwe, B., Fuchs, B.D., Hansen-Flaschen, J., 2005. Failure modes and effects analysis
application to critical care medicine. Critical Care Clinics 21, 21–30.
Everdij, M. 2004. Review of Techniques to Support the EATMP Safety Assessment
Methodology, EEC Note 01/04. Volume II: Annex. Project: SRD-3–E1. Euro-
control Experimental Centre, Bretigny.
Farmery, K., Dean, P., 2006 Mar.. Human reliability analysis for enhanced patient
safety – a pilot study. Health Care Risk Report 12 (4), 12–14.
Forester, J., Bley, D., Cooper, S., Lois, E., Siu, N., Kolaczkowski, A.M., Wreathall, J.,
2004. Expert elicitation approach for performing ATHEANA quantification.
Reliability Engineering and System Safety 83, 204–220.
Gertman, D.I., 1993. Representing cognitive activities and errors in HRA trees.
Reliability Engineering and System Safety 39 (1), 25–34.
Gertman, D.I., Haney, L.N., Siu, N.O., 1996. Representing context, cognition and crew
performance in a shutdown risk assessment. Reliability Engineering and
System Safety 52, 261–278.
Grunwald, T., Clark, D., Fisher, S.S., McLaughlin, M., Narayanan, S., Piepol, D., 2004.
Using cognitive task analysis to facilitate collaboration in development of
simulator to accelerate surgical training. Studies in Health Technology and
Informatics 98, 114–120.
Hahn, H.A., deVries, J.A. 1991. Identification of human errors of commission using
Sneak Analysis. In: Proceedings of the Human Factors Society 35th Annual
Meeting. San Francisco, 2–6 Sep, pp. 1080–1084.
Hahn, B.B., Rall, E., Klinger, D.W. 2003. Cognitive Task Analysis of the Warning
Forecaster Task. Final Report. The National Weather Service, Office of Climate,
Water and Weather Services, Norman, OK, Order Number: RA1330-02-SE-0280.
Haney, L.N.,1999 Jul.. FRamework Assessing Notorious Contributing Influences for Error
(FRANCIE). International Workshop on Human Factors in Space, Tokyo, Japan.
Hannaman, G.W., Spurgin, A.J., Lukic, Y.D., 1984. Human Cognitive Reliability Model
for PRA Analysis. Electric Power Research Institute, Palo Alto, CA, USA.
Harris, D., Stanton, N.A., Marshall, A., Young, M.S., Demagalski, J., Salmon, P., 2005.
Using SHERPA to predict design-induced error on the flight deck. Aerospace
Science and Technology 9, 525–532.
Hollnagel, E., 1998. Cognitive Reliability and Error Analysis Method (CREAM), first
ed. Elsevier Science.
Hollnagel, E., 2004. Barrier Analysis and Accident Prevention. Ashgate,
Aldershot, UK.
Humphreys, P., 1988 Oct.. Human Reliability Assessors Guide. NCSR, AEA Tech-
nology, Warrington.
Hyman, W.A., 2003 Jul/Aug/Sep.. The application of HACCP in clinical engineering.
Journal of Clinical Engineering 28 (3), 158–162.
Johnson, C.W., 2003. Failure in Safety-critical Systems: A Handbook of Incident and
Accident Reporting. Glasgow University Press, Glasgow.
Johnson, C.W., 1995. The application of Petri nets to reason about human factors
problems during accident analyses. In: Palanque, P., Bastide, R. (Eds.), The
Design, Specification and Verification of Interactive Systems. Springer Verlag,
Berlin, Germany, pp. 93–112.
Kallus, K.W., Barbarino, M., Van Damme, D.1998. Integrated Task and Job Analysis of
Air Traffic Controllers. Phase 1. Development of methods.
HUM.ET1.ST01.1000.DEL03. EATCHIP, EUROCONTROL, Brussels.
Kecklund, L.J., Edland, A., Wedin, P., Svenson, O., 1996. Safety barrier function
analysis in a process industry: a nuclear power application. International
Journal of Industrial Ergonomics 17, 275–284.
Kennedy, R.J., Slater, K. 2005, Airspace Complexity and Safety Analysis Hazard and
Operability (HAZOP). EUROCONTROL Experiment Centre, Bretigny. EEC Note
14/06.
Kennedy, R., Jones, H., Shorrock, S., Kirwan, B., 2000. A HAZOP analysis of a future
ATM system. In: McCabe, P.T., Hanson, M.A., Robertson, S.A. (Eds.), Contempo-
rary Ergonomics. Taylor and Francis, London.
Kepner, C.H., Tregoe, B.B., 1976. The Rational Manager, second ed. Kepner Tregoe
Inc., Princeton.
Kirwan, B., 1988. A comparative evaluation of five human reliability assessment tech-
niques. In: Sayers, B.A. (Ed.), Human, Factors and Decision Making. Elsevier, London.
Kirwan, B., 1992a. Human error identification in human reliability assessment. Part
1: overview of approaches. Applied Ergonomics 23 (5), 299–318.
Kirwan, B., 1992b. Human error identification in human reliability assessment. Part
2: detailed comparison of techniques. Applied Ergonomics 23 (6), 371–381.
Kirwan, B., 1994. A Guide to Practical Human Reliability Assessment. Taylor &
Francis, London.
Kirwan, B., 1997. The development of a nuclear chemical plant human reliability
management approach: HRMS and JHEDI. Reliability Engineering and System
Safety 56 (2), 107–133.
Kirwan, B., 1998a. Human error identification techniques for risk assessment of high
risk systems – Part 1: review and evaluation of techniques. Applied Ergonomics
29 (3), 157–177.
Kirwan, B., 1998b. Human error identification techniques for risk assessment of high
risk systems – Part 2: towards a framework approach. Applied Ergonomics 29
(5), 299–318.
Kirwan, B., 2008. Personal Correspondence. 2nd June 2008.
Kirwan, B., Ainsworth, L., 1992. A Guide to Task Analysis. Taylor & Francis, London.
Kletz, T.A., 1986. Hazop and Hazan: Notes on the Identification and Assessment of
Hazards. Institution of Chemical Engineers, London.
Kontogiannis, T., Leopoulos, V., Marmaras, N., 2000. A comparison of accident
analysis techniques for safety-critical man-machine systems. International
Journal of Industrial Ergonomics 25, 327–347.
Latorella, K., Pliske, R., Hutton, R., Chrenka, J., 2001. Cognitive Task Analysis of
Business Jet Pilots’ Weather Flying Behaviors: Preliminary Results, NASA/TM-
2001–211034. National Aeronautics and Space Administration. Langley
Research Center, Hampton, Virginia.
Linkin, D.R., Sausman, C., Santos, L., Lyons, C., Fox, C., Aumiller, L., Esterhai, J.,
Pittman, B., Lautenbach, E., 2005. Applicability of healthcare failure mode
and effects analysis to healthcare epidemiology: evaluation of the steriliza-
tion and use of surgical instruments. Clinical Infectious Diseases 41 (7),
1014–1019.
Luczak, H., Reuth, R., Schmidt, L., 2003. Development of error-compensating UI for
autonomous production cells. Ergonomics 46 (1-3), 19–40.
Lyons, M., Adams, S., Woloshynowych, M., Vincent, C., 2004a. Human reliability
analysis in healthcare: a review of techniques. International Journal of Risk and
Safety in Medicine 16, 223–237.
Lyons, M., Woloshynowych, M., Adams, S., Vincent, C. 2004b. Error Reduction in
Healthcare: Final Report to The Nuffield Trust, The Nuffield Trust.
McDermott, R.E., Mikulak, R.J., Beauregard, M.R., 1996. The Basics of FMEA.
Productivity Inc, Portland.
Militello, L.G., Hutton, R.J.B., 1998. Applied Cognitive Task Analysis (ACTA): a prac-
titioner’s toolkit for understanding cognitive task demands. Ergonomics 41 (11),
1618–1641.
O’Hare, D., Wiggins, M., Williams, A., Wong, W., 1998. Cognitive task analyses for
decision centred design and training. Ergonomics 41 (11), 1698–1718.
Ostrom, L., Nelson, W., Haney, L., Richards, R., Wilhelmsen, C., Owen, R.1997 Oct.
Structured Human Error Analysis for Airplane Maintenance and Design, INEEL/
EXT-97–01093.
Palmer, G., 1997 May.. Hazard Analysis Control moves into healthcare.
Manufacturing Chemistry 68, 5.
Paz Barroso, M., Wilson, J.R., 2000. HEDOMS: A Framework and Toolkit for Analysis
of Human Error and Disturbance Occurrence in Manufacturing Systems, A
Framework and a Toolkit for Practical Analysis. Cognition, Technology and
Work. Springer-Verlag London, Limited, pp. 51–61.
Pines, E., Goldberg, J.H., 1992. Task and reliability analysis of lathe operation.
International Journal of Human Factors in Manufacturing 2 (4), 311–324.
Pocock, S., Harrison, M., Wright, P., Johnson, P., 2001. In: Hirose, M. (Ed.), THEA: A
Technique for Human Error Assessment Early in Design, Human-Computer
Interaction – INTERACT ’01. IOS Press, Amsterdam, pp. 247–254.
Pradhan, M., Edmonds, M., Runciman, W.B., 2001. Quality in healthcare: process.
Best Practice & Research Clinical Anaesthesiology 15 (4), 555–571.
Rahimi, M., 1986. System safety for robots: an energy barrier analysis. Journal of
Occupational Accidents 8, 127–138.
Reason, J., 1990. Human Error. Cambridge University Press.
Redmill, F., Chudleigh, M., Catmur, J., 1999. System Safety. HAZOP and Software
HAZOP. John Wiley & Sons, Chichester.
Roth, E.M., Malsch, N., Multer, J., 2001 May.. Understanding How Train Dispatchers
Manage and Control Trains: Results of a Cognitive Task Analysis. National
Technical Information Service, Springfield, VA. DOT/FRA/ORD-01-02.
Seaver, D.A., Stillwell, W.G., 1983. Procedures for Using Expert Judgement to Esti-
mate Human Error Probabilities in Nuclear Power Plant Operations. NUREG/CR-
2743, Washington DC.
Shepherd, A., 2001. Hierarchical Task Analysis. Taylor & Francis, London.
Shorrock, S.T., Kirwan, B., Smith, E.2003 27–28 Feb. Individual and Group
Approaches to Human Error Prediction – a Tale of Three Systems. IBC Confer-
ence on Preventing Human Errors and Violations, London.
Slagle, J., Weinger, M.B., Dinh, M.T., Brumer, V.V., Williams, K., 2002 May.. Assess-
ment of the intrarater and interrater reliability of an established clinical task
analysis methodology. Anesthesiology 96 (5), 1129–1139.
Sneed, J., Strohbehn, C.H., Gilmore, S.A., 2007 Apr/Jun. Impact of mentoring on food
safety practices and HACCP implementation in Iowa assisted-living facilities.
Topics in Clinical Nutrition 22 (2), 162–174.
Spath, P., 2000 May. Analyse process changes to identify system failures. Hospital
Peer Review 25 (5), 65–67.
Stanton, N.A., 1997. Human error identification: comparing the performance of
SHERPA and TAFEI. In: Robertson, S. (Ed.), Contemporary Ergonomics. Taylor &
Francis, London.
Stanton, N.A., 2008. Personal correspondence. 3rd June 2008.
Stanton, N.A., Stevenage, S.V., 1998. Learning to predict human error: issues of
acceptability, reliability and validity. Ergonomics 41 (11), 1737–1756.
Stanton, N.A., Young, M.S., 2003. Giving ergonomics away? The application of
ergonomics methodsby novices. Applied Ergonomics 34, 479–490.
Stanton, N.A., Salmon, P.M., Walker, G.H., Baber, C., Jenkins, D.P., 2005. Human
Factors Methods A Practical Guide for Engineering and Design. Ashgate,
Aldershot.
Stoykova, B., Dowie, R., Gregory, R.P.F., Rowsell, K.V., Lane, S., 2004. ‘‘Event tree’’
approach for facilitating audit of ambulance cardiac services. Clinical Gover-
nance: An International Journal 9 (2), 115–122.
Suokas, J., 1988. The role of safety analysis in accident prevention. Accident Analysis
& Prevention 20 (1), 67–85.
Svenson, O., 1991. The accident evolution and barrier function (AEB) model applied
to incident analysis in the processing industries. Risk Analysis 11 (3), 499–507.
 M. Lyons / Applied Ergonomics 40 (2009) 379–395
Svenson, O., 2001. Accident and incident analysis based on the accident evolution
and barrier function (AEB) model. Cognition, Technology and Work 3, 42–52.
Svenson, O., Lekberg, A., Johansson, A.E.L., 1999 Nov. On perspective, expertise and
differences in accident analyses: arguments for a multidisciplinary integrated
approach. Ergonomics 42 (11), 1561–1571.
Swain, A.D., Guttman, H.E. 1983. A Handbook of Human Reliability Analysis with
Emphasis on Nuclear Power Plant Applications. NUREG/CR-1278, Nuclear
Regulatory Commission-Washington DC.
Swann, C.D., Preston, M.L., 1995. Twenty five years of HAZOPs. Journal of Loss
Prevention in the Process Industries 8 (6), 349–353 (5).
Swuste, P., Goossens, L., Bakker, F., Schrover, J., 1997. Evaluation of accident scenarios in
a Dutch steel works using a hazard and operability study. Safety Science 26 (1-2), 63–74.
Tang, B., Hanna, G.B., Joice, P., Cuschieri, A., 2004 Nov. Identification and categorization
of technical errors by observational clinical human reliability assessment (OCHRA)
during laparoscopic cholecystectomy. Archives of Surgery 139 (11), 1215–1220.
Tang, B., Hanna, G.B., Cuschieri, A., 2005 Jul. Analysis of errors enacted by surgical
trainees during skills training courses. Surgery 138 (1), 14–20.
Thompson, C.M., Forester, J.A., Cooper, S.E., Kolaczkowski, A.M., Bley, D.C., Wreathall,
J.1997, The Application of ATHEANA. IEEE Sixth Annual Meeting of Human
Factors Meeting, Orlando Florida 9:13–19:17.
395
USDHHS (US Department of Health and Human Services). 2006. Managing Food
Safety: A Manual for the Voluntary Use of HACCP Principles for Operators of
Food Service and Retail Establishments. USDHHS, Food and Drug administra-
tion, Center for Food Safety and Applied Nutrition, April.
Whitaker-Sheppard, D., Wendel, T.1996. Analysis of the Causes of Chemical Spills
from Marine Transportation or Related Facilities. U.S. Coast Guard Research and
Development Center, Report No. CG-D-08–96.
Williams, J.C. 1986. HEART – a Proposed Method for Assessing and Reducing Human
Error. In: 9th ANonEnances in Reliability Technology Symposium (ARTS).
University of Bradford.
Wong, B.L.W., Ng, B.P., Clark, S.A., 2000. Identifying Expertise for CAL Program
Development in Operative Dentistry. Journal of Dentistry: Educational Tech-
nology Section. Elsevier Science.
Wreathall, J., Nemeth, C., 2004. Assessing risk: the probabilistic risk assessment
(PRA) in patient safety improvement. Quality and Safety in Healthcare 13,
206–212.
Zeng, Y., Okrent, D., 1991. Assessment of off-normal emissions from hazardous
waste incinerators. Part 1:assessment of off-normal emission frequency and
duration. Journal of Hazardous Materials 26, 47–62.