EUROPEAN

JOURNAL
OF OPERATIONAL
RESEARCH
ELSEVIER European Journal of Operational Research 103 (1997) 350-372

Supporting a complex audit judgment task:

An expert network approach
J e f f e r s o n T . D a v i s a, A n n e P . M a s s e y b,,, Ronald E.R. Lovell II a
a Department of Accounting, Management Information Systems and Marketing. Clarkson University, Potsdam, NY 13699-5795, USA
b Department of Accounting and lnfi)rmation Systems, School of Business, Indiana University, 1309E. lOth Street,
Bloomington, IN 47405, USA

Abstract

An auditor considers a tremendous amount of data when assessing the risk that the internal control (IC) structure of an
entity will fail to prevent or detect significant misstatements in financial statements. The myriad of relationships between 1C
variables that must be identified, selected, and analyzed often makes assessing control risk a difficult task. While some
general procedures and guidelines are provided, audit standards dictate no specifically set procedures and rules for making a
preliminary control risk asscssmcnt (CRA). Rather, the proccdures and rules are left mostly to auditor judgment. This paper
considers the appropriateness of applying artificial intelligence (A1) techniques to support this audit judgment task. It details
the construction of a prototype expert network; an integration of an expert system (ES) and a neural network (NN). The
rules contained in the ES model basic CRA heuristics, thus allowing for efficient use of well-known control variable
relationships. The NN provides a way to recognize patterns in the large number of control variable inter-relationships that
even experienced auditors cannot express as a logical set of specific rules. The NN was trained using actual case decisions of
practicing auditors. © 1997 Elsevier Science B.V.
Keywords: Expert network; Neural network; Expert system; Control risk assessment; Auditing

1. Introduction ber of variables in a problem increase and the speci-

Many decision-making tasks do not lend them-
selves to formulation through the sole use of quanti-
tative models, nor simple intuitive problem solving
(Rosenhead, 1992). Building and incorporating quali-
tative and quantitative reasoning and modeling into a
decision-aiding system is a challenge for practition-
ers and researchers (Gupta, 1994; Silverman, 1995;
I.iberatore and Stylianou, 1993, 1995). As the num-
"Corresponding author. E-mail: amassey@juliet.ucs.indiana.
edu.
ficity and measurability of their efficacy relation-
ships diminishes, difficulties are exacerbated. The
literature suggests that builders of systems to support
complex decision-making should draw from and at-
tempt to combine a multitude of paradigms (for
example, decision support, neural networks, case-
based reasoning, and models with rules or objects)
(Silverman, 1995; Beulens and Van Nunen, 1988;
Turban and Watkins, 1986; Yoon et al., 1993). This
paper analyses the process of building a prototype
intelligence-based system for application to a com-
plex problem within the field of auditing. The proto-
type system is constructed as a decision-support tool
for auditors analyzing the internal control (IC) struc-

0377-2217/97/$17.00 © 1997 Elsevier Science B.V. All rights reserved.

PII S0377-221 7(97)001 25-2
 J.T. Davis et al. / European Journal of Operational Research 103 (1997) 350-372
ture of a business entity in order to derive a prelimi-
nary control risk assessment (CRA). It is designed as
an expert network that combines two AI paradigms
- expert systems (ESs) and neural networks (NNs).
A review of the literature suggests that AI tech-
nologies such as ESs and NNs have found only
limited use as decision aids in audit practice. More-
over, developers of such systems have generally
relied on singular means of knowledge representa-
tion and reasoning. Intelligence-based systems in
auditing have been primarily employed as traditional
knowledge-based systems (O'Leary and Watkins,
1991; Gray et al., 1991; Eining et al., 1994; Smith,
1996), traditional statistical systems (Bell et al., 1990;
Eining et al., 1994; Scott and Wallace, 1994), and as
stand-alone neural network systems (Bell et al., 1990;
Coakley and Brown, 1993).
This paper forms a part of a wider study into the
increasing use of AI technologies in audit practice.
As for any system, effectiveness and efficiency gains
are not automatic or achievable without users accept-
ing the system as a decision aiding tool. Thus,
system design, development and implementation are
key factors in promoting acceptance (Gillett et al.,
1995). Although design and development include,
for example, traditional system aspects such as the
design of the user interface, etc., it may be that the
characteristics of the underlying system technologies
- here ESs and NNs - also have a major impact on
user acceptance (Davis, 1994; Eining et al., 1994).
More specifically, users such as auditors may be
leery of technology that appears to take the decision
out of their hands. However, acceptance and use may
be enhanced as users gain a better understanding of
the characteristics, benefits and limitations of these
intelligence-based systems, and the role of the tech-
nology as a decision aid as opposed to a decision
maker.
Thus, a motivation for the development of the
prototype system is to enable future research de-
signed to explore practitioners' acceptance (or rejcc-
tion) of such systems. However, the objective of the
research presented in this paper is narrower in scope.
More specifically, this paper focuses on: (1)demon-
strating the appropriateness and applicability of an
expert network within audit practice; and (2) describ-
ing the methodology employed to design and de-
velop the prototype system.
351
This paper, therefore, contributes to the literature
on the practical application of AI-based systems to
audit judgment tasks in at least two ways: (1) consid-
eration of the application of hybrid intelligent com-
puting techniques to an area of auditing that is at
present making almost no use of them, but where
there is developing interest and a potential for bene-
fits to be accrued from their use; and (2) develop-
ment of an integrated ES and NN prototype system,
the CRA Expert Network, constructed for a PC-based
environment using tools that lead to a PC-based
product appropriate for field experimentation.
The remainder of the paper is organized as fol-
lows. Section 2 describes the nature of the audit task,
providing the background to the selection of an
expert network approach. Section 2 also describes
the principles behind the integration of ESs and NNs
and introduces the prototype CRA Expert Network.
Section 3 details the design and construction of the
prototype system, including a description of an ex-
periment conducted with practicing auditors used to
obtain judgment data for training the NN. Section 4
contains brief concluding comments.
2. O v e r v i e w of domain and system architecture
When a public accounting firm audits a business
entity, the potential exists that the auditors may not
discover material misstatements in the entity's finan-
cial statements. The likelihood of not discovering a
significant misstatement is called audit risk. During
the audit process, the business entity's internal con-
trol (IC) structure is evaluated to determine the
nature, timing and extent of audit tests to be per-
formed that will be most effective in reducing audit
risk. The purpose of the IC structure is to prevent
a n d / o r detect erroneous, fraudulent, or missing ac-
counting transactions. Consequently, a complete and
proper assessment of this structure is critical to a
successful audit.
The attempt to build intelligence-based systems to
support audit judgment tasks, such as control risk
assessment (CRA), is dependent on the acquisition
and representation of knowledge and heuristics
gained through audit experience (Deng, 1994). Gen-
eral audit theory and heuristics may be captured and
represented using logical constructs. But, obtaining
352 J.T. Davis et al./ European Journal of Operational Research 103 (1997) 350-372
knowledge from auditors for making small yet im-
portant distinctions between situations in the form of
specific rules may be nearly impossible. That is,
assessing the IC structure potentially requires evalu-
ating hundreds of variables with thousands of possi-
ble inter-relationships. It is not likely done by a
serial, step-by-step reasoning process, but rather by
recognizing patterns in a given situation and reacting
appropriately based on experience (Dreyfus and
Dreyfus, 1986; Anderson, 1983). Evaluating these
many complex relationships is a difficult task even
for the most experienced auditors - articulating them
is essentially impossible. However, neural network
systems can be used to automate judgment tasks that
require this pattern recognition.
Thus, the innate complexities of the audit judg-
ment task suggest that an appropriate approach to
system design is to integrate sub-systems of AI
techniques each of which address distinct aspects of
domain knowledge and reasoning (Lymer and
Richards, 1995). Given the characteristics of the
CRA task, the prototype system is constructed as an
expert network reflecting the integration of an expert
system (ES) and neural network (NN). Briefly, the
ES incorporates general audit theory and well-known
control relationships using a logical set of explicit
rules. The NN in the CRA Expert Network is used to
recognize and establish patterns among the large
number of inter-related variables inherent in the task.
When exercised, the ES provides the user interface
and evaluates the complexity and basic structure of
an entity's internal controls. If the ES determines
that the IC structure is sufficiently complex to war-
rant further analysis, the data collected during inter-
action with the ES is fed as input into the NN. The
NN is stimulated by the input data and provides a
preliminary CRA. The NN evaluates approximately
two hundred variables with thousands of potential
inter-relationships used to make a preliminary CRA.
The following section examines the principles be-
hind expert networks such as the CRA Expert Net-
work.
2.1. Expert networks: An integrated approach
An expert network is one form of integrated
systems designed to address limitations of using a
single representation and reasoning approach (Lymer
and Richards, 1995; Frisch and Cohn, 1991). ESs -
based on symbolic computation - are best at model-
ing structured problem domains (or aspects thereof)
that conform well to logical constructs. Conversely,
NNs - based on numerical computation - can suc-
cessfully model problems that do not conform well
to explicit logical constructs. It is when these situa-
tions cross, such as in the CRA application, that the
combined capabilities of a blended solution should
be considered (Caudill, 1991; Medsker and Bailey,
1992; Medsker, 1994). By combining the deductive
reasoning approach of an ES with the inductive
approach of a NN, difficult and somewhat unstruc-
tured tasks may be performed. Integration takes ad-
vantage of the strengths of each type of system,
while mitigating the inherent weaknesses of each
when used alone. Let us briefly examine the subsys-
tems of an expert network.
2.1.1. Components of expert networks
,sognitive models, focused on representations of
knowledge, suggest that human problem solvers or-
ganize knowledge through the use of propositions. A
proposition, in the form of an if-then rule, is the
atomic building block of rule-based expert systems.
Domain knowledge - generally elicited from a sin-
gle or multiple subject matter experts - is codified
into a series of linearly executed discrete rules (Solso,
1988; Greeno, 1973; Johnson-Laird, 1983; Minsky,
1986). However, significant development barriers
arise because experts often cannot articulate knowl-
edge and complex relationships as discrete rules.
Only when asked do they produce a justification for
the judgments made. Even then, the justification is a
generalization that more than likely has a fair num-
ber of exceptions (Deng, ! 994).
Propositional logic is rarely a sufficient means to
represent complex reasoning (Kunz et al., 1987;
Jackson, 1990). Rather, ESs are a particularly good
approach for closed-system applications that have
literal and precise inputs that lead to logical outputs
(MacLennan, 1993). However, they are relatively
inflexible since performance degrades sharply when
they are applied to problems outside their original
scope (Jackson, 1990; Kunz et al., 1987; Edwards
and Connell, 1989, p. 25). Moreover, changes in
domain knowledge structure and content often re-
 J.T. Davis et al./ European Journal of Operational Research 103 (1997) 350-372 353

quire substantial system modifications to continue or A N N is a statistical modeling technique. How-

enhance system viability.
Conversely, NNs can analyze large numbers of
inter-related variables to establish patterns and char-
acteristics in situations where precise, discrete rules
are not known or discernible (MacLennan, 1993).
Rather than depending on explicit knowledge as
expressed by a domain expert, NNs model the im-
plicit relationships in exemplar domain data. The
continuous nature of the stimulus/response approach
allows for efficient modeling of complex tasks. Sim-
ply put, a neural network discovers its own numeric,
continuous rules as it learns through the examples it
is provided. NN systems may be able to perform
certain types or parts of audit judgment tasks that are
difficult and perhaps inappropriate for the capabili-
ties of other types of intelligent systems.
An artificial NN consists of processing elements
linked via weighted uni-directional signal channels
called connections to form a distributed, parallel
processing architecture (Rumelhart and McClelland,
1986; Hecht-Nielsen, 1990). Each processing ele-
ment can possess local memory and carry out local-
ized information processing operations. The process-
ing element or neuron is the atomic building block of
the NN (Fig. 1). NN paradigms differ in how the
processing elements are connected and the manner in
which the weights are updated (Markham and Rags-
dale, 1995; Rumelhart and McClelland, 1986).
ever, NNs can make less stringent assumptions con-
ceming independence of variables and the shapes of
underlying data distributions than other statistical
techniques, e.g., regression or multiple discriminant
analysis (Rumelhart and McClelland, 1986; Lipp-
mann, 1987; Lacher et al., 1995). Rich discussions of
the statistical aspects of NNs and their relation to
more traditional statistical models may be found in
Ripley (1993, 1994), Cheng and Titterington (1994)
and Sarle (1994).
While NN systems are flexible in terms of fault
tolerance to missing or noisy data, they do not have
some basic characteristics for flexible precise com-
monsense reasoning, e.g., symbolic processing or
interpretation of internally stored knowledge (Sun,
1994, p. 247). Unlike ESs, NNs have no inherent
explanatory function 'module'. This has hindered
acceptance in practice as it is not clear to a non-tech-
nical user how the network derived a given conclu-
sion. However, research is being conducted to ex-
tract comprehensible symbolic representations from
trained NNs (e.g., Craven and Shavlik, 1996).
2.1.2. Relationship o f the CRA expert network to
existing systems
A number of hybrid ES and NN systems have
been described in the literature. Kandel and Langholz
(1992), Gallant (1993), Medsker (1995), and Sun

Xo= 1
(Bias)

Neuron or
XI
Processing
Element Neuron or
Inputs XL 1 Processing
Element
(Outputs from Output
other neurons
or processing
nodes)

Summation ] Transfer
Function ! Function

e.g. Sigmoid
Fig. I. Neural network processingelement structure.
354 J.T. Davis et al./ European Journal of Operational Research 103 (1997) 350-372
and Bookman (1995) have compiled research con-
cerning the integration of symbolic and numeric
systems. These compilations include descriptions of
expert networks applied to application domains such
as natural language, signal processing, biology and
medicine, management, and engineering (see particu-
larly Medsker, 1995, pp. 3 9 - 5 6 ) .
The relationship between the ES and the NN in
the CRA Expert Network is structured similarly to
systems reported by Lin and Hendler (1995), Macln-
tyre et al. (1995), and Bataineh et al. (1996). Lin and
Hendler (1995) use a NN to classify ballistic signals
and the output is passed on to the ES for further
processing and interpretation. Using the same neural
network software employed by the CRA Expert Net-
work, Maclntyre et al. (1995) and Bataineh et al.
(1996) present expert networks for application within
the electric utilities industry. The ESs in these appli-
cations also provide processing inputs to the NNs.
Thus, while expert networks (and other forms o f
integrated systems) are not an explicitly new tech-
nique, we have found no evidence o f the technique
being applied to audit judgment tasks such as CRA.
However, given the characteristics of the task, there
is significant scope for consideration of their applica-
bility to this domain. Furthermore, the ability to
construct such systems using PC-based tools should
facilitate experimentation with them in the field. The
following sections detail the design and construction
of the CRA Expert Network.
3. C o n t r o l r i s k a s s e s s m e n t (CRA) e x p e r t n e t w o r k
As introduced earlier, the prototype CRA Expert
Network was constructed by integrating an ES and
NN. The ES and NN components respectively ad-
dress two types of knowledge and logic relevant to
auditors: (1) well-known audit relationships - ap-
proximately 20% of the distinct internal control
structure variables - encoded as logical rules in the
knowledge-base; and (2) more complex control t;ari-
able relationships determined by the 'trained' NN.
The CRA Expert Network uses a loosely coupled
model of an integrated system in which the CRA
process is decomposed into separate ES and NN
components that communicate via data files (Medsker
and Bailey, 1992; Medsker, 1994). The ES provides
the user interface and conducts preprocessing of data
that is fed into the NN. After the NN performs its
pattern-matching activities, output is passed back to
the ES for display. Advantages of loosely coupling
the components of the prototype system are twofold:
(1) system development is amenable to commercially
available software; t and (2) maintenance is reduced
due to the simplicity o f the data file interface ap-
proach. The disadvantage of a great deal of redun-
dancy that usually accompanies loosely coupled sys-
tem has been largely avoided in the C R A Expert
Network, perhaps due the nature of the problem
more than any other reason. Before proceeding to an
in-depth discussion of the structure of the C R A
Expert Network, the following two sections describe
the knowledge - and sources of that knowledge -
reflected in the ES and NN components, respec-
tively.
3.1. Expert system development
The rules for the ES were primarily derived from
structured logic and questions found in Grant Thom-
ton's internal control documentation and evaluation
software - Information and Control Understanding
System (lnfocus) (Grant Thornton, 1992), which is
used in audit and consulting practice. Infocus does
not make any risk assessment itself and is not an AI
decision tool. However, it provides general relation-
ships among IC variables to assist the auditor in
structuring the preliminary C R A process. The knowl-
edge and logic of Infocus were formalized in a
logical rule structure. The interface provided by the
ES allows the auditor to document the IC data of a
given client. It should be noted, however, that the
user interface screens that are provided to an auditor
The ES component of the CRA Expert Network was devel-
oped using Microsoft Visual Basic 3.0. This tool provided the
means to: (a) develop an object-oriented GUI user interface; (b)
encode the well-known audit relationships in a rule structure that
logically drives data collection; and (c) directly link to a Microsoft
Access 2.0 database. The data collected as an auditor interfaces
with the ES component is stored in an Access 2.0 database file. If
warranted, an ASCII text file of this collected data is fed to the
NN as input. The NN, stimulated by these input variables, derives
a CRA. The NN was created in NeuralWorks Professional I1/Plus
(NeuralWorks, 1992).
 J.T. Daois et aL/ EuropeanJournal of OperationalResearch 103 (1997)350-372
during a given session are controlled by the logic
structure encoded in the ES - that is, data is neither
requested of a user or screens provided that are not
relevant to a given client. Use of Infocus' logic and
question was necessary because lnfocus was an inte-
gral part of the knowledge acquisition process con-
ducted with audit seniors - whose knowledge was
ultimately used to train the NN. In addition, lnfocus
also provides a real world, accepted and tested basis
for the majority of the ES logic.
Added to the knowledge-base - that is not from
Infocus' logic and questions - are CRA threshold
rules and rules translating the preliminary CRA out-
put of the NN to a CRA category. The CRA thresh-
old rules are used to control the current assessment
of control risk while a session is in progress. For
example, depending on the earliest data collected by
the ES component and the current threshold level
setting, the ES may determine that further processing
is not warranted, i.e., the data collection by the ES is
halted and the NN is bypassed. This process is
detailed in a following section.
3.2. Neural network development
3.2.1. Data source for training and testing
The data used to train the NN was from an
experiment conducted with 64 senior auditors from
Grant Thornton (see Davis, 1996). These subjects
averaged 4.5 years of audit experience and bad
completed a CRA an average of 37 times.
Each auditor was given one of three client cases.
The first represented an actual small sized client of
the firm. The other two were based on example cases
found in the SAS No. 55 Audit Guide (AICPA,
1989). All three cases involved merchandising enti-
ties. The preliminary CRA was restricted to the sales
stream within the revenue cycle of each entity. The
financial statement assertion group consisted of com-
pleteness, existence/occurrence, and valuation -
core assertions related to the IC system goal of
preventing and detecting errors.
The three cases were chosen because the entities
had computerized accounting systems and reflected
situations in which the auditor could potentially rely
on all three types of internal controls - manual
controls, programmed controls, and segregation of
355
duties. 2 However, the cases represented different
situations in terms of firm size, complexity of control
system, and strength of controls. The first case repre-
sented an actual small-sized client, with low control
system complexity, yet a fairly strong IC structure.
This case was also compatible with a purely substan-
tive audit approach. 3 The two cases from the audit
guide represented larger entities than the first case.
The first, a closely held company, did not have as
strong of an IC structure as the second, a larger
public corporation that had a very strong IC struc-
ture. While both had computerized accounting sys-
tems, the public corporation had a more complex and
well-controlled computer system.
Very broadly, while conducting a preliminary
CRA, an auditor raises and reviews questions regard-
ing the ICs (e.g., manual, computerized) of an entity
that are designed to ensure that significant misstate-
ments on financial statements will be prevented or
detected. However, the questions that an auditor
chooses to examine are left primarily to the auditor's
judgment. Thus, the purpose of working with the 64
auditors was to determine not only what their CRA
was for a specific case, but also what set of questions
each auditor was using during the CRA process.
3.2.1.1. Collection of CRA process data. Using lnfo-
cus, each auditor began their CRA by selecting
variables a n d / o r addressing specific questions from
a set of potential judgment variables and questions
related to the revenue cycle IC environment, EDP
environment, and accounting controls. Values as-
signed to selected variables and the responses to
chosen questions constituted each auditor's 'cue set'
2 Segregation of duties refers to the principle that an individual
responsible for the conduct of a process cannot also be responsible
for the controls associated with that process. For example, if an
individual is responsible for receiving checks from customers,
they should not be also responsiblefor recording those payments
in the accountingsystem.
3 Essentiallythere are two audit approaches. In the substantive
approach, the auditor elects to ignore the controls in place and
focuses their analysis directly on the numbers represented in the
accounting statements. Conversely, in the control approach the
auditor focuses first on an analysis of the IC structure of the client
in an effort to reduce the amount of substantive testing that will
follow.
356 J.T. Davis et al./European Journal of Operational Research 103 (1997) 350-372

- issues deemed as relevant to the specific case and, based on the numeric scale. The point estimate corre-
ultimately, assessment of control risk. The IC poten- sponds to the categorical judgment responses as fol-
tial variables/questions, and the possible values that lows:
each may be assigned, are presented in Appendix A.
The appendix reflects the complexities and relation-
Risk Category Point Estimate Interval
ships of the questions and variables that are poten-
tially considered by an auditor (including the 64
Limited (LTD) 0-25
auditors that participated in the experiment) during
Moderate (MOD) 2 6 - 50
the CRA task. Appendix A indicates the percentage
Slightly below 51-75
of the auditors, by case, that deemed each
maximum (SBM)
variable/question as relevant.
IC environmental features, EDP environmental Maximum (MAX) 76-100
features, accounting controls, and segregation of du-
ties operate as inter-dependent parts of the IC struc-
Each auditor's selected cue set and corresponding
ture. While these variables were included in this
derived CRA point estimate 4 make up one observa-
experiment and addressed by the auditors in various
tion. Although there were only three cases, the ob-
fashions, the complex inter-dependencies between
servations were distinct as each auditor indicated a
the variables makes it virtually impossible for an
different cue set for making their preliminary CRA.
auditor to explain the structure of their cue set, i.e.
For example, it is possible for an auditor to rely on
the relationships and strengths of relationships be-
programmed controls and choose a CRA of MOD.
tween items in the cue set, and the impact of this
Conversely, another auditor (examining the same
structure on the CRA. These inter-dependencies serve
case) could choose to rely on manual controls and
to illustrate the complexity of the domain and the
make the same preliminary CRA.
task facing an auditor when conducting a CRA. This
The 64 observations were used to develop and test
level of complexity also highlights the role that the
the NN model. The network training (within sample)
NN will play in structuring these relationships.
and testing (out-of-sample or hold-out) data sets each
In addition to identifying their respective cue set,
contained 32 observations. Once trained, using the
each auditor recorded: (1) whether s / h e planned to
32 within sample observations, the NN model serves
rely on manual controls, programmed controls,
as a proxy for the typical knowledge structure used
a n d / o r segregation of duties - indicating whether
by the experienced senior auditors in the training set.
more tests of controls as opposed to more substan-
The 32 observations in the hold-out sample were
tive tests would be used to reach an acceptable level
used to test the resulting NN model.
of audit risk; (2) the existing controls selected as key
controls, i.e., controls the auditor intended to rely on
3.2.2. Training and validation of NN
and test to determine if the control is operating as it
The CRA Expert Network employs a feedforward
should; and (3) any controls that s / h e felt were
classification NN that was trained using the back-
missing.
propagation learning algorithm (Rumelhart and Mc-
3.2.1.2. Collection of CRA output data. Following
analysis of the case, each auditor recorded their
preliminary CRA in Infocus. lnfocus provides a
choice of four risk categories: maximum; slightly 4 While it m a y seem tvdd to use a point estimate here, the
below maximum; moderate; and, limited. These risk motivation is rather simple. It allows for the use of a t-test to
categories are consistent with SAS No. 55 Audit c o m p a r e the N N ' s C R A (a value between 0 and 1) to that of the
auditors, rather than relying solely on classification a c c u r a c y . A
Guide. In addition to the categorical response, each
classification NN does not yield a c c u r a c y in relation to a t-test -
auditor recorded their CRA using a point estimate (0 just classification a c c u r a c y . Furthermore, this a p p r o a c h allows for
to 100 scale) which provides an indication of how the NN response to be m a p p e d to the C R A risk categories that are
close to the category border their judgment would be inherent in thc experimental task.
 J.T. Davis et a l . / European Journal o f Operational Research 103 (1997) 350-372
Clelland, 1986; Tam and Kiang, 1992). A feedfor-
ward NN using sigmoidal activation functions is
mathematically capable of any continuous function
and thus is applicable to a large variety of knowl-
edge intensive tasks by distributing knowledge en-
coded into link weights that is learned from data
examples (Hornick et al., 1989). In the network the
processing element (refer to Fig. 1) takes inputs from
other neurons and sums these inputs, X 0 . . . X,, us-
ing a summation function, L i = EN_oWI,jXj. The
transfer function 'normalizes' the input summation
by vectoring it into a predetermined range. The
activation level of the processing element is deter-
mined by an activation function: X i = 1/(1 + e-~').
The backpropagation learning algorithm was used
to find the functional relationship between the inputs
(judgment cues/variables) and the target outputs
(the auditors preliminary CRAs). A variety of net-
work configurations were tested during the design
phase. Networks with a larger number of middle
nodes learned the training data sample quite well, but
performed poorly on the test data sample. Networks
Network Accuracy - Training Data
"E 0.9
0.7
iii,iiiiiil
0.5
'~ 0.3
0.1
o -0.1
O
o -0.3
Network Accuracy - Testing Data
1
0.8
,¢
0.2 ;~!~:i~i~!~:.~?~:~i~:N:~! ~: i~.:~4i~: %::~ ! ~ ~i:!-~i!~!i~::~~-:: ~ ~.........
0
O
o
357
with a smaller number of middle nodes were too
general and performed poorly overall. The final NN
architecture has 210 input nodes (control cues/varia-
bles), 30 hidden (middle) layer nodes, and one out-
put node for the preliminary CRA using the auditors
point estimates within their chosen CRA category.
Delta rule summations were employed with sigmoid
transfer functions on the middle and output nodes,
while the standard root mean square error (RMSE)
function was used for "all network layers. Training
stopped at 1600 iterations using the early stopping
technique to avoid overfitting of the training sample
- the RMSE of the sample began to increase instead
of decrease at that point. In addition, the NN was
analyzed after training at 4800 iterations. While ac-
curacy - both RMSE and category accuracy - on the
training sample was better than at 1600 iterations,
accuracy on the hold-out sample and overall accu-
racy on the total sample declined.
The final trained network (at 1600 iterations) had
a Pearson correlation coefficient of 0.869, and a
CRA category accuracy rate of 72% for the training
¢ Desired Output I
.._~.... Network Output I
! ,t Difference J
Observations
~ :~:~i~i~ii~
r_.4k_. Desired Output
[ ~ Netw ork Output
!~:~:;
,¢ Difference
Obs e rvations
Fig. 2. Network validation.
358 J.T. Davis et al./European Journal o f Operational Research 103 (1997) 350-372

General ~. [ Compu,er L I General Control

Processing ~ _1 Questions
Environment ~ .... 1 Overview ~ ]Computer Controls~ Phase 1
Questions (CPO) r////~, | (Gce)

. . . . t ounting
A Controls ~.
Phase 2

211
Vmmbles

1 2 3 .... 210 Input Layer

Neural i Phase 3
1 2 .... 30 Middle Layer
, -A .... LJ
I
1 Output

Fig. 3. Expert network: Knowledge base structure with embedded neural network.

data set. Testing using the hold-out sample resulted
in a Pearson correlation coefficient of 0.695, with a
category accuracy rate of 78%. 5 The risk category
prediction errors for the test sample included: four
observations that were one category higher than the
assessment by the auditor: one observation each that
were one category lower than the auditor, two cate-
gories higher than the auditor, and two categories
The correlation coefficient is a measure of relationship be-
tween paired observations in two data sets - here, the relationship
between the auditor's point estimate and the model's point esti-
mate for each observation. Category accuracy rates were derived
by comparing the NN point estimate output - based on an
auditor's input cue set - to the auditor's chosen risk category. For
comparison, error rates for NN models built for financial distress
applications (Bell et al., 1990; Tam and Kiang, 1992) were in the
range of 10-23%. However, it is important to note that the models
in those studies were much simpler than the NN developed in this
study. In those studies there were less than 10 input variables with
two response levels for output. Predicting a four category response
with 210 input nodes is a significantly more difficult problem.
lower than the auditor. Fig. 2 presents the point
estimate network accuracy in relation to both the
training and testing samples.
A paired t-test (see Koopmans, 1987, p. 325) was
also
• run on the predicted and actual network output.
No statistical difference was found for the training
sample (mean difference = - 0 . 0 0 0 3 , one-tail p -
value = 0.48) or the testing sample (mean difference
= 0.0239, one-tail p-value = 0.24).
In addition, two classification networks were de-
veloped - the first employed backpropagation and
the second a radial basis function - using only the
C R A classification categories and not the auditors'
point estimates. The hold-out sample classification
accuracy for both these models was approximately
49%. Conversely, as described above, the point esti-
mates for the trained network in the CRA Expert
Network were within the classification ranges chosen
by the auditors 78% of the time. Clearly, for this
data, using a point estimate within C R A category
provided superior model precision.
 J.T. Davis et al./ European Journal of Operational Research 103 (1997) 350-372 359

(a)
!
! Ask Control
Environment Questions

Yes PHASE1

No

Yes Set CRT = MAX

1
Set CRT >= Slightly
Below Maximum

--No--
I SetManu,,Cootro,---)
Neural Network
Inputs = 0 I
I

yes
I

Set CRT >= Limited CRA = Current

Threshold

I Yes

Computer Processing
Overview (CPO)

Set CPO & GCQ---~

Neural Network
Inputs = 0
(
)

Fig. 4. (a) Expert network processes flowchart; (b) Expert network flowchart.
360 J.T. Davis et al. / European Journal of Operational Research 103 (1997) 350-372

(b)

General Controls
Questionnaire
(GCQ)
PHASE1
No

Set GCQ Neural

Network No
Inputs = 0

l
Rely on
-'--!
Rely on
Manual
Controls
(MC)
t
Y~--

Manual . S -- Cu ent i l
Controls
(MC)

I
Yes
I

Accounting
. ~ ~Accounting Controls Accounting Controls
Controls
_bJ Questionnaire: Questionnaire:
-I Set PC Questions = 0 PHASE2 Set MC Questions = 0 Questionnaire:
Ask MC Questions Ask PC Questions Ask MC & PC
Questions

_ _ _ T . . _ _ _ _

General Environment PHASE3

Variable Values

Fig. 4 (continued).

3.3. CRA Expert Network Design
The ES and trained NN constitute the CRA Expert
Network. The system employs a three phased ap-
proach to determine a preliminary CRA as follows:
cessing overview, and general control questions
(computer-related);
• Phase 2. Accounting controls phase; and,
• Phase 3. The Neural Network phase.

• Phase I. Environment phase that encompasses Fig. 3 illustrates the relationships among these
the general environment questions, computer pro- three system phases. Fig. 4 provides a flowchart
 J.T. Davis et al. / European Journal of Operational Research 103 (1997) 350-372
overview of the main logic contained within the
CRA Expert Network and its three phases.
Phases 1 and 2 address the logical data collection
aspect of the CRA Expert Network by incorporating
and presenting to the user the judgment variables/
questions found in Appendix A. As noted previously,
the data collected during these phases is stored in a
database file, which is fed to the NN. The NN
(during Phase 3) accepts and utilizes values for the
following (as detailed in Appendix A): (1) the Rev-
enue Cycle Environment variables; (2) the Planned
Reliance variables; (3) responses to the Computer
Processing Overview (CPO) questions; (4) responses
to the General Control Questions (GCO, and; (5) all
the Accounting Controls variables. Values accepted
by the NN are: a ' 1 ' for a Yes response to a question
or the indication that an Accounting Control variable
is a key control; a ' - 1 ' for a No response to a
question or that a specific Accounting Control vari-
able is missing, and; a ' 0 ' indicating that the ques-
tion/variable is not applicable to the CRA. As Fig. 3
indicates, there exist 210 values passed to the NN -
each constituting a separate node in the input layer
of the NN.
However, it should be emphasized here that not
all questions/variables are applicable to all cases.
Data is collected in Phases 1 and 2 via user interface
screens - the presentation of which is controlled by
the logic structure encoded in the ES. This logic
structure is illustrated in Fig. 4. The ES does not
continue along a data collection path that is not
relevant. For example, if the answer by the auditor to
a particular CPO question is ' N o ' , then the related
GCQ questions are not applicable to that case. The
ES then bypasses the related GCQ questions - the
user never sees those screens - and assigns '0s'
(which are passed to the NN), representing not appli-
cable, to those GCQ questions (see Fig. 4(b)). The
following sections present specific details concerning
each phase.
3.3.1. Phase 1: Environment phase
During the environment phase, information neces-
sary for the ES to apply broad internal control audit
heuristics is collected. These heuristics should be
well known to most auditors. However, the possibil-
ity exists that one or more of these heuristics could
be omitted or misapplied. Thus, including these
361
heuristics as rules in the ES should improve an
auditor's control variable selection consistency as
well as inter-auditor consistency.
The questions are designed to collect information
concerning the control environment and computer
environment (if any) in which the accounting con-
trols, addressed in Phase 2, must operate. This infor-
mation is used by the ES, to make (if possible) a
preliminary CRA without proceeding to Phases 2
and 3. In addition, the information is used to set a
control risk threshold (CRT). The purpose of the
CRT is to prevent a final CRA that is below a
predetermined level, given that certain conditions
exist in the client environment.
Broadly, during Phase 1 (see the flowchart in Fig.
4 and the specific questions found in Appendix A),
the auditor is asked if s / h e plans to rely on the
control environment. If not, i.e., the auditor believes
that the control environment is not reliable or testing
the controls is inefficient, the ES sets the CRA to
maximum (MAX) and the task is completed. This
indicates that the auditor will conduct a purely sub-
stantive audit approach. Conversely, a yes response
indicates that a control audit will be conducted and
the auditor is then asked if s / h e plans to rely on
segregation of duties. If so, reducing the control risk
threshold (CRT) to slightly below maximum (SBM)
is justified; if not, the CRT is set to maximum
(MAX). In either case, the auditor is asked if they
plan to rely on manual controls (MC). If s / h e does,
the CRT is reduced to limited (LTD), and the system
is triggered to obtain information concerning MC
during Phase 2. If no reliance is planned, the CRT
remains unchanged and the MC questions are omit-
ted from Phase 2. The control variables related to the
omitted questions will be input to the NN as '0s'
(i.e., not applicable).
The next set of questions concerns whether any
part of the accounting system is automated. If there
is no planned reliance on MC and none of the
accounting system is automated, the CRA is set to
the current CRT and the task is complete. If there is
automation, the Computer Processing Overview
(CPO) will be completed, thus gathering information
about the general computer environment and applica-
tion complexity. The responses to these questions
determine thrce things: (1) the level of computer
environment risk; (2) the level of computer applica-
362 J.T. Davis et aL / European Journal of Operational Research 103 (1997) 350-372
tion complexity, and; (3) the relevant questions to be
included in the General Controls Questionnaire
(GCQ). At this point the auditor may decide not to
rely on p r o g r a m m e d controls. If this is the case, no
GCQ questions are asked and all the GCQ inputs to
the NN will be set to 0. In the event that manual and
programmed controls are not relied on, the system
sets the C R A to the current value of the CRT and
exits. Conversely, if the auditor is relying solely on
manual controls, the system sets the program control
NN input variables to 0, and proceeds to Phase 2
where it asks the manual controls questions. If the
auditor elects to rely on program controls, the GCQ
questions are presented for completion.
Finally, after completing the General Controls
Questionnaire (GCQ), the auditor (usually consulting
with the computer audit specialist) will be given
another chance to reject reliance on programmed
controls. If the auditor decides that computer con-
trois are too weak to support reliance on pro-
grammed accounting controls, all the GCQ inputs to
the NN will be set to 0. The system then moves on to
Phase 2 for the manual accounting controls informa-
tion, if the auditor has previously indicated reliance
on manual accounting controls. Once again, in the
event that manual and programmed controls are not
relied on, the system uses the current CRT to com-
plete the CRA task.
3.3.2. Phase 2: Accounting controls" phase
The accounting controls phase includes the com-
pletion of the manual accounting control questions,
the programmed control questions, or both depend-
ing on the decision path taken during Phase 1. If
either the manual controls or the programmed were
considered not applicable in Phase 1, the ES sets the
corresponding NN input values to 0 and does not
present the user with any questions related to these
controls. If either or both are considered relevant, the
appropriate Accounting Controls Questionnaire is
presented to the user.
The questions for Phase 2 relate to potential
accounting controls (see Appendix A). The auditor
responds to each question by labeling each individ-
ual potential control as a key control (coded as a ' 1 '
for the neural network); missing control ( - l); or not
applicable (0). Upon completion, the CRA Expert
Network proceeds to Phase 3.
3.3.3. Phase 3: Neural network
Assuming the auditor chooses to rely on manual
accounting controls, programmed accounting con-
trois, or both, the responses collected from Phases 1
and 2 are fed into the NN (Fig. 4). Each question
from the ES is mapped to one of the 210 NN input
nodes. Given this input, the NN derives a prelimi-
nary CRA - an evaluation among 0 and 1 - which is
presented to the user by the CRA Expert Network. In
addition to presenting the numerical evaluation, the
ES transforms the response to the CRA risk cate-
gory. For example, if the response was 0.52, the risk
category would be slightly below maximum (SBM),
although it is close to the S B M / M O D category
border. The point estimate from the NN does provide
information to the auditor as to how close the NN
response is to a risk category border. Whether this
information ultimately contributes to an auditor's
Judgment a n d / o r leads to better system acceptance
are empirical questions.
3.4. Directions f o r future development and research
The prototype system offers the opportunity for
experimentation with practicing auditors, and pro-
vides a basis from which standards may be estab-
lished for a task in which no specifically set proce-
dures and rules are currently delineated. Future de-
velopments will center on addressing current limita-
tions in the prototype CRA Expert Network.
The first limitation relates the knowledge acquired
and represented in the system. More specifically, the
system is based on a limited number of cases and the
fact that the auditor subjects were from the same
firm. However, a sizable set of cases and additional
auditors could be obtained from audits already com-
pleted a n d / o r by providing auditors with additional
case scenarios.
A related issue is whether the NN model using
point estimates within categories provides a more
precise decision-aid than a pure classification model.
For this data, the extra information as to how close
to a category border the auditors' judgments were,
seemed to give the point estimate model an advan-
tage over the pure classification models. Of course,
using a point estimate within category assumes more
judgment precision on the part of the auditors. There
may be a trade-off between judgment precision and
 J.T. Davis et al. / European Journal of Operational Research 103 (1997) 350-372
generalizability between the point estimate within
the categories approach and the purely classification
approach. This empirical question should be tested in
future research particularly with regard to acceptabil-
ity by auditors.
Second, most auditors are not computer audit
specialists. When the computer processing overview
(CPO) indicates a complex system, a computer audit
specialist should be consulted. The computer audit
specialist assists the auditor in deciding whether the
computer controls are adequate to support reliance
on accounting programmed controls. Thus, efforts
are underway to capture and represent in the system
this specialized knowledge.
Third, the system is currently narrow in scope. In
addressing the risk assessment task, only the sales
stream within the revenue cycle is included. How-
ever, the sales stream is general to nearly all busi-
nesses and important in most all audit situations. In
order to increase the usefulness of the system, a set
of integrated systems each designed to deal with
particular parts of all the accounting cycles should be
developed.
Additionally, the system does not include assign-
ment of accounting control variables to particular
accounting processes or control objectives. The NN
only considers whether a particular control is a key
control and should be relied on, is a missing control,
or is not applicable. However, the same control may
apply to more than one accounting process. The
system does not take into consideration these com-
pensating controls - controls that are in operation
later in the accounting process stream that may
strengthen the overall effectiveness of the IC system.
Further design efforts will involve including this
assignment of controls to processes and accounting
objectives. Efforts in this vein may take the form of
additional rules in the knowledge-base, more neural
network variables (and, perhaps even a different NN
architecture), and more than three levels or values
that a particular control variable could be given.
Currently, research efforts with regard to the CRA
Expert Network involve two main areas: (1) estab-
lishment of experiments with auditors in the field,
and (2) design and development efforts focused on
enhancements to the user interface and the construc-
tion of an explanation module. While the NN pro-
vides the capability to model complex relationships
363
and derive a conclusion, it does not provide any
justification for the conclusion. A major concem for
the auditor is being able to document audit judg-
ments in order to defend his/her judgment, if neces-
sary. For example, if the auditor agrees or disagrees
with the suggested CRA, they must be able to docu-
ment their reasoning. This dilemma is similar to the
dilemma faced by the medical profession for using
intelligent systems as decision aids. However, inter-
pretation of how a NN produced a particular conclu-
sion is not a trivial task, given that the very purpose
of using a NN is to model complex relationships that
are not well understood. Much research in this area
is left to be done.
4. Conclusion
The creation of the CRA Expert Network provides
a good example of the potential applicability of
integrate d AI technologies to audit practice and
demonstrates the appropriateness of the integrated
technique to the specific audit judgment task of
assessing control risk. Potentially, intelligent systems
such as this can provide several benefits to audit
firms, including: preservation, replication, and distri-
bution of expertise; new insights into the decision
process; decision-aiding support: consistency of de-
cisions; and, increased productivity (Borthick and
West, 1987).
In conclusion, the CRA Expert Network takes
advantage of both the deductive approach of an ES
and the inductive approach of a NN to provide a
unique decision aid designed to support and facilitate
the process of conducting a CRA. The system is
tailored to auditors requirements in terms of data
collection and analysis and allows an auditor to take
a more considered and structured approach. The
system offers audit firms the opportunity to improve
upon the accuracy of the CRA by capturing the
expertise of multiple auditors and multiple client
experiences. Finally, the nature of the system en-
courages efficiency in the analysis process as it
follows a logic-driven data collection path, request-
ing the user to only address questions that are logi-
cally required to make a CRA. The validation results
to date indicate that an expert network shows promise
364 J.T. Davis et al. / European Journal o f Operational Research 103 (1997) 350-372
for addressing the large complex judgment processes
inherent in control risk assessments. While intelli-
gent systems are domain and task specific, this ap-
proach is conceivably transferable to other large
complex judgment tasks.
Acknowledgements
The authors wish to thank Dr. George W. Krull,
Jr., National Director of Professional Development at
Grant Thornton, whom was the driving force behind
the experiment presented as part of this research. The
experiment took place in conjunction with the firm's
national professional education program. Dr. Kmll
also provided valuable comments with regard to the
experiment. In addition, we appreciate the direction
provided by Stephen Yates, Partner and National
Director of Advanced Audit Techniques, who pro-
vided the cases, instruction and opportunity to utilize
Infocus for the study.
Appendix A. List of judgment variables/
questions 6
A.1. General environment questions (Phase 1)
Revenue cycle environment variables.
Volume of transactions: High, Medium, Low
(59%, 100%, 100%);
6 Each of the following represent an input node to the Neural
Network. There are 210 input nodes, including three each for
volume and dollar value (high, medium, low). These inputs can
take on values of 1 (Yes or Key Accounting Control), " - I' (No
or Missing Accounting Control), and "0' (Not Applicable to the
CRA).
A 'Yes' response to a particular CPO question will trigger a
corresponding GCQ analysis to determine the status of a GCQ
objective.
The percentages in the parentheses, following each variable/ques-
tion, represent the percentage of the auditors in the experiment
that had included and addressed this item in their "cue set'. The
order respectively reflects values for the three experimental cases
- small client, nonpublic client, and public client. An n / a indi-
cates that the item was not applicable to the case.
Dollar value of transactions: High, Medium, Low
(63%, 93%, 100%);
Whether part of the cycle is automated (100%,
93%, 91%);
Whether management override is a concern (81%,
60%, 61%);
Whether client has an internal audit staff that
works to prevent or detect material misstatements
in the financial statements (89%, 33%, 87%).
Planned reliance on type of controls.
Whether reliance is planned on manual controls
(required)
Whether reliance is planned on programmed con-
trois (required)
Whether reliance is planned on segregation of
duties (required)
A.2. Computer processing overview (CPO) and gen-
eral controls questionnaire (GCQ) company wide
(Phase I)
CPO question: Is a third party service organi-
zation used to process all transactions involving
electronic data processing? (n/a, n/a, n/a)
Has the service auditor reported on the processing
of transactions? (n/a, n / a , n / a )
Does the report only cover policies and procedures
in place? (n/a, n / a , n / a )
Does the report test the policies and procedures?
(n/a, n / a , n / a )
Do we have a copy of the report? (n/a, n / a , n / a )
CPO question: Are the client's computers in a
dedicated physical area or facility? (n/a, 80%,
87%)
GCQ objective: Computer system physical secu-
rity is adequate.
Is entry to the computer and supervisor terminal
areas restricted to those staff required for com-
puter operations? (n/a, 80% 87%)
Are lists of authorized personnel maintained and
kept up-to-data? (n/a, 80%, 78%)
Is a log of all visitors maintained and reviewed?
(n/a, 80%, 78%)
 J.T. Davis et al. / European Journal of Operational Research 103 (1997) 350-372
Do procedures exist to control access by non-DP
personnel (e.g. engineers, janitors)? (n/a, 80%,
87%)
Are DP management and security personnel noti-
fied when DP staff /eave the client's employ-
ment? (n/a, 80%, 87%)
Are employees who work outside normal opera-
tion hours properly authorized and adequately su-
pervised? (n/a, 80%, 87%)
Is the computer room physical security adequate
to restrict unauthorized access to program and data
files? (n/a, 80% 87%)
CPO question: Is there a separate EDP depart
ment? (n/a, 93%, 96%)
GCQ objective: Separation of EDP duties is ade-
quate.
Systems Programming?
Are systems programmers prohibited from operat-
ing the computer system when production files or
application programs are resident? (n/a, n/a,
91%)
Are systems programmers prohibited from chang-
ing production files and programs? (n/a, n/a,
96%)
Are responsibilities for various processors or soft-
ware products periodically rotated among mem-
bers of the systems programming staff?. (n/a,
n/a, 70%)
Are systems programmers' activities logged? (n/a,
n/a, 87%)
Are systems programmers' activity logs reviewed
by management? (n/a, n/a, 96%)
Are system utilities (ZAP, Super-ZAP) in use and
appropriately controlled? (n/a, n/a, 87%)
Is there a written computer development strategy?
(n/a, n/a, 52%)
Are there adequate reporting procedures to moni-
tor the progress of computer development? (n/a,
n/a, 39%)
Application Programming?
Is development work carried out by using separate
source libraries, separate object libraries, and sep-
arate development machines? (n/a, n/a, 74%)
Are application programmers restricted from ac-
cessing program libraries or data files which are
365
used for production runs? (n/a, n/a, 96%)
Are application programmers prohibited from set-
ting up and operating the computer, even during
program testing? (n/a, n / a 96%),
Does management approve development at key
stages: feasibility systems proposals, outline sys-
tems proposals, detailed system design including
systems specifications, parallel running or system
acceptance testing? (n/a, n/a, 65%)
Computer Operations?
Is access to operators' consoles restricted to com-
puter operators and other authorized staff?. (n/a,
93%, 96%)
Are computer operators restricted from performing
programming functions or running unauthorized
jobs? (n/a, 93%, 96%)
Are computer operators excluded from access to
cash and accounting source documents? (n/a,
93%, 96%)
Is access to production source libraries by com-
puter operations' personnel restricted? (n/a, n/a,
96%)
Are logs or records of computer system activity
(jobs and program runs, reruns, abnormal termina-
tions of jobs and programs, system console opera-
tor commands) maintained and reviewed? (n/a,
93%, 96%)
Are system log exceptions investigated and results
of the investigation documented? (n/a, 93%, 96%)
Are system utilities (ZAP, Super-ZAP, etc.) in use
and appropriately controlled? (n/a, 93%, 83%)
Input/Output Scheduling?
Is the preissuance of files for night and weekend
shifts in accordance with approved job schedules?
(n/a, 93%, 87%)
Is authorization required for unscheduled job re-
quests? (n/a, 93%, 87%)
Are any scheduling overrides automatically logged
and subject to review? (n/a, 93%, 91%)
Are all terminals closed down at the end of opera-
tions? (n/a, 93%, 96%)
Is there a secure location for sensitive output (e.g.
check printing)'? (n/a, 93%, 91%)
Library maintenance?
Is formal authorization required to transfer pro-
366 J.T. Davis et aL / European Journal of Operational Research 103 (1997) 350-372
grams to the production library? (n/a, n / a , 78%)
Is the transfer of files to the production library
carried out by computer operations staff?. (n/a,
n / a , 78%)
Are all additions to the utility library authorized?
(n/a, n / a , 83%)
Do header labels hold reference and control data
(i.e. file name, generation numbers, volume ID,
time/date stamps on creation, expiration dates,
control totals)? (n/a, n / a , 78%)
Does program library software, if any, monitor
and record program changes? (n/a, n / a , 91%)
Are the activities of the EDP department (includ-
ing management) appropriately segregated?
Is there an organizational plan that defines and
allocates responsibilities and identifies lines of
communication? (n/a, 93%, 70%)
Are the responsibilities of the EDP department,
accounting department and other DP users clearly
defined? (n/a, 93%,96%)
Are there appropriate procedures for: rotation of
duties/shifts, holiday arrangements, and termina-
tion of employment? (n/a, 93%, 91%)
Are source documents handled within data pro-
cessing only by data preparation and data control
staff?. (n/a, 93%, 87%)
Is a lack of segregation compensated for by in-
creased management supervision?
Is there independent management review and ap-
proval of various data processing functions? (n/a,
n/a, n/a)
Are there controls over certain privileged func-
tions, such as requiring that all program library
updates are assigned to designated personnel?
(n/a, n / a , n / a )
CPO question: Can separate u ~ r s access the sys-
tem concurrently? (19%, 93%, 91%)
GCQ objective: Access control over programs
and data files is adequate.
Is access control software used to restrict access to
the production programs?
Are master lists of authorized personnel main-
tained, indicating restrictions on access? (n/a,
n / a , 91%)
Are access and security features within the operat-
ing system utilized to restrict access to and
amendment of program files? (n/a, n / a , 91%)
Is terminal activity automatically monitored and
rejected access attempts reported? (n/a, n / a , 91%)
Does the access control software identify each
terminal via the logical address? (n/a, n / a , 91%)
Does the access control software identify each
terminal via the physical address? (n/a, n / a , 91%)
Is access control software used to restrict access to
the data files?
Are master lists of authorized personnel main-
tained, indicating restrictions on access? (19%,
93%, 91%)
Are access and security features within the operat-
ing system utilized to restrict access to and
amendment of program files? (19%, 93%, 91%)
Is terminal activity automatically monitored and
rejected access attempts reported? (19%, 93%,
91%)
Is there encryption of all sensitive data files?
(15%, 93%, 91%)
Does the access control software identify each
terminal via the logical address? (11%, 93%, 87%)
Does the access control software identify each
terminal via the physical address? (11%, 93%,
87%)
For data base applications, is access to the data
possible only through the DBMS? (15%, 93%,
91%)
Are reports generated by such software reviewed
by management?
Is the record of jobs actual/y run, and console
logs, reviewed by supervisory personnel? (15%,
93%, 87%)
Is the computer usage summary produced and
reviewed? (15%, 93%. 87%)
A.3. Computer processing overview (CPO) and gen-
eral controls questionnaire (GCQ) accounting cycle
specific (Phase 1)
Note: The next two questions determine whether
the environment for the particular accounting cycle
is/is not low risk. The next ten questions determine
 J.T. Davis et al. / European Journal of Operational Research 103 (1997) 350-372
whether the application complexity for this account-
ing cycle is simple or advanced.
CPO question: Are transactions for this cycle
processed by a third party service organization?
(n/a, n/a, n/a)
Has the service auditor reported on the processing
of transactions?
Does the report only cover policies and procedures
in place?
Does the report test the policies and procedures?
Do we have a copy of the report?
CPO question: Is this application PC-based? (n/a,
n/a, n/a)
Is this application running on a standalone
PC?
CPO question: Does the client have the source
code for the computer programs used in this
accounting cycle? (n/a, n/a, 100%)
GCQ objective: Access control over source code is
adequate.
Is access to source code controlled?
ls the copying or renaming of sensitive programs
or utilities prevented/detected? ( n / a , n / a , 91%)
Are utilities, which have been made widely avail-
able (e.g. for reporting), restricted to readonly
access? (n/a, n / a , 96%)
Does the operating system provide operation log-
ging which records: all use of a compiler and the
targeted files, access and amendments to the pro-
gram libraries, attempts to copy program files
to/from the production library? ( n / a , n / a , 91%)
If source code is stored off-line on tape or car-
tridge, is the location physically secure? (n/a,
n / a , 96%)
If source code is stored off-line on tape or car-
tridge, is the individual responsible an individual
other than an applications programmer, systems
programmer or computer operator? ( n / a , n / a ,
96%)
Could or does the client make modifications to the
source code?
Is there a computer program that reports all
changes to each program and program library?
(n/a, n / a , 91%)
367
Are logs of all modifications to the source code
maintained? ( n / a , n / a , 91%)
Is access to and use of the application software
restricted? ( n / a , n / a , 87%)
Are source modifications authorized, tested, and
documented?
Does testing follow the predefined objectives of
the test? ( n / a , n / a , 74%)
Does testing use test data files instead of produc-
tion files for its tests? ( n / a , n / a , 91%)
Is a chronological record of all program amend-
ments maintained? ( n / a , n / a , 91%)
Is the system documentation updated to reflect
amendments? ( n / a , n / a , 87%)
Are modifications made by third-party vendors to
existing systems adequately recorded and con-
trolled to ensure that all procedures are appropri-
ately updated? (n/a, n / a , 91%)
Is the vendor software tested in the same way as
in-house developments? ( n / a , n / a , 83%)
Are emergency fixes to production programs fully
reported and independently reviewed by manage-
ment? (n/a, n / a , 96%)
Are the tests of the modified source code properly
supervised by management? ( n / a , n / a , 91%)
Do the tests of the modified source code use
complete computer systems to test the interaction
of different programs? ( n / a , n / a , 91%)
CPO question: Do multiple applications share the
same data bases (files)? (n/a, n/a, n/a)
GCQ objective: Data base application control is
adequate.
Is there an integrated dictionary system?
Is the data base administrator responsible for the
development of the data dictionary? (n/a, n/a,
n/a)
Are the data base changes requested by the data
base administrator approved by some other author-
ity? (n/a, n / a , n / a )
Is the distribution of the data dictionary restricted?
(n/a, n / a , n / a )
Is there a data base administrator?
Is the data base administrator responsible for con-
trolling, developing and maintaining the data dic-
tionary? (n/a, n / a , n / a )
368 J.T. Davis et al./ European Journal of Operational Research 103 (1997) 350-372
Does the data base administrator monitor data
base usage? ( n / a , n / a , n / a )
Are procedures established which allow access to
the data base only through the DBMS software
and prevent unauthorized access while the data
base is not under DBMS software control? ( n / a ,
n/a, n/a)
Does the data base administrator approved and log
all changes to the DBMS library? (n/a, n / a , n / a )
Is access to the computer operation area by the
data base administrator restricted or supervised?
(n/a, n/a, n/a)
Are utility programs under control of the data base
administrator? ( n / a , n / a ,
Does the data base administrator maintain and
review a log of programs run? ( n / a , n / a , n / a )
CPO question: Are there real-time updates to
files when transactions are entered? ( n / a , n / a ,
n/a)
GCQ objective: Control over real-time updates to
files is adequate.
Are control totals from the transaction history file
reconciled to the totals from the updated files?
(n/a, n/a, n/a)
If memo updating (A method in which the system
issues memo transactions to temporarily update a
copy of the file and then actual update is per-
formed in batches overnight) is used, is the up-
dated copy of the file matched against the master
file used during actual on-line processing? (n
/a n/a, n/a)
CPO question: Is the data processing function tbr
this accounting cycle decentralized (Distributed
Proce~ing)? (n/a, n / a , n / a )
GCQ objective: Control over distributed process-
ing applications is adequate.
Is a software log maintained for all transactions
including errors and retransmissions? (n/a, n / a ,
n/a)
Are data dictionary tools used to document and
monitor the responsibilities for various data items
or elements among the distributed network'? ( n / a ,
n/a, n/a)
Arc transaction logs maintained and reviewed for
all elements of the distributed data system? ( n / a ,
n/a, n/a)
Are software controls used to prevent update inter-
ference on central databases in distributed sys-
tems? (n/a, n / a , n / a )
CPO Question: Are telecommunications or net-
works used in this accounting cycle? (n/a, 100%,
100%)
GCQ Objective: Control over telecommunications
a n d / o r networks is adequate.
Is physical access to the terminals controlled?
Is the physical access of microcomputers or termi-
nals maintained by either equipment locks or con-
trol over location? ( n / a , 100%, 100%)
Is the use of terminals controlled by passwords?
Are terminals automatically locked out after failed
sign-on? (n/a, 100%, 96%)
Are terminals automatically locked out if the ter-
minal is inactive for a specified period of time?
( n / a , 100%, 96%)
Are sign-on and sign-off procedures specified and
then verified by the computer system? ( n / a , 100%,
96%)
Is access to all data files that are in the process of
being updated (i.e. 'live' files) controlled by pass-
word? (n/a, 100%, 100%)
Do users refrain from using common passwords
(first name, spouse's name, birth date, etc.)? (n/a,
100%, 100%)
Do users refrain from displaying passwords exter-
nally (e.g. post-it notes) on the terminals? (n/a,
100%, 96%)
Is the use of batch files to log onto the system
prohibited'? (n/a, 100%, 96%)
Are passwords protected and changed on a regular
basis?
Is the display of passwords prevented during log
on? (n/a, 100%, 100%)
Upon termination or resignation, are employees
immediately denied any computer access? ( n / a ,
100%, 100%)
Are passwords stored using a on-way encryption
algorithm? (n/a, 100%, 96%)
Are the system user rights transaction or applica-
tion specific?
 J.T. Davis et al. / European Journal of Operational Research 103 (1997) 350-372
Are system user rights established, monitored and
changed by a system or network administrator?
(n/a, 100%, 91%)
Is the ability to change system user rights pro-
tected by a system administrator or supervisor
password? ( n / a , 100%, 91%)
Is the system administrator or supervisor password
changed frequently? ( n / a , 100%, 91%)
Has the system administrator or supervisor pass-
word been changed from default password set-
tings? ( n / a , 100%, 91%)
Is the confidentiality of the system administrator
or supervisor Password maintained? ( n / a , 100%,
91%)
Is the system administrator or supervisor password
encrypted? ( n / a , 100%, 91%)
Is the system accessible by modem?
Is data encrypted during transmission and on the
files? ( n / a , n / a , n / a )
Are the phone numbers changed periodically?
(n/a, n / a , n / a )
Are the phone numbers listed on the modem or
terminal? ( n / a , n / a , n / a )
Are there procedures to prevent line tapping, un-
listed numbers and auto connection of dial-up
lines? (n/a, n / a , n / a )
Is password and transaction code documentation
secured against unauthorized access? ( n / a , n / a ,
n/a)
CPO question: Does the software in this account-
ing cycle generate transactions or pass informa-
tion to other cycles? (19%, 93%, 96%)
GCQ objective: Control over computer generated
transactions is adequate.
Are the methods used in the program to generate
the data and related control record appropriate?
(19%, 93%, 96%)
Is there an adequate check over the accuracy of
the data generated (e.g. reasonableness check,
manual review of data generated, etc.)? (19%,
93%, 96%)
Are the results of the check for accuracy reviewed
and approved? (19%, 93%, 96%)
Are controls such as computer sequence checks,
computer matching or batch totals used to ensure
the completeness of input a n d / o r updates? (19%,
93%, 96%)
369
Are exception reports investigated and reconciled?
(19%, 93%, 96%)
CPO question: Is there a significant loss of visible
audit trail in this accounting cycle? (n/a, n / a ,
n/a)
GCQ Objective: Control over a 'paperless' audit
trail is adequate.
For paperless or EDI transactions, are control
totals such as batch totals, sequence numbers and
'line item counts' maintained which: track missing
transactions, validate completeness, ensure one-
time-only receipt of a transaction, match func-
tional acknowledgment reports, review exception
reports to ensure corrections? (n/a, n / a , n / a )
CPO question: Have there been any hardware or
software malfunctions which resulted in a loss of
data in this accounting cycle? (n/a, n / a , n / a )
GCQ objective: Backup control is adequate.
Was data restored by use of backup?
Were there restart facilities which enabled pro-
cessing to continue from point of interruption?
(n/a, n/a, n/a)
Were there procedures which identified the pro-
cessing stage reached at the time of malfunction?
(n/a, n / a , n / a )
Were there controls to ensure that program li-
braries (most recent versions) and data files (most
recent backup or copy) were restored subsequent
to the failure or malfunction? (n/a, n / a , n / a )
Was data restored by manual re-entry?
Were controls (batch totals, control totals, etc.)
used to ensure that the restoration of data was
complete and accurate? (n/a, n / a , n / a )
Was data lost and not restored'?
Was the nature of the data insignificant? (n/a,
n/a, n/a)
Was the affect upon the financial statements im-
material? ( n / a , n / a , n / a )
A.4. A c c o u n t i n g control variables (Phase 2)
Balancing receivable subledger (1 9%, 20%, 43%)
Balancing run to run control totals (11%, 13%,
57%)
Balancing the G / L with the subledgers (52%,
47%, 17%)
370 J.T. Davis et al. / European Journal of Operational Research 103 (1997) 350-372
Canceling original documents (11%, 27%, 13%)
Checking by computer for duplicate entries (11%,
27%, 30%)
Checking by computer the numerical sequence of
a file (7%, 20%, 39%)
Checking for a third party signature (7%, 13%,
9%))
Checking manually the numerical sequence of a
document, journal or report (0%, 20%, 17%)
Checking one-to-one (26%, 7%, 9%)
Comparing batch totals (4%, 7%, 43%))
Comparison of budgeted amounts to actual
amounts (0%, 7%, 9%)
Comparison of cash receipts listing to deposit slips
(0%, 27%, 17%)
Computer generation of transactions (0%, 0%,
o%)
Dual control over cash receipts (11%, 20%, 4%)
Electronic authorization (0%, 7%, 0%)
Independent review of edit reports for data file
changes (0%, 20%, 13%)
Interactive dependency edit (0%, 0%, 4%)
Interactive document reconciliation (0%, 0%, 4%)
Interactive edit controls (0%, 7%, 0%)
Interactive existence edit (0%, 0%, 0%)
Interactive feedback edit (0%, 0%, 0%)
Interactive format edit (0%, 0%, 0%)
Interactive key verification (0%, 7%, 0%)
Interactive mathematical accuracy check (7%,
20%, 9%)
Interactive prior data matching (0%, 0%, 0%)
Interactive reasonableness edit(O% 13%, 4%)
Interactive range check edit (0%, 0%, 0%)
Interactive check digit (0%, 0%, 4%)
Matching to a previously validated document
(37%, 40%, 30%)
Matching to a previously validated file (0%, 27%,
52%)
Matching to an authorized list (26%, 47%, 22%)
Monthly review of bank reconciliations (0%, 13%,
4%)
Monthly review of receivable (7%, 13%, 13%)
Non-interactive format edit (0%, 0%, 0%)
Non-interactive mathematical accuracy (0%, 0%,
o%)
Non-interactive edit controls (0%, 0%, 0%)
Non-interactive existence edit (0%,0%, 0%)
Non-interactive dependency edit (0%, 0%, 0%)
Non-interactive range check edit (0%, 0%, 0%)
Non-interactive reasonableness edit (0%, 0%, 0%)
Non-interactive prior data matching (0%, 0%, 0%)
Non-interactive check digit (0%, 0%, 0%)
Performance of analytical procedures and investi-
gation of unusual items (0%, 13%, 17%)
Periodic reconciliation of books to physical (7%,
13%, 4%)
Periodic revision of budgeted amounts based on
updated information (0%, 0%, 4%)
Physical access controls (7%, 13%, 4%)
Physical safeguards (7%, 7%, 4%)
Reconciliation of manual totals to run totals (0%,
7%, 17%)
Reconciliation of master file balance to control
account (0%, 34%, 9%)
Reconciliation of master file balance to run totals
(7%, 7%, 35%)
Reperformance (0%, 7%, 0%)
Restrictive endorsement of checks received (0%,
7%, 4%)
Reviewing adjustment transactions (0%, 7%, 4%)
Reviewing internal signatures (7%, 20%, 9%)
Reviewing permanent data file exception reports
(0%, 0%, 0%)
Reviewing reference file data (0%, 0%, 0%)
Software access controls (0%, 33%, 9%)
Use of a Iockbox (7%, 7%, 0%)
Use of prerecorded input (OCR, MICR, OMR)
(0%, 0%, 0%)
Verifying mathematical accuracy (7%, 7%, 4%)
Other: Review, compare invoice adjusted for out
of stock items with the packing slip (11%, 14%,
0%)
Other: Driver gets signature on annotated sales
invoice. (19%, 0%, 0%)
Other: Cost of goods entry is calculated from sales
invoices and calculations reviewed (4%, 0%, 0%)
Other: Credit Check on invoice customer (0%,
0%, 0%)
Other: Compares amounts written off to aged trial
balance and customer detail (0%, 0%, 0%)
References
A I C P A , 1989. A m e r i c a n Institute o f Certified Public Accountants.
Audit guide: Consideration of the internal control structure in
a financial statement audit. N e w York, NY.
 J.T. Davis et al. / European Journal of Operational Research 103 (1997) 350-372
Anderson, J., 1983. A spreading activation theory of memory.
Journal of Verbal Learning and Verbal Behavior, 261-295.
Bataineh, S., AI-Anbuky, A., Al-Aqtash, S., 1996. An expert
system for unit commitment and power demand prediction
using fuzzy logic and neural networks. Expert Systems, 29-40.
Bell, T., Ribar, S., Verchio, J., 1990. Neural nets vs. logistic
regression: A comparison of each model's ability to predict
commercial bank failures. KPMG Peat Marwick Working
paper, May 1990.
Beulens, A., Van Nunen, J., 1988. The use of expert system
technology in DSS. Decision Support Systems 4, 421-431.
Borthick, A.F., West, O., 1987. Expert systems - A new tool for
the professional. Accounting Horizons 1, 9-16.
Brown, C., Coakley, J., Phillips, M.E., 1995. Neural networks
enter the world of management accounting. Management Ac-
counting, 51-57.
Brown, C., Phillips, M., 1990. Expert systems for management
accountants. Management Accounting, 18-22.
Brown, C., O'Leary, D., 1994. Introduction to Artificial Intelli-
gence and Expert Systems. Published Monograph, 1994.
Caudill, M., 1991. CRA expert networks. Byte, 108-116.
Cheng, B., Titterington, D., 1994. Neural networks: A review
from a statistical perspective. Statistical Science 9, 2-54.
Coakley, J., Brown., C. 1993. Artificial neural networks applied to
ratio analysis in the analytical review process. International
Journal of Intelligent Systems in Accounting, Finance and
Management, 19-39.
Craven, M., Shavlik, J., 1996. Extracting tree-structured represen-
tations of trained networks. In: Advances in Neural Informa-
tion Processing Systems 8. MIT Press, Cambridge, MA.
Davis, E., 1994. Effects of decision aid type on auditors' going
concern evaluation. Audit Judgment Symposium, Co-spon-
sored by Grant Thornton and University of Southem Califor-
nia, pp. 21-22.
Davis, J. 1996. Experience and auditors' selection of relevant
information for preliminary control risk assessment. Auditing:
Journal of Practice and Theory, 16-37.
Dcng, P., 1994. Automating knowledge acquisition and refine-
ment for decision support: A connectionist inductive inference
model. Decision Sciences 24 (2), 371-393.
Dreyfus, H., Dreyfus, S., 1986. Mind Over Machine: The Power
of llurnan Intuition and Expertise in the Era of the Computer.
Free Press, New York, NY.
Eining, M., Jones, D. and Loebbecke, J., 1994. An experimental
examination of the impact of decision aids on the assessment
and evaluation of management fraud. Audit Judgment Sympo-
sium, Co-sponsored by Grant Thornton and University of
Southern California, February 21-22.
Frisch, A., Cohn, A., 1991. Thoughts and after-thoughts on the
1988 Workshop on Principles of Hybrid Reasoning. A1 Maga
zinc: Special Issue, January 1991, 77-.83.
Gallant, S. 1993. Neural Network Learning and Expert Systems.
MIT Press, Cambridge, MA.
Gillett, P., Bamber, E., Mock, T., Trot.man, K., 1995. Audit
judgment. In: Bell, T., Wright, A. (Eds.), Auditing Practice,
Research, and Education: A Productive Collaboration. Ameri-
can Institute of Certified Public Accountants in Cooperation
371
with the Auditing Section of the American Accounting Associ-
ation, New York.
Grant Thornton, 1992. Information and control understanding
system version 2.08.
Gray, G., McKee, T., Mock, T., 1991. The future impact of expert
systems and decision support systems on auditing. Advances
in Accounting 9, 249-273.
Greeno, J.G. 1973. The structure of memory and the process of
solving problems. In: Solso, R.L. (Ed.), Contemporary Issues
in Cognitive Psychology: The Loyola Symposium. Wiley,
New York, NY.
Gupta, U., 1994. How case-based reasoning solves new problems.
Interfaces 24 (6), 110-119.
Hecht-Nielsen, R., 1990. Neurocomputing. Addison-Wesley,
Reading, MA.
Hedberg, S., 1995. Where's AI hiding? AI Expert, 17-20.
Hornick, K., Stinchcombe, M., White, M., 1989. Multilayer feed-
forwared networks are universal approximators. Neural Net-
works 2, 359-366.
Jackson, P. 1990. Introduction to Expert Systems. Addison-Wes-
ley, Workingham, MA.
Johnson-Laird, P.N., 1983. Mental Models: Toward a Cognitive
Science of Language, Inference, and Consciousness. Harvard
University Press, Cambridge, MA.
Kandel, A., Langholz, G., 1992. Hybrid Architectures for Intelli-
gent Systems. CRC Press, Boca Raton, FL.
Koopmans, L. 1987. Introduction to Contemporary Statistical
Methods. Duxbury Press, Boston, MA.
Kunz. J., Kehler, T., Williams, M., 1987. Applications develop-
ment using a hybrid A1 development system. In: Hawley, R.
(I-M.), Artificial Intelligence in Programming Environments.
Ellis Horwood, Chichester.
Lacher, R., Coats, P., Shanker, C., Franklin, L., 1995. A neural
network for classifying the financial health of a firm. Euro-
pean Journal of Operational Research 85 (1), 53-65.
Liberatore, M., Stylianou, A., 1993. The development manager's
advisory system: A knowledgebased DSS tool for project
assessment. Decision Sciences 24 (5), 953-976.
Liberatore, M., Stytianou, A., 1995. Expert support systems for
new product development decision making: A modeling
framework and applications. Management Science 41 (8),
1296-1316.
Lin, C., Hendler, J., 1995. Examining a hybrid connections/sym-
bolic system for the analysis of ballistic signals. In Sun, R.,
Bookman, L. (Eds.), Computational Architectures Integrating
Neuraland Symbolic Processes. Kluwer Academic Publishers,
Norwell, MA, pp. 319-348.
Lippmann, R., 1987. An introduction to computing with neural
nets. IEEE ASSP Magazine, 4-22.
t.ymer, A., Richards, K., 1995. A hybrid-based expert system for
personal pension planning in the UK. Intelligent Systems in
Accounting, Finance and Management 4, 71-88.
Maclntyre, J., Smith, P., Harris, T., Industrial experience: The use
of hybrid systems in the power industry. In: Medsker, L. (Ed.),
Hybrid Intelligent Systems. Kluwer Academic Publishers,
Norwell, MA, pp. 57-74.
MacLennan, B., 1993. Continuous symbol systems - The logic of
372 J.T. Davis et al. / European Journal of Operational Research 103 (1997) 350-372
connectionism. In: D.S. Levine, M. Aparicio IV (Eds.), Neural
Networks for Knowledge Representation and Inference.
Lawrence Erlbaum, Hillsdale, NJ.
Markham, I., Ragsdale, C., 1995. Combining neural networks and
statistical predictions to solve the classification problem in
discriminant analysis. Decision Sciences 26 (2), 229-242.
Medsker, L., 1994. Design and development of hybrid neural
network and expert systems. In: Proceedings of the IEEE
International Conference on Neural Networks III, Orlando, FL,
pp. 1470-1474.
Medsker, L., Bailey, D., 1992. Models and guidelines for integrat-
ing expert systems and neural networks. In: Kandel, A.,
Langholz, G. (Eds.), Hybrid Architectures for Intelligent Sys-
tems. CRC Press, Boca Raton, FL, pp. 153-171.
Medsker, L., 1995. Hybrid Intelligent Systems. Kluwer Academic
Publishers, Norwell MA.
Minsky, M., 1986. The Society of Mind. Simon and Schuster,
New York, NY.
NeuralWare: Technical Publications Group, 1992. Neural comput-
ing - A technology handbook for professional ll/plus and
neuralWorks explorer. Pittsburgh, PA.
O'Leary, D., Watkins, P., 1991. Review of expert systems in
auditing. Expert Systems Review for Business and Account-
ing, 3--22.
Ripley, B., 1994. Neural networks and related methods for classi-
fication. Journal of the Royal Statistical Society B 56 (3),
409-456.
Ripley, B., 1993. Statistical aspects of neural networks. In: Barn-
dorff-Nielsen, O., Jensen, J., Kendall, S. (Eds.), Networks and
Chaos - Statistical and Probabilistic Aspects. Chapman and
Hall, London, pp. 40-123.
Rosenhcad, J., 1992. Into the swamp: The analysis of social
issues. Journal of Operational Research Society 43 (4), 293-
305.
Rumelhart, D., McClelland, J., 1986. Parallel Distributed Process-
ing: Volumes I and II. The MIT Press, Cambridge, MA, pp.
110-146.
Sarle, W., 1994. Neural networks and statistical models. In:
Proceedings of the Nineteenth Annual SAS Users Group
International Conference. SAS Institute, Cary NC, pp. 1538-
1550.
Scott, D., Wallace, W., 1994. A second look at an old tool:
Analytical procedures. The CPA Journal, 30-35.
Silverman, B., 1995. Knowledge-based systems and the decision
sciences. Interfaces 25 (6), 67-82.
Solso, R., 1988. Cognitive Psychology, 2nd edition. Allyn and
Bacon, Boston, MA.
Sun, R., Bookman, L., 1995. Computational Architectures Inte-
grating Neural and Symbolic Processes. Kluwer Academic
Publishers, Norwell, MA.
Sun, R., 1994. A two-level hybrid architecture for structuring
knowledge for commonsense reasoning. In: Sun, R., Book-
man, L. (Eds.), Computational Architectures Integrating Neu-
ral and Symbolic Processes. Kluwer Academic Publishers,
Norwell, MA, pp. 247-282.
Sutton, S., Young, R., McKenzie, P., 1995. An analysis of
potential legal liability incurred through audit expert systems.
Intelligent Systems in Accounting, Finance and Management
4, 191-204.
Tam, K., Kiang, M., 1992. Managerial applications of neural
networks: The case of bank failure predictions. Management
Science 38 (7), 926-947.
Turban, E., Watkins, P.R., 1986. Integrating expert systems and
decision support systems. MIS Quarterly 10 (2), 121-136.
Yoon, Y., Guimaraes, T., Swales, G., 1993. Integrating artificial
neural networks with rule-based expert systems. DSS Special
Issue on Artificial Neural Networks.