Extracting Rules from Neural Networks Using

Symbolic Algorithms: Preliminary Results

Claudia Regina Milar
Andr Carlos Ponce de Leon Ferreira de Carvalho
Maria Carolina Monard
Laboratory of Computational Intelligence - LABIC
Department of Computer Science and Statistics - SCE
Institute of Mathematics and Computer Science - ICMC
University of So Paulo USP
P. O. Box 668, 13560-970 So Carlos, SP, Brazil
e-mail: {claudia, andre, mcmonard}@icmc.sc.usp.br
Abstract Although Artificial Neural Networks (ANNs) have satisfactory emploied
in several problems, such as clustering, pattern recognition, dynamic systems control
and prediction, they still suffer of a significant limitation. The representations they
learn are not usually comprehensible to humans. Several techniques have been
suggested to extract meaninful knowledge from trained ANNs. This paper proposes
the use of symbolic learning to extract symbolic representations from trained ANNs.
These algorithms are commonly used by the Machine Learning community. They
learn concepts represented by propositional description. The procedure proposed is
similar to that used by the Trepan algorithm (Craven, 1996). This algorithm extract
comprehensible, symbolic representations (decision trees) from trained ANNs.
Trepan queries a trained network to induce a decision tree that describes the concept
represented by the ANN.
Keywords: Artificial Neural
Representation, Decision Trees.

Networks,

Knowledge

Extraction,

Symbolic

1. Introduction
Although ANNs have achieved a good performance when applied to a large number of problems (Hall, 1992; LeCun et
al., 1990; Moreno et al., 1998), it is not clear to the user of ANNs how they make their decisions. Artificial Neural
Networks (ANNs) usually represent the acquired knowledge as a set of numerical values associated to connections
between two nodes. There are several applications where the ability to explain the decisions taken is so or even more
important than the performance. This lack of comprehensability of their decision making process is one of the main
barriers to the widespread use of ANNs to solve practical problems. This ability is essential in applications where safety
is an important aspect. Examples of these applications are: electric motor failure prediction, aircraft flight, nuclear
power plant control, surgery support, etc.. Due to this limitation, ANNs are commonly referenced as black boxes.
This paper proposes the of use symbolic learning algorithms, commonly used by the Machine Learning community,
to extract symbolic representations from trained ANNs. The symbolic representations that these algorithms learn,
specifically decision trees and rules, are more comprehensible to ANNs users. The procedure proposed in this paper is
similar to that used by the Trepan algorithm (Craven, 1996). Trepan extracts comprehensible, symbolic representations
from trained ANNs. Trepan queries a given network to induce a decision tree that describes the concept represented by
the ANNs.
This paper is organised as follow. Section 2 briefly describes previous works on knowledge extraction from ANNs,
in particular the Trepan algorithm. Section 3 describes experiments using three different algorithms: C4.5, CN2 and
Trepan. The results obtained are compared and analysed. Finally, Section 4 presents the main conclusions of this work
and suggest a few future works.

2. Related Works in Knowledge Extraction

In order to overcome the incomprehensibility of the representations that ANNs learn, various attempts have previously
been made to extract knowledge from ANNs. The methods that are used to extract knowledge from ANNs can be
categorised as pedagogical (also known as global), decompositional (also know as local), eclectic and compositional
(Andrews et al., 1995; Tickle et al., 1998). This classification is based on two aspects: the approach used to characterise
the internal model of the network or/and the classification given by network for the examples presented.

2.1 Pedagogical or Global Methods

The pedagogical or global methods see a trained network as a black box. The methods of this approach extract
knowledge that characterise the output classes directly from the inputs.
The method developed by Saito and Nakano (1988) search for combinations of input values which activate each
output unit. One problem with this method is that the size of the search space can grow exponentially with the number
of input values. To deal with this problem, the authors used two heuristics that limited the search space. Other limitation
is that this method can only be used with discrete-valued features.
Gallant (1993) developed a method similar to Saito and Nakanos method. In this method, the search for
combinations of input values that activate a given output value is also heuristically reduced. Gallants method use a
procedure to test the combinations of input values (rules) against the network.
Thrun (1995) developed a method called Validity Internal Analysis (VIA). Similar to the Gallants method, Thrun
uses a procedure to test the rules against the network. In this method, linear programming is used to determine if a set of
constraints placed on the network activation values is consistent.
The method known as EN (Pau and Gtzche, 1992), allows How?, Why? and Trace queries for multilayer networks.
EN permits a given input/output to be associated to the corresponding output/input. The explanation facility provided by
EN is based on a general explanation degree calculation as well as on how the most relevant units in each layer are
chosen.
The ANN-DT algorithm (Artificial Neural-Network Decision Tree) (Schmitz et al., 1999) uses the network to
generate outputs for samples interpolated from the training data set. ANN-DT can extract rules from feedforward
networks with continuous outputs.
Another pedagogical method was developed by Tchoumatchenko and Ganascia (1994). This method extracts
majority-vote rules (a list of literals that can be considered as either evidence for, or evidence against, a particular class)
from trained networks. The knowledge extracted by this method do not provide a good approximation of the knowledge
stored in a network, because it do not take into account how the hidden units combine evidence from multiple input
units (Craven, 1996).
One of the methods used in this paper, the Trepan (TREes Parroting Networks) (Craven, 1996), extracts a decision
tree from a trained network. Given a trained network and the data on which it was trained, Trepan seeks to create a
comprehensible concept description, the decision tree, that classifies instances in the same way as the network does
(Craven and Shavlik, 1996). The trained network is used as an oracle that is able to answer queries during the
learning process. The oracle (the network) determines the class of each instance that is presented in the query. A query
can be a training data instance or an artificially generated instance created by randomly selecting values for each
feature, while ensuring that the constraints for each feature value are satisfied. Unlike most decision tree algorithms,
which use single feature to partition the input space to separate the instances of different classes, Trepan uses m-of-n
expressions for its splits. A m-of-n expression is a Boolean expression which is satisfied when at least m (a integer
threshold) of its n conditions (Boolean conditions) are satisfied. This algorithm can be applied to any ANN architecture
and training method.
2.2 Decompositional or Local Methods
Decompositional or local methods extract rules from individual units. The network is decomposed in small networks
(one unit). The rules extracted from these small networks are combined to form a global relationship.
The KT method proposed by Fu (1991, 1994) search for combinations of weights that, when satisfied, guarantee that
a given unit is active, regardless of the state of the other inputs to the unit. To reduce the size of the search space, KT
limits the number of antecedents per extracted rule.
Similar to Fus method, Towell and Shavlik (1993) developed a method called M-of-N. For each unit, this method
searches for sets of input connections whose summed weights exceed its bias, irrespective of the weights associated to
the other input connections. The rules extracted by this method are m-of-n expressions. Remember that a m-of-n
expression is a Boolean expression which is satisfied when at least m (a integer threshold) of its n conditions (Boolean
conditions) are satisfied.
Hayashi (1991) developed a method similar to the KT method developed by Fu. The main difference is on the rule
conditions. The rules extracted by this method can present fuzzy conditions.
Another method, developed by Tan (1994), extracts rules from a particular network called Fuzzy ARTMAP
(Carpenter et al., 1992). The rule extracting process consists directly translating parts of the networks architecture into
rules. The rules extracted by this method have an associated certainty factor.
2.3 Eclectic Methods
Eclectic methods combine the previous approaches. They analyse the ANN at the individual unit level but also extract
rules at the global level.
One example of this approach is the method proposed by Tickle et al. (1994) called DEDEC. DEDEC extracts ifthen rules from MLP networks trained with backpropagation algorithm (Rumelhart et al., 1986). DEDEC combines the
knowledge embedded on the networks weight and rules extracted by a symbolic algorithm.

2.4 Compositional Methods

The methods from this category can be neither decompositional, because they do not extract rules from individual units,
nor eclectic, because they do not use the network as black box, like the pedagogical approach. The compositional
approach was proposed to accommodate methods which extract rules from ensembles of units.
One example this approach is the method proposed by Schelllammer et al. (1997). The authors trained recurrent
neural networks (Elman networks and Recurrent Cascade Correlation) to learn sequences of word categories from a
first-year primary school text. The knowledge extracted of network by this method is a deterministic finite-state
automaton (DFA). This automaton represents the grammar induced by the network.

3. Proposed Approach
The methods for knowledge extraction discussed in Section 2 have some restrictions. Exampoes of these restrictions
are:
The network models to which they can be applied. Sometimes a network model or training algorithm is
created specifically for a knowledge extraction method employed.
The difficulty to apply them to large networks. Some methods do not scale well with the network size. The
size of the rule sets extracted by decompositional methods (which extract a rule set for each hidden unit) tends
to increase with the network size.
Some methods can only be applied to problems that have exclusively discrete-valued features. This either
restricts the application domain or requires a discretization of the attribute values. The the attribute values are
discretized, the rule set extracted from network can become more complex than the neural network.
The Trepan algorithm does not suffer from the limitations previously described. Trepan is a general algorithm, i.e., it
does not assume that the networks have any particular architecture, nor that they were trained in any special way. The
decision trees derived by Trepan are not directly affected by the network size. Trepan can be used for applications
involving both real-valued and discrete-valued feature. In spite of all this advantages, it is not clear to the authors why
Trepan, and not another symbolic learning algorithm like CN2 (Clark and Niblett, 1987) and C4.5 (Quinlan, 1993),
should be used to extracted the knowledge stored in ANNs.
The work presented in this paper aims to evaluate the performance of alternative knowledge extraction algorithms to
extract knowledge from ANNs. Its main goal is the development of a general procedure that, like the Trepan algorithm,
can be applied to any network model, independent its architecture, training process, feature values and size.
Basically, as described in the Section 1, the main idea is to investigate the use of symbolic algorithms commonly
used by the Machine Learning community that learn concepts represented by propositional description, such as C4.5
(Quinlan, 1993) and CN2 (Clark and Niblett, 1987), to extract symbolic representations from trained ANNs. The
concepts learned by C4.5 are represented by a decision tree. A decision tree can be translate into a set of if-then rules.
The concepts learned by CN2 are represented by a set of if-then rules. If-then rules are usually comprehensible and
natural to humans.

4. Experiments
To evaluate these alternative techniques, a set of experiments were carried out using the Wisconsin Breast Cancer
Dataset (called in this work breast cancer) from the UCI repository (Blake et al., 1998). Each instance has one of 2
possible classes: benign or malignant. The number of attributes is 10 and the number of instances is 699. There are 16
instances that contain a single missing attribute value. The class distribution is: benign with 458 instances (65.5%) and
malignant with 241 instances (34.5%). In this experiments, the 16 instances that contain a single missing attribute value
were removed. The first attribute (sample code number) was also removed.
The main purpose of the experiments performed is to compare the knowledge extracted by C4.5, CN2 and Trepan
according to two criteria: predictive accuracy and fidelity to the networks to which they were applied. Predictive
accuracy is defined as the percentage of test-set examples that are correctly classified. Fidelity is defined as the
percentage of test-set examples where the classifications made by a tree or rules set agrees with those made by the
associated ANN. The experiments are conducted using a 10-fold cross validation methodology. The accuracy and
fidelity were measured on the examples in the test sets.
The network architecture used in the experiments was chosen after experiments with several topologies. The chosen
architecture has 9 input units, 3 hidden units and 2 output units. The experiments with the ANNs were performed using
the SNNS simulator (SNNS, 1995). The networks were trained using backpropagation with momentum and a validation
set was used to decide when the training should stop.
The C4.5, CN2 and Trepan algorithms were used with default values assigned to their parameters. Additional values
were also investigated for the parameters associated to Trepan. Table 1 shows the variations of Trepan used in these
experiments. The MinParameter parameter specifies the minimum sample size (i.e. number of queries) used by Trepan
at each node in the decision tree. The default value is 1000. When this parameter is set to 0, no instance is generate by
Trepan. The TreeSizeLimit parameter specifies the maximum size of the tree to be returned by Trepan. The value

indicates the maximum number of internal nodes the tree may have. The default value is 100. The SplitTest parameter
defines if the tests at the internal nodes are m-of-n expressions (mofn) or single-attribute tests (lo_mofn), as is the case
at the internal nodes of the C4.5 algorithm.
MinParameter TreeSizeLimit
1000
100
Trepan (mofn)
0
100
Trepan (mofn)*
1000
100
Trepan (lo_mofn)
0
100
Trepan (lo_mofn)*
Table 1: Variants of Trepan.

SplitTest
mofn
mofn
lo_mofn
lo_mofn

Table 2 shows the test-set accuracy results obtained in the experiments. They show that the ANNs generalised
better the decision trees learned by C4.5 and Trepan, with their variations. The generalisation achieved by the ANNs
was also better than those obtained by the rules learned by CN2. Among the knowlege extraction techniques, the rules
learned by CN2 generalised better that the decision trees created by C4.5 and Trepan. Only one Trepan variant, Trepan
(mofn)*, generalised better that another symbolic algorithm, the C4.5 algorithm.
Error Rate
C4.5
4.98 1.31
CN2
3.95 0.70
Neural Network
3.81 0.77
Trepan (mofn)
5.84 1.28
Trepan (mofn)*
4.39 1.06
Trepan (lo_mofn)
6.59 0.93
Trepan (lo_mofn)*
5.12 1.09
Table 2: Test set accuracy.
Table 3 shows the test-set fidelity results for the experiments. To measure the fidelity between C4.5 and CN2 and
the ANN, the whole data set was applied to Trepan, C4.5 and CN2, using now the classification provided by the ANN
for the whole dataset. Similar to Trepan, C4.5 and CN2 induce their hypotheses from the classification given by the
trained ANN. Thus, the class labels are not necessarily the true class labels, but instead they are those determined by
networks classification of the instances.
According to Table 3, the decision trees extracted by Trepan (mofn)* provide a closer approximations to their
respective networks than the decision trees extracted by C4.5 and other Trepan variants and than the rules extracted by
CN2. It must be pointed out, however, that this variant do not uses an extra set of examples (query instances), which are
artificially created in order to improve its fidelity to a trained ANN. The rules extracted by CN2 also provide closer
approximations to their respective networks than the decision trees extracted by C4.5 and other three Trepan variants.
The differences in fidelity between Trepan and their variations and C4.5 and CN2 were not statistically significant. This
can be observed in Table 4.
Error Rate
C4.5
3.35 0.49
CN2
2.78 0.83
Trepan (mofn)
4.09 0.97
Trepan (mofn)*
2.33 0.69
Trepan (lo_mofn)
4.25 0.86
Trepan (lo_mofn)*
3.37 0.83
Table 3: Test set fidelity.
C4.5
CN2
0.96
1.45
Trepan (mofn)
-1.70
-0.59
Trepan (mofn)*
1.29
1.74
Trepan (lo_mofn)
0.03
0.71
Trepan (lo_mofn)*
Table 4: Significance test for fidelity.
The Trepans authors argue that a major limitation of conventional tree-induction algorithms is that the amount of
training data available to select splitting tests and label decreases with the depth of the tree. Tests and class labels near
the bottom of a tree are often poorly chosen because such decisions are based on a small group of data. Because Trepan

uses query instances in addition to training examples, it can use a larger set of samples to support the decisions made
near the botton of the tree.
However, the generation of new instances can be dangerous. Even though it is possible to certify that each artificial
instance has a valid feature value, it is very difficult to certify that each combination of the features values is valid.
Consequently, some of the generated instances would not be found in the real world. Experiments, although
preliminary, suggest that the generated instances may, in their attempt to increase the fidelity of Trepan to a trained
ANN, reduce the fidelity of Trepan to the trained ANN regarding the real samples and even degrade Trepans accuracy.

5. Conclusions
This paper proposed a new procedure to extract knowledge from trained ANNs. This procedure uses symbolic
algorithm, C4.5 and CN2, to induce comprehensible hypotheses from trained networks. In this process, each ANN is
used as a black box, providing classification for a set of training instances. Each symbolic algorithm induces its
hypotheses from instances classified by the trained network. The procedure is similar to that used by the Trepan
algorithm. The knowledge extracted by CN2 and C4.5 were compared to those extracted by Trepan. The results
obtained suggested that the fidelity and accuracy obtained by the CN2 and C4.5 algorithms can be superior to those
achieved by some variants of Trepan. The results also indicates the potential of both C4.5 and CN2 algorithms to be
used for extracting comprehensible information from trained ANNs. Another advantage of the proposed procedure is
its generality. Like Trepan, they can be applied to all the types of network without any restrictions.
In future works, the accuracy and fidelity of procedure proposed will be analised and validated for additional
domains. Experiments are being carried out with 9 additional datasets. Other aspect that willl be measured is the
comprehensibility of the decision trees and rules extracted.

Acknowledgments
We wish to thank Gustavo Batista for helpful suggestions.

References
Andrews, R., Diederich, J. and Tickle, A. B. (1995). Survey and Critique of Techniques for Extracting Rules from
Trained Artificial Neural Networks. Knowledge-Based Systems, 8(6): 373389.
Blake, C., Keogh, E. and Merz, C. (1998). UCI Irvine Repository of Machine Learning Databases.
http://www.ics.uci.edu/~mlearn/MLRepository.html.
Carpenter, G. A., Grossberg, S., Markuzon, N., Reynolds, J. H., and Rosen, D. B. (1992). A Neural Network
Architecture for Incremental Supervised Learning of Analog Multidimensional Maps. IEEE Transactions on Neural
Networks, 3:698-713.
Clark, P. and Niblett, T. (1989). The CN2 Induction Algorithm. Machine Learning, 3(4):261283.
Craven, M. W. (1996). Extracting Comprehensible Models from Trained Neural Networks. PhD Thesis, University of
Wisconsin, Madison.
Craven, M. W. and Shavlik, J. W. (1996). Extracting Tree-Structured Representations of Trained Networks. Appears in
Advances in Neural Information Processing Systems, vol. 8, MIT Press, Cambridge, MA.
Fu, L. M. (1991). Rule Learning by Searching on Adapted Nets. In Proceedings of the Ninth National Conference on
Artificial Intelligence, 590595.
Fu, L. M. (1994). Rule Generation from Neural Networks. In IEEE Transactions on Systems, Man, and Cybernetics,
24:11141124.
Gallant, S. I. (1993). Neural Networks Learning and Expert Systems. MIT Press.
Hall, C. (1992). Neural Net Technology: Ready for Prime Time?. IEEE Expert, 24.
Hayashi, Y. (1991). A Neural Expert System with Automated Extraction of Fuzzy If-Then Rules. In Advances in Neural
Information Processing Systems, vol. 3, San Mateo, CA. Morgan Kaufmann.
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W. and Jackel, L. D. (1990).
Backpropagation Applied to Handwritten Zip Code Recognition. Neural Computation, 1:541551.
Moreno, J., Sebastian, G., Fernandez, M. and Caballero, A. (1998). A Neural Architecture for the Identification of
Number Sequences. In Vth Brazilian Symposium on Neural Networks, 247252, Belo Horizonte, MG.
Pau, L. F., Gtzche, T. (1992). Explanation Facility for Neural Networks. Journal of Intelligent and Robotic Systems,
5:193-206.
Quinlan, J. (1993). C4.5: Programs for Machine Learning. Morgan Kaufmann.
Rumelhart, D. E., Hilton, G. E., and Willians, R. J. (1986). Learning Representations by Backpropagating Errors.
Nature, 323:533-536.
Saito, K. and Nakano, R. (1988). Medical Diagnostic Expert System Based on PDP Model. In Proceedings of the IEEE
International Conference on Neural Networks, 1:255262, San Diego, CA. IEEE Press.

Schelllmmer, I., Diederich, J., Towsey, M. and Brugman, C. (1997). Knowledge Extraction and Recurrent Neural
Networks: An Analysis of an Elman Network Trained on a Natural Language Learning Task, QUT-NRC Technical
Report 97-ISI.
Schmitz, G. P. J., Aldrich, C. and Gouws, F. S. (1999). ANN-DT: An Algorithm for Extraction of Decision Trees from
Artificial Neural Networks. IEEE Transactions on Neural Networks, 10(6):13921401.
SNNS V4.1 (Stuttgart Neural Network Software). Users Guide, 1995.
Tan, A. H. (1994). Rule Learning and Extration with Self-Organizing Neural Network. In Proceedings of the 1993
Connectionist Models Summer School, 192199, Hillsdale, NJ. Lawrence Erlbaum Associates.
Tickle, A. B., Andrews, R., Golea, M. and Diederich, J. (1998). The Truth will Come to Light: Directions and
Challenges in Extracting the Knowledge Embedded within Trained Artificial Neural Networks. IEEE Transactions
on Neural Networks, 9(6):10571068.
Tickle, A. B., Orlowski, M. and Diederich, J. (1994). DEDEC: Decision Detection by Rule Extraction from Neural
Network, Technical Report, QUT NRC.
Tchoumatchenko, I. and Ganascia, J. G. (1994). A Bayesian Framework to Integrate Symbolic and Neural Learning. In
Proceedings of the Eleventh International Conference on Machine Learning, 302308, New Brunswich, NJ.
Morgan kaufmann.
Thrun, S. (1995). Extracting Rules from Artificial Neural Networks with Distributed Representations. In Advances in
Neural Information Processing Systems, vol. 7, Cambridge, MA. MIT PRESS.
Towell, G., Shavlik, J. W. (1993). The Extraction of Refined Rules from Knowledge-Based Neural Networks. Machine
Learning, 131:71101.