European Journal of Operational Research 126 (2000) 526533

www.elsevier.com/locate/dsw

Theory and Methodology

Pattern classication with principal component analysis and fuzzy

rule bases
a,*,1
V. Ravi , P.J. Reddy b, H.-J. Zimmermann a

a
Lehrstuhl f
ur Unternehmensforschung, RWTH, Templergraben 64, D-52056 Aachen, Germany
b
Computer Center, Indian Institute of Chemical Technology, Uppal Road, Hyderabad 500 007, Andhra Pradesh, India
Received 8 December 1998; accepted 13 April 1999

Abstract

For the rst time, the principal component analysis has been used to reduce the feature space dimension in fuzzy rule
based pattern classiers. A modied threshold accepting algorithm (MTA) proposed elsewhere by V. Ravi and H.-J.
Zimmermann [European Journal of Operational Research 123 (1) (2000) 1628] has been used to minimize the number
of rules in the classier while guaranteeing high classication power. The proposed methodology has been demon-
strated for (1) the wine classication problem, which has 13 features and (2) the Wisconsin breast cancer determination
problem, which has 9 features. The inuence of the type of aggregator used in the classication algorithm and the
number of partitions used for each of the feature spaces is also studied. In conclusion, the results are encouraging as
there is no reduction in the classication power in both the problems, despite the fact that some of the principal
components have been deleted form the study before invoking the classier. On the contrary, however, the rst ve
principal components in both the problems yielded 100% classication power in some cases. The high classication
power obtained for both the problems while working with reduced feature space dimension is the signicant outcome of
this study. 2000 Elsevier Science B.V. All rights reserved.

Keywords: Fuzzy sets; Data analysis; Feature selection; Principal component analysis; Modied threshold accepting

1. Introduction

Ever since fuzzy set theory was propounded by

*
Corresponding author. Tel.: +91-40-715-1310; fax: +91-40-
717-3387.
E-mail address: vravi@iict.ap.nic.in (V. Ravi).
1
On deputation from Computer Center, Indian Institute of
Chemical Technology, Uppal Road, Hyderabad 500 007,
Andhra Pradesh, India. Fax: +91-40-7173387 or +91-40-
7173757.
Zadeh [13] as a new paradigm, its applications
grew dramatically over the past two decades.
Knowledge based systems are among the earliest
applications, where fuzzy ifthen rules play a
paramount role. Most of these systems derive
these fuzzy ifthen rules from human experts [3].
In the literature, one comes across several methods

0377-2217/00/$ - see front matter 2000 Elsevier Science B.V. All rights reserved.
PII: S 0 3 7 7 - 2 2 1 7 ( 9 9 ) 0 0 3 0 7 - 0
 V. Ravi et al. / European Journal of Operational Research 126 (2000) 526533
to generate these fuzzy ifthen rules directly from
numerical data. Kosko [6] employed neural net-
works to achieve this goal. Later, Ishibuchi et al.
[3] proposed a sound methodology to generate
such rules from numerical data and then they went
ahead to apply a genetic algorithm to determine a
compact rule set with a high classication power
[4]. Then, a software by name W I N R O S A [12],
which automatically generates fuzzy ifthen rules
from numerical data using statistical methods,
became available in the market. However, all the
aforementioned studies dier from that of [3,4] in
several aspects.
Throughout this paper, the partition of a pat-
tern space means its granularity. To generate fuzzy
ifthen rules from numerical data one must (i) nd
the partition of a pattern space into fuzzy sub-
spaces and (ii) determine the fuzzy ifthen rule for
each fuzzy partition [3,4]. Using those fuzzy if
then rules, either the training data or the test data
are classied, which is essentially the classication
phase. The performance of such a classier de-
pends very much on the choice of a fuzzy partition.
If a fuzzy partition is too coarse, many patterns
may be misclassied. On the other hand, if it is too
ne, many fuzzy ifthen rules cannot be generated
due to the lack of training patterns in the corre-
sponding fuzzy subspaces. In their earlier paper,
Ishibuchi et al. [3] have proposed distributed fuzzy
rules, by considering the fuzzy rules corresponding
to both coarse and ne partitions of a fuzzy sub-
space. For example, a two-dimensional pattern
space gives rise to 90 22 32 42 52 62
fuzzy ifthen rules, assuming that each feature
dimension is divided into 6 partitions at the most.
Thus, they considered all 5 rule tables corre-
sponding to all the partitions simultaneously. Also
by considering all the fuzzy partitions simulta-
neously, the above mentioned diculty in choos-
ing an appropriate partition is obviated.
The main drawback of this approach, however,
is that the number of fuzzy ifthen rules increase
exponentially for classication problems with high
dimensional pattern spaces [5] such as wine clas-
sication problem [3] where 13 feature variables
are present. For example, if up to 5 partitions are
used for each of the 13 feature variables, the total
number of rules would be 213 313 413 513 . To
527
overcome this problem, Ishibuchi et al. [6] have
introduced a method where the fuzzy ifthen rules
with a small number of antecedent conditions are
generated as candidate rules. However, the au-
thors are of the opinion that it is still not a com-
plete remedy because the method is not a general
one and it is not dicult to nd problems where
rules with a small number of antecedent conditions
are intractable. This motivated us for the devel-
opment of other alternative methods which con-
centrate on feature selection or reduction of
feature space dimension by transforming it. Thus
it is meaningful to look for any unimportant
variables (features) and remove them from the
classication process. This results in reduced
computational time and memory requirement and
an easy-to-use classier with a manageable num-
ber of features. Thus the point we would like to
drive home is that the feature selection is an es-
sential component of any classier, specially in
dealing with problems having a large number of
features. Ravi and Zimmermann [9] addressed this
problem by resorting to the use of a software plug-
in to DataEngine, viz FeatureSelector [11]. They
used it as a pre-processor to select the most salient
features from the original set of features and went
on to derive a compact set of fuzzy ifthen rules
with high classication power.
In the present paper, however, the authors
propose another way of reducing the feature space
dimension, via the principal component analysis
(PCA). It is a traditional multivariate statistical
technique frequently used for data compression
[10]. However, the authors make it abundantly
clear that a comparison of the present study with
our earlier one [9] is simply not meaningful be-
cause the principal components are only linear
combinations of the original feature variables and
hence, they do not reect on the importance or
otherwise of the original variables. The rest of the
paper has been structured as follows. Section 2
gives an overview of the principal component
analysis and an algorithm to compute the principal
components. Section 3 briey presents the fuzzy
rule based classication method and the formula-
tion of the multi-objective optimization problem.
Results of the numerical simulations are discussed
in Section 4, and Section 5 concludes the paper.
528 V. Ravi et al. / European Journal of Operational Research 126 (2000) 526533
2. Principal component analysis
2.1. Algorithm to determine principal components
[10]
1. In the data matrix X of order n  j, where
n > j; n is the number of patterns or observa-
tions and j is the number of features. Center
and scale all the features, i.e. for a given col-
umn, its mean is subtracted from all the obser-
vations in that column (each column
corresponds to a feature) and every time the re-
sult is divided by the standard deviation of the
column. This is essential to make the features
dimensionless.
2. Form the matrix of squares and products of the
features Z T Z, where Z is the centered and scaled
version of the matrix X.
3. To perform the principal component analysis
on the matrix Z, either the singular value de-
composition of Z or the eigen analysis of the
correlation matrix of Z is carried out. In the
present paper, the latter approach is followed.
The correlation matrix is obtained by dividing
each entry of Z T Z by n 1. Eigen analysis
of the correlation matrix results in as many ei-
genvalues and eigenvectors as the number of
features and each eigenvector contains as many
entries. Each entry is called the principal com-
ponent loading. The principal components are
calculated as P XE, where X is the original
data matrix of order n  j, E the matrix of or-
der j  j, whose columns are the eigenvectors
of Z T Z and P is the matrix of order n  j of
principal components P1 ; P2 ; . . . ; Pj , where Pi is
an n  1 vector, i 1; 2; . . . ; j. The principal
components are pairwise orthogonal unlike
the original features.
4. The ratio of each of the eigenvalue to the total
sum of all the eigenvalues indicates the propor-
tion of variation accounted for by the corre-
sponding principal component. Small
eigenvalues correspond to those dimensions
which cause multicollinearity. A salient feature
of the principal component analysis is that the
eigenvalues are automatically sorted in the de-
scending order and also their corresponding ei-
genvectors are rearranged accordingly. Then,
the rst principal component accounts for the
maximum variance and the second principal
component accounts for the second largest vari-
ance and so on.
2.2. A method to reduce feature space dimension
using PCA
To reduce the feature space dimension, those
principal components, which correspond to
smaller eigenvalues should be deleted. To do this,
the guiding rule is to prespecify the percentage of
total variation we would like to account for using
the principal components. Using all the principal
components as new features in the subsequent
analysis amounts to using the original features
themselves, because each principal component is
nothing but a linear combination of the original
features. Thus the information contained in the
original features is recast into a more rened form
where the principal components are pairwise or-
thogonal. In view of the foregoing, in this paper, a
threshold limit of 84% (90%) is set up for problem
1 (problem 2), because it leads to the rst 5 prin-
cipal components as the most important ones.
These ve principal components, which account
for 89% and 92% of the variance respectively in
problems 1 and 2, are treated as the new set of
features.
3. A fuzzy classication method with fuzzy ifthen
rules
Following the notation of [3,4], let the pattern
space be two-dimensional (i.e. there are two
features in the feature space) and given by the
unit square 0; 1  0; 1. Suppose that Xp
xp1 ; xp2 ; p 1; 2; . . . ; m; are given as training
patterns from M classes (where M  m): Class 1
(C1), Class 2 (C2); . . . ;Class M (CM). The
problem is to generate fuzzy ifthen rules that
divide the pattern space into M disjoint classes.
For more details of the extension to the case of
higher dimensions, the reader is referred to [4].
Let each axis of the pattern space be partitioned
into K fuzzy subsets fAK1 ; AK2 ; . . . ; AKK g, where AKi
 V. Ravi et al. / European Journal of Operational Research 126 (2000) 526533
is the ith fuzzy subset and the superscript K in-
dicates the number of fuzzy subsets on each axis.
Thus, K denotes the grid size of a fuzzy partition
and dierentiates the rules belonging to dierent
rule tables corresponding to 2; 3; . . . ; L partitions,
in the distributed representation of fuzzy rules. A
symmetric triangular membership function in the
unit interval 0; 1 is used for AKi ; i 1; . . . ; K
[3,4,9]. Fig. 1 indicates the distributed represen-
tation of the fuzzy ifthen rules for 2 features
when each feature is divided into a maximum of
Fig. 1. Labels and indices of fuzzy ifthen rules: (a) k 2 and
(b) k 3 [4].
529
3 partitions. For problems involving M classes
and 2 features, a fuzzy ifthen rule corresponding
to K 2 fuzzy subspaces has the following struc-
ture:
Rule RKij : If xp1 is AKi and xp2 is AKj then Xp
belongs to Class CijK ;
with CF CFijK ; i 1; 2; . . . ; K and
j 1; 2; . . . ; K;
where RKij is the label of the fuzzy ifthen rule, CijK
is the consequent (i.e. one of the M classes) and
CFijK is the grade of uncertainty of the fuzzy if
then rule, which is dened in [3,4,9]. CijK and CFijK
of the fuzzy ifthen rule in (1) are determined by
the procedures which can be found in detail in
[3,4]. These procedures (i) generate the fuzzy if
then rules and (ii) classify the new patterns. These
procedures are slightly modied by the authors in
[9]. Let SK be the set of generated K 2 fuzzy if
then rules given by S K fRKij j i 1; 2; . . . ; K;
j 1; 2; . . . ; Kg. Let the set of all fuzzy ifthen
rules corresponding to K 2; 3; . . . ; L partitions
be SALL which is:
SALL S 2 [ S 3 [


[ S L
fRKij j i 1; 2; . . . ; K; j 1; 2; . . . ; K
and K 2; 3; . . . ; Lg; 2
where L is an integer to be specied depending
on the classication problem. Let S be a subset
of SALL . The algorithm for the classier [9] for
computer simulations is coded as follows. Start-
ing from the rules corresponding to 2 partitions
rule table and going up to L partitions rule table,
all the rules no matter they are dummy or not,
are arranged in the form of a vector. Thus S is
nothing but a permutation of this vector and
SALL is set of all possible rules. The idea is to
determine a compact rule set S which has very
high classication power. To start with, S is
initialized randomly using a uniform random
number generator following a biased probability
method [9]. As the iterations progress, S gets
modied.
530 V. Ravi et al. / European Journal of Operational Research 126 (2000) 526533
3.1. The multi-objective combinatorial optimization
problem
As in [4,9], the main objective is to nd a
compact rule set S with very high classication
power by employing a combinatorial optimization
algorithm. The two objectives in the present
problem are: (i) maximize the number of correctly
classied patterns and (ii) minimize the number of
fuzzy ifthen rules. Accordingly, we have
Maximize NCPS and Minimize jSj
subject to S  SALL ;
where NCPS is the number of correctly classied
patterns by S and jSj is the number of fuzzy if
then rules in S. This is further reformulated as a
scalar optimization problem below.
Maximize f S WNCP
NCPS WS
jSj
subject to S  SALL :
Since the classicatory power of the system is
more important than its compactness [3,4,9], the
weights have been specied as 0 < WS  WNCP and
taken as WNCP 10:0 and WS 1:0 following [4].
For the details regarding the coding of the rules
used in the optimization module, the reader is re-
ferred to [9]. We employ a meta-heuristic viz,
modied threshold accepting algorithm [9] to solve
the problem just described. It should be kept in
mind, however, that the algorithm used is a heu-
ristic and that, depending on the initial feasible
solution the best solutions reported here may not
be ecient.
4. Numerical simulations and results
The rst illustrative example solved using the
methodology presented here is the well-known
wine classication problem for which the data are
freely available in the Internet [2]. It has 13 fea-
tures (attributes) which classify 178 patterns into
three types of wines. The second numerical ex-
ample concerns the determination of the breast
cancer in humans from Wisconsin University
hospital in Madison, USA. This is also freely
available in the Internet via anonymous ftp from
ics.uci.edu in directory /pub/machine-
learning-databases/wisconsin-breast-
cancer. This problem has 9 features or attributes
which determine whether a patient is benign or
malign. There are 683 samples or patterns. This
data has been used in the past by Mangasarian
et al. [7,8] and Bennet et al. [1]. A software in
A N S I . C has been developed by the authors on a
Pentium 100 MHz machine under Windows 95
platform using the MS-Visual C++ 5.0 compiler to
implement the model.
The methodology presented is tested in two
ways: (i) using the training data itself as the test
data (ii) using the leave-one-out technique in the
testing phase. The latter method is preferable as
there is the danger of over-tting in the former
method. In each of these methods, all the feature
spaces have been divided into a maximum of 5
partitions for both the examples. This is done in
3 order to keep the computational complexity to a
reasonable level, as we work with a new set of 5
features (the principal components) in both the
examples. Further, the study has been conducted
for 5 cases each corresponding to dierent aggre-
gator viz (1) product operator (2) min operator (3)
c-operator (compensatory and) [15] (4) fuzzy and
[14] and (5) a convex combination of min and max
operators [15].
Results of the wine classication problem (see
Table 1) indicate that the product operator per-
formed consistently well and gave the best solution
with 100% classication with just 11 rules whereas
the c-operator with c 0:1, came closely behind
giving 100% classication with 13 rules for cases of
both 4 partitions and 5 partitions of the features.
min operator scored over the others when one
looks at high classication power of 100% with 18
and 20 rules, respectively for 4 and 5 partitions.
Fuzzy and and a convex combination of min and
max operators did not provide good solutions
though the former performed better among the
two when 4 partitions were considered.
The same example when studied with leave-
one-out technique (see Table 2), produced dif-
ferent results. Both the product operator and
c-operator (with c 0:1) provided the best
 V. Ravi et al. / European Journal of Operational Research 126 (2000) 526533 531

Table 1
Results of example 1 (training data used as test data)a
# Partitions Operator
Product Minimum c-Operator Fuzzy and min/max
CP jSj CP jSj CP jSj CP jSj CP jSj
5 100 11 100 20 100 13 84.27 69 58.43 53
4 100 11 100 18 100 13 98.31 13 97.19 15
3 86.52 7 74.16 6 84.83 7 58.98 8 62.36 4
a
min/max indicates convex combination of min and max operators.

Table 2
Results of example 1 (leave-one-out technique)a
# Partitions Operator
Product Minimum c-Operator Fuzzy and min/max
CP jSj CP jSj CP jSj CP jSj CP jSj
5 100 3.04 91.01 2.74 100 3.04 82.02 62.56 91.57 87.9
4 100 3.07 93.26 2.87 100 3.07 99.44 3.38 98.88 3.36
3 60.11 1.8 60.11 1.8 60.11 1.8 60.11 1.9 60.11 1.8
a
min/max indicates convex combination of min and max operators.

Table 3
Results of example 2 (training data used as test data)a
# Partitions Operator
Product Minimum c-Operator Fuzzy and min/max
CP jSj CP jSj CP jSj CP jSj CP jSj
5 98.54 25 97.51 29 98.24 27 95.6 35 95.02 29
4 98.39 13 97.36 16 97.95 13 96.78 17 95.31 9
3 97.95 12 96.33 13 97.8 10 95.46 5 87.99 7
a
min/max indicates convex combination of min and max operators.

Table 4
Results of example 2 (leave-one-out technique)a
# Partitions Operator
Product Minimum c-Operator Fuzzy and min/max
CP jSj CP jSj CP jSj CP jSj CP jSj
5 100 9.3 99.85 7.3 100 7.3 96.34 22.23 98.68 22.82
4 92.39 2.78 92.68 2.79 92.38 2.78 95.75 2.87 100 3
3 98.09 3 82.28 2.47 97.66 2.93 98.83 2.96 71.88 2.15
a
min/max indicates convex combination of min and max operators.

solution with 100% classication with 3.04 rules max operators did not perform well, but they
on average, when 5 partitions were considered. outperformed the min operator when 4 partitions
Fuzzy and and a convex combination of min and were used.
532 V. Ravi et al. / European Journal of Operational Research 126 (2000) 526533
Results of the Wisconsin breast cancer problem
(see Table 3) shows that, when 5 partitions were
considered, the product operator yielded the best
result of 98.54% classication power with 25 rules;
the c-operator c 0:1 gave rise to the next best
solution, whereas the min operator yielded better
solution than the fuzzy and and a convex combi-
nation of min and max operators. The min oper-
ator could produce a maximum of 97.51%
classication power with 29 rules.
When the leave-one-out technique was applied
to the same problem (see Table 4), the best solu-
tion of 100% classication power with just 3 rules
on the average was achieved by the convex com-
bination of min and max operators when 4 parti-
tions were considered. The next best solution of
100% classication power with 7.3 rules on the
average was achieved by the c-operator c 0:1
when 5 partitions were considered. The product
operator followed closely behind with 100% clas-
sication power and 9.3 rules on the average with
the same number of partitions. When 5 partitions
were considered, the min operator yielded 99.85%
classication power with just 7.3 rules on the
average.
5. Conclusions
This paper addresses the paramount aspect of
dimensionality reduction of the feature space in
classication problems involving a large number
of dimensions and demonstrates the utility of
principal component analysis thereof. The chosen
principal components are fed to a fuzzy rule
based classier as new set of features. This
yielded a very high classication power of 100%
in leave-one-out technique with a few rules in the
case of some aggregators. This is particularly a
signicant result because the chosen principal
components accounted for only 89% and 92% of
the total variance in examples 1 and 2, respec-
tively. Hence, the authors conclude that the
principal component analysis can be used as a
useful alternative to other methods of feature
selection existing in literature while solving clas-
sication problems of higher dimensions using
fuzzy rule based classiers.
Acknowledgements
The rst author wishes to thank Deutscher
Akademischer Austauschdienst (DAAD) for
providing him the nancial support for this work
through a research fellowship leading to his
Ph.D. degree. The authors also express their
gratitude to Dr. W.H. Wolberg (physician),
University of Wisconsin Hospitals Madison,
Wisconsin, USA and Prof. Olvi Mangasarian
(mangasarian@cs.wisc.edu) who is the principal
donor of the breast cancer database to the in-
ternet.
References
[1] K.P. Bennett, O.L. Mangasarian, Robust linear program-
ming discrimination of two linearly inseparable sets,
Optimization Methods and Software 1 (1992) 2334.
[2] M. Forina et al., Wine Recognition Database, Available
via anonymous ftp from ics.uci.edu in directory /
pub/machine-learning-databases/wine, 1992.
[3] H. Ishibuchi, K. Nozaki, H. Tanaka, Distributed
representation of fuzzy rules and its application to
pattern classication, Fuzzy Sets and Systems 52 (1992)
2132.
[4] H. Ishibuchi, K. Nozaki, N. Yamamoto, H. Tanaka,
Selecting fuzzy ifthen rules for classication problems
using genetic algorithms, IEEE Transactions on Fuzzy
Systems 3 (1995) 260270.
[5] H. Ishibuchi, T. Murata, Minimizing the fuzzy rule base
and maximizing its performance by a multi-objective
genetic algorithm, in: Sixth FUZZIEEE Conference,
Barcelona, Spain, 1997, pp. 259264.
[6] B. Kosko, Neural Networks and Fuzzy Systems, Prentice-
Hall, Englewood Clis, NJ, 1992.
[7] O.L. Mangasarian, W.H. Wolberg, Cancer diagnosis via
linear programming, SIAM News 23 (1990) 118.
[8] O.L. Mangasarian, R. Setiono, W.H. Wolberg, Pattern
recognition via linear programming: Theory and applica-
tion to medical diagnosis, in: T.F. Coleman, Y. Li (Eds.),
Large-Scale Numerical Optimization, SIAM, Philadelphia,
1990, pp. 2230.
[9] V. Ravi, H.-J. Zimmermann, Fuzzy rule based classica-
tion with FeatureSelector and modied threshold accept-
ing, European Journal of Operational Research 123 (1)
(2000) 1628.
[10] J.O. Rawlings, Applied Regression Analysis:A Research
Tool, Wadsworth and Brooks/Cole Statistics and Proba-
bility series, Wadsworth, Belmont, CA, 1988.
[11] J. Strackeljan, D. Behr, F. Detro, FeatureSelector: A
plug-in for feature selection with DataEngine, in: First
 V. Ravi et al. / European Journal of Operational Research 126 (2000) 526533 533

International Data Analysis Symposium, Aachen, Ger- [14] H.-J. Zimmermann, P. Zysno, Latent connectives in
many, 1997. human decision making, Fuzzy Sets and Systems 4 (1980)
[12] WINROSA, Manual, MIT GmbH, Aachen, Germany, 3751.
1997. [15] H.-J. Zimmermann, Fuzzy Set Theory and Its Applica-
[13] L. Zadeh, Fuzzy sets, Information and Control 8 (1965) tions, 2nd ed., Kluwer Academic Publishers, Dordrecht,
338353. 1991.