Journal of Information & Computational Science 7: 3 (2010) 777783 Available at http://www.joics.

com

Possibilistic Fuzzy Learning Vector Quantization

Xiaohong Wu a,,
a School b School

Haijun Fu b , Bin Wu c , Jiewen Zhao a

of Food and Biological Engineering, Jiangsu University, Zhenjiang 212013, China of Information Engineering, ChuZhou Vocational Technology College ChuZhou 239000, China

of Electrical and Information Engineering, Jiangsu University, Zhenjiang 212013, China

c Department

Abstract Fuzzy learning vector quantization (FLVQ) is a well-known fuzzy clustering network. However, FLVQ is sensitive to noises or outliers. In this paper, two novel fuzzy learning vector quantization algorithms, called possibilistic fuzzy learning vector quantization (PFLVQ), are proposed to overcome the noise sensitivity problem of FLVQ. PFLVQ integrates possibilistic fuzzy c-means (PFCM) clustering model into fuzzy learning vector quantization. Dierent from FLVQ that only oers membership, PFLVQ can produce both membership and typicality values simultaneously. PFLVQ can deal with noisy data better than FLVQ and does not generate coincident clusters that occur in possiblistic c-means (PCM). The experimental results show the better performance of PFLVQ. Keywords: Fuzzy Clustering, Fuzzy Learning Vector Quantization, Possibilistic Fuzzy C-means

Introduction

Fuzzy clustering is one of the most important methods of unsupervised learning, and it always has signicant advantages over traditional clustering. The fuzzy clustering based on fuzzy set theory [1] is used to deal with ill-dened boundaries between clusters. The most widely used fuzzy clustering is the fuzzy c-means (FCM) algorithm [2][3] which is generalized from hard cmeans (HCM) algorithm. Similarly, learning vector quantization (LVQ) [4] has been generalized to fuzzy learning vector quantization (FLVQ) [5]. Furthermore, FLVQ overcomes several major problems coming from Kohonen clustering networks (KCNs) and integrates FCM model into the learning rate and updating strategies of the Kohonen network. FCM is based on least-squared error clustering criterion and assigns the memberships of a data point across classes sum to one by the probabilistic constraint. This constraint avoids the trivial solution of all memberships being equal to 0, and it is appropriate to interpret memberships as degrees of sharing [2]. However, the memberships do not always correspond to the intuitive concept of degree of belonging or compatibility. Furthermore, the FCM is sensitive to noises or outliers [6]. Since FLVQ is based

Corresponding author. Email address: wxh419@ujs.edu.cn (Xiaohong Wu).

c 15487741/ Copyright 2010 Binary Information Press March 2010

778

X. Wu et al. /Journal of Information & Computational Science 7: 3 (2010) 777783

on FCM, FLVQ is also sensitive to noises or outliers. To overcome noise sensitivity shortcoming of FCM Krishnapuram and Keller have presented the possibilistic c-means (PCM) algorithm [6] by abandoning the constraint of FCM and constructing a novel objective function. PCM can deal with noisy data better than FCM. But PCM is very sensitive to initializations and sometimes generates coincident clusters [7]. PCM considers the possibility (typicality) but neglects the important membership. To combine the benets of FCM and PCM, possibilistic fuzzy c-means model (PFCM) [8][9] was proposed to produce memberships and possibilities simultaneously. PFCM overcomes the noise sensitivity shortcoming of FCM and the coincident clusters problem of PCM. In this paper we integrate possibilistic fuzzy c-means (PFCM) clustering model into fuzzy learning vector quantization and propose a novel fuzzy learning vector quantization called possibilistic fuzzy learning vector quantization (PFLVQ). PFLVQ can produce memberships and possibilities simultaneously and it overcomes the noise sensitivity shortcoming of FLVQ. Our experimental results show the better performance of PFLVQ.

Fuzzy Learning Vector Quantization

Fuzzy learning vector quantization (FLVQ) is self-organization, non-sequential and competitive clustering network. Its learning rate is dened as: , = (, ) ; = (0 1)/ (1)

where , is the membership coming from FCM: [ ( ]1 2 ) 1 = , , =1

(2)

and = = 0 , 0 (0 > 1) is a positive constant. is the iteration limit. = , c is the number of clusters in equation (2). The weight vectors for FLVQ is calculated as: , ( ,1 ) , = ,1 + =1 (3) =1 , Given an unlabeled data set = {1 , 2 , ..., } , nd the partition of into 1 < < fuzzy subsets. FLVQ algorithm is described as follows: Step 1 Initialization (1) Fix , 0 > 1 and > 0 some small positive constant; Set iteration counter = 0 and maximum iteration ; (2) Initialize 0 = (1,0 , 2,0 , ..., ,0 ) Step 2 Repeat (1) Compute all (cn) learning rates {, } with equation (1) and (2). (2) Update all (c) weight vectors {, } with equation (3). (3) Increment t; Step 3 Until ( 1 < ) or ( > ).

X. Wu et al. /Journal of Information & Computational Science 7: 3 (2010) 777783

779

Possibilistic Fuzzy C-means Clustering Model

Given an unlabeled data set = {1 , 2 , ..., } , nd the partition of into 1 < < fuzzy subsets by minimizing the following objective function: , (, , ) =
=1 =1 2 ( + ) + =1

=1

(1 )

(4)

Subject to the constraints: =1 = 1, ,and 0 , 1, = . Here > 0, > 0. The constants and dene the relative importance of fuzzy membership and typicality values in the objective function.And is the number of clusters, n is the number of data points, is the membership of in class , is the typicality values which depends on all data, weighting exponent , > 1 Krishnapuram and Keller [6] suggest choosing by computing 2 =1 = , > 0 (5) =1 Here is always chosen to be 1. If = > 0 for all and , , > 0 and contains at least c distinct data points, (,, ) , (, , ) is optimized under constraints and the possibilistic fuzzy c-means model is obtained as follows [ ( ]1 [ 2 ( ) 1 ]1 ) 1 2 1 = , , ; = 1 + , , (6) =1 =1 ( + ) = , =1 ( + ) (7)

Here is the cluster center or prototype of . Some interesting properties of PFCM can be found in [8].

Possibilistic Fuzzy Learning Vector Quantization

Inspired by FLVQ algorithm that integrates FCM and LVQ, we integrate possibilistic fuzzy cmeans (PFCM) clustering algorithms into fuzzy learning vector quantization and obtain the possibilistic fuzzy learning vector quantization (PFLVQ) algorithms in this section. Firstly, we dene the two learning rates for PFLVQ (it integrates FLVQ and PFCM) as , = (, ) ; = 0 ; = 0 ; , = (, ) = (0 1)/ = (0 1)/ (8) (9) (10)

Here, , and , are calculated with equation (6). = = 0 ,and = = 0 and 0 (0 > 1) and 0 (0 > 1) are both positive constants. is the iteration limit. , is referred to as fuzzy learning rate and , is typicality learning rate. The weight vectors for PFLVQ is dened as (, + , )( ,1 ) (11) , = ,1 + =1 =1 (, + , )

780

X. Wu et al. /Journal of Information & Computational Science 7: 3 (2010) 777783

Given an unlabeled data set = {1 , 2 , ..., } ,nd the partition of X into 1 < < fuzzy subsets. PFLVQ algorithm is described as follows: Step 1 Initialization (1) Fix , 0 > 1 and > 0 some small positive constant; Set iteration counter = 0 and maximum iteration ; Fix a and b, > 0 and > 0. (2) Run FCM until termination and use equation (5) to get . (3) Initialize 0 = (1,0 , 2,0 , ..., ,0 ) Step 2 Repeat (1) Compute all (cn) learning rates {, } and {, } with equation (8), (9),(10) and (6). (2) Update all (c) weight vectors {, } with equation (11). (3) Increment j; Step 3 Until ( 1 < ) or ( > ).

Experimental Results

In this section, FLVQ and PFLVQ are made experiments on 12 [8][9][10] and IRIS [11] data sets to compare their performances. One example is that we make numerical experiments on 12 which is a two-dimensional data set with 12 data points. The coordinates of 12 are given in Table 1. There are ten points (except 6 and 12 ) form two diamond shaped clusters with ve points each on the left and right sides of the y axis. We can see 6 and 12 as noisy points and each has the same distance from the two clusters. The initialization of cluster centers are the same as the reference paper [8][10]: Computational condition: FLVQ: = 0.01, = 50, 0 = 2.0; PFLVQ: = 0.01, = 50, 0 = 2.0, 0 = 2.0, = 1.0 and = 1.0. Table 1 shows the terminal membership values of FLVQ by running FLVQ algorithm on 12 .The membership values of 6 and 12 assigned by FLVQ are 0.50. Because the probabilistic constraint used by FLVQ, the membership values of 6 and 12 reect the share of this two data points between two clusters by FLVQ algorithm. So their values are 0.50. But in fact 6 and 12 are noises, and their membership values should be very small. Furthermore, because 12 is farther from cluster center (3 or 9 ) than 6 .The membership of 12 should be smaller than that of 6 However, FLVQ cannot reect these facts so it is sensitive to noises. Table 2 shows the terminal membership values and typicality values of PFLVQ. Because 12 is farther away from cluster centers than 6 ,the PFLVQ assigns the smaller typicality values of 12 than that of 6 .This reects the real situations. That is to say, 6 and 12 are more atypical than other 10 data points. So PFLVQ can distinguish the noises from datasets.

X. Wu et al. /Journal of Information & Computational Science 7: 3 (2010) 777783

781

Table 1: The Coordinates Pt. x 1 -5.00 2 -3.34 3 -3.34 4 -3.34 5 -1.67 6 0.00 7 1.67 8 3.34 9 3.34 10 3.34 11 5.00 12 0.00

of 12 and Terminal U from FLVQ

y 0.00 1.67 0.00 -1.67 0.00 0.00 0.00 1.67 0.00 -1.67 0.00 10.00

1 0.96 0.98 1.00 0.93 0.95 0.50 0.06 0.02 0.00 0.07 0.04 0.50

2 0.04 0.02 0.00 0.07 0.05 0.50 0.94 0.98 1.00 0.93 0.96 0.50

Table 2: Terminal U Pt. 1 1 0.97 2 0.98 3 1.00 4 0.95 5 0.96 6 0.50 7 0.04 8 0.02 9 0.00 10 0.05 11 0.03 12 0.50

and T from PFLVQ for 12

2 0.03 0.02 0.00 0.05 0.04 0.50 0.96 0.98 1.00 0.95 0.97 0.50 1 0.66 0.86 0.98 0.68 0.89 0.48 0.22 0.11 0.11 0.10 0.07 0.04 2 0.07 0.11 0.12 0.10 0.22 0.48 0.89 0.86 0.98 0.68 0.66 0.04

Table 4 shows the terminal centroid values of FLVQ and PFLVQ. The true centroid [8] [ ] 3.34 3.34 = 0.00 0.00 Let and be the centroids in Table 3 from FLVQ and PFLVQ for 12 .To see how near these centroids from the true centroid ,we calculate [10] = 12 2 Here, denotes FLVQ/PFLVQ. The results are: = 0.4505 and = 0.3485. So we nd that < The other example is that we perform experiments by running FLVQ and PFLVQ, respectively on the IRIS data set which is widely used in experiments. It is fourdimensional data sets which include three class: Setosa, Versilcolor and Viginica and each class

782

X. Wu et al. /Journal of Information & Computational Science 7: 3 (2010) 777783

has 50 data points. The computational condition is = 0.01,maximum number of iterations = 50, = 1.0 and = 1.0, and the initialization of cluster centers are the same as the reference paper [10]. We change the values of and , then the numbers of re-substitution errors from FLVQ and PFLVQ on IRIS data set are illustrated with Table 4. From Table 4 we know PFLVQ algorithm has better clustering accuracy than FLVQ. Table 3: Terminal V from FLVQ and PFLVQ for 12 FLVQ PFLVQ -2.97 2.97 -2.88 2.88 0.56 0.56 0.37 0.37

= 1.0

Table 4: Numbers of resubstitution errors from FLVQ and PFLVQ on IRIS Data Set with = 1.0 and m 1.5 1.5 1.5 2.0 2.0 2.0 3.0 3.0 FLVQ PFLVQ U U T 1.5 17 16 15 2.0 17 15 15 3.0 17 15 16 1.5 16 14 14 2.0 16 15 14 3.0 16 14 14 1.5 15 16 15 2.0 15 15 15

Table 5 shows the experimental results by running FLVQ and PFLVQ on IRIS data with computational condition: = 0.01, maximum number of iterations = 50, = 2.0, = 2.0 and dierent values of and .PFLVQ shows better clustering accuracy than FLVQ in Table 5. Table 5: Numbers of re-substitution errors from FLVQ and PFLVQ on IRIS Data Set with = 2.0, =
2.0

1 1 1 1 1 2 2 2 3 3 3 3

FLVQ PFLVQ U U T 1 16 15 14 2 16 15 14 3 16 15 14 4 16 15 14 5 16 15 14 1 16 15 16 2 16 15 15 3 16 15 16 1 16 15 15 2 16 15 15 3 16 15 15 4 16 15 15

X. Wu et al. /Journal of Information & Computational Science 7: 3 (2010) 777783

783

Conclusions

In this paper, we propose two novel fuzzy learning vector quantization models called fuzzy learning vector quantization (PFLVQ) models for integrating FLVQ model and PFCM models. PFLVQ combines the advantages of FLVQ and PFCM, and overcomes noise sensitivity problem of FLVQ. We make experiments on data set X12 to nd that PFLVQ can get better centroid than FLVQ, and PFLVQ is more insensitive to noises or outliers than FLVQ. The other example is that we make experiments on IRIS data set by running FLVQ and PFLVQ, respectively. The experimental results show that PFLVQ has better clustering accuracy than FLVQ.

Acknowledgement
The authors would like to thank China Postdoctoral Science Foundation funded project (No. 20090460078) for nancially supporting this research.

References
[1] [2] [3] [4] [5] [6] [7] [8] [9] Zadeh L A. Fuzzy sets. Inf. Control, 1965, 8: 338-353 Bezdek J C. Fuzzy Mathematics in pattern classication. Ph.D. dissertation, Dept. Appl. Math., Cornell Univ., Ithaca, NY, 1973 Bezdek J C. Pattern Recognition with Fuzzy Objective Function Algorithms. Plenum Press, New York, 1981 Kohonen T. Self-Organization and Associative memory. 3 Edition, Spring-verlag, Berlin, 1989 Tsao E C, Bezdek J C, Pal N R. Fuzzy Kohonen clustering networks. Pattern Recognition, 1994, 27(5): 757-764 Krishnapuram R, Keller J. A possibilistic approach to clustering. IEEE Trans. Fuzzy Systems, 1993, 1(2): 98-110 Barni M, Cappellini V, Mecocci A. comments on A possibilistic approach to clustering. IEEE Trans. Fuzzy Systems, 1996, 4(3): 393-396 Pal N R, Pal K, Bezdek J C. A new hybrid c-means clustering model. Proceedings of the IEEE International Conference On Fuzzy Systems, 2004, pp. 179-184 Pal N R, Pal K, Bezdek J C. A possibilistic fuzzy c-means clustering algorithm. IEEE Trans. Fuzzy Systems, 2005, 13(4): 517-530

[10] Pal N R, Pal K, Bezdek J C. A mixed c-means clustering model. Processings of the IEEE Trans. Fuzzy Systems, Spain, 1997, pp. 11-21 [11] Bezdek J C, Keller J M, Krishnapuram R, et al. Will the real IRIS data please stand up?. IEEE Trans. on Fuzzy System, 1999, 7 (3): 368 - 369