See

discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/45891314

Performance Analysis of AIM-K-means & Kmeans in Quality Cluster Generation

Article December 2009
Source: arXiv

CITATIONS

READS

14

28

2 authors:
Samarjeet Borah

Mrinal Kanti Ghose

Sikkim Manipal Institute of Technology

( Formerly, Sr. Scientist, Vikram Sarabhai Sp

19 PUBLICATIONS 23 CITATIONS

246 PUBLICATIONS 708 CITATIONS

SEE PROFILE

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Development of Morph Analyser for Nepali Language View project

IJSE Special Issue on Data Aanalytics View project

All in-text references underlined in blue are linked to publications on ResearchGate,

letting you access and read them immediately.

Available from: Mrinal Kanti Ghose

Retrieved on: 14 October 2016

JOURNAL OF COMPUTING, VOLUME 1, ISSUE 1, DECEMBER 2009, ISSN: 2151-9617

HTTPS://SITES.GOOGLE.COM/SITE/JOURNALOFCOMPUTING/

175

Performance Analysis of AIM-K-means &

K-means in Quality Cluster Generation
Samarjeet Borah, Mrinal Kanti Ghose
Abstract: Among all the partition based clustering algorithms K-means is the most popular and well known method. It generally
shows impressive results even in considerably large data sets. The computational complexity of K-means does not suffer from
the size of the data set. The main disadvantage faced in performing this clustering is that the selection of initial means. If the
user does not have adequate knowledge about the data set, it may lead to erroneous results. The algorithm Automatic
Initialization of Means (AIM), which is an extension to K-means, has been proposed to overcome the problem of initial mean
generation. In this paper an attempt has been made to compare the performance of the algorithms through implementation
Index Terms Cluster, Distance Measure, K-means, Centroid, Average Distance, Mean

1 INTRODUCTION
Clustering [2][3][4] is a type of unsupervised learning
method in which a set of elements is separated into ho
mogeneous groups. It seeks to discover groups, or clus
ters, of similar objects. Generally, patterns within a valid
cluster are more similar to each other than they are to a
pattern belonging to a different cluster. The similarity
betweenobjectsisoftendeterminedusingdistancemeas
uresoverthevariousdimensionsinthedataset.Thevari
ety of techniques for representing data, measuring simi
laritybetweendataelements,andgroupingdataelements
has produced a rich and often confusing assortment of
clustering methods. Clustering is useful in several ex
ploratory patternanalysis, grouping, decisionmaking,
and machinelearning situations, including data mining,
document retrieval, image segmentation, and pattern
classification[5][3].

jorityofthemcouldbeconsideredasgreedyalgorithms,
i.e., algorithms that at each step choose the best solution
and may not lead to optimal results in the end. The best
solutionateachstepistheplacementofacertainobjectin
theclusterforwhichtherepresentativepointisnearestto
the object. This family of clustering algorithms includes
thefirstonesthatappearedintheDataMiningCommu
nity. The most commonly used are Kmeans [JD88,
KR90][6], PAM (Partitioning Around Medoids) [KR90],
CLARA (Clustering LARge Applications) [KR90] and
CLARANS(ClusteringLARgeApplicatioNS)[NH94].All
ofthem are applicable to data sets with numerical attrib
utes.

2.1 K-means Algorithm

Kmeans[7]isaprototypebased,simplepartitionalclus
teringtechniquewhichattemptstofindauserspecifiedK
number of clusters. These clusters are represented by
2 PARTITION BASED CLUSTERING METHODS
theircentroids.Aclustercentroidistypicallythemeanof
Partition based clustering methods create the clusters in thepointsinthecluster.Thisalgorithmisasimpleitera
onestep.Onlyonesetofclustersiscreated,althoughsev tive clustering algorithm. The algorithm is simple to im
eral different sets of clusters may be created internally plementandrun,relativelyfast,easytoadapt,andcom
withinthevariousalgorithms.Sinceonlyonesetofclus mon in practice. It is historically one of the most impor
tersisoutput,theusersmustinputthedesirednumberof tant algorithms in data mining. The general algorithm
clusters. Given a database of n objects, a partition based was introduced by Cox (1957), and (Ball and Hall, 1967;
[5]clusteringalgorithmconstructskpartitionsoftheda MacQueen,1967)[6]firstnameditKmeans.Sincethenit
ta, so that an objective function is optimized. In these has become widely popular and is classified as a parti
clustering methods some metric or criterion function is tional or nonhierarchical clustering method (Jain and
usedtodeterminethegoodnessofanyproposedsolution. Dubes,1988).Ithasanumberofvariations[8][11].
This measure of quality could be average distance be TheKmeansalgorithmworksasfollows:
tweenclustersorsomeothermetric.Onecommonmeas
a. Select initial centres of the K clusters. Repeat
ureofsuchkindisthesquirederrormetric,whichmeas
steps b through c until the cluster membership
uresthesquireddistancefromeachpointtothecentroid
stabilizes.
fortheassociatedcluster.Partitionbasedclusteringalgo
b. Generateanewpartitionbyassigningeachdata
rithmstrytolocallyimproveacertaincriterion.Thema
toitsclosestclustercentres.
Samarjeet Borah is with the Department of Computer Science & Engineerc.
Compute new cluster centres as the centroids of
ing, Sikkim Manipal Institute of Technology, Majitar, Rangpo, East Siktheclusters.
kim-737132.
Mrinal Kanti Ghose is with the Department of Computer Science & Engi-
neering as Professor & HOD, Sikkim Manipal Institute of Technology,
Thealgorithmcanbebrieflydescribedasfollows:
Majitar, Rangpo, East Sikkim-737132.
Let us consider a dataset D having n data points x1, x2

JOURNAL OF COMPUTING, VOLUME 1, ISSUE 1, DECEMBER 2009, ISSN: 2151-9617

HTTPS://SITES.GOOGLE.COM/SITE/JOURNALOFCOMPUTING/

xn. The problem is to find minimum variance clusters

from the dataset. The objects have to be grouped into k
clustersfindingkpoints{mj}(j=1,2,,k)inDsuchthat

(1)

is minimized, where d(xi, mj) denotes the Euclidean dis

tancebetweenxiandmj.Thepoints{mj}(j=1,2,,k)are
known as cluster centroids. The problem in Eq.(1) is to
find k cluster centroids, such that the average squared
Euclidean distance (mean squared error) between a data
pointanditsnearestclustercentroidisminimized.

The Kmeans algorithm provides an easy method to im

plement approximate solution to Eq.(1). The reasons for
the popularity of Kmeans are ease and simplicity of im
plementation,scalability,speedofconvergenceandadap
tability to sparse data. The Kmeans algorithm can be
thoughtofasagradientdescentprocedure,whichbegins
atstartingclustercentroids,anditerativelyupdatesthese
centroidstodecreasetheobjectivefunctioninEq.(1).The
Kmeans always converge to a local minimum. The par
ticular local minimum found depends on the starting
cluster centroids. The Kmeans algorithm updates cluster
centroidstilllocalminimumisfound.BeforetheKmeans
algorithm converges, distance and centroid calculations
aredonewhileloopsareexecutedanumberoftimes,say
l,wherethepositiveintegerlisknownasthenumberof
Kmeans iterations. The precise value of l varies depend
ing on the initial starting cluster centroids even on the
same dataset. So the computational time complexity of
the algorithm is O(nkl), where n is the total number of
objectsinthedataset,kistherequirednumberofclusters
weidentifiedandlisthenumberofiterations,kn,ln.
Kmean clustering algorithm is also facing a number of
drawbacks. When the numbers of data are not so many,
initial grouping will determine the cluster significantly.
Again the number of cluster, K, must be determined be
forehand.Itissensitivetotheinitialcondition.Different
initialconditionsmayproducedifferentresultsofcluster.
The algorithm may be trapped in the local optimum.
Weakness of arithmetic mean is not robust to outliers.
Very far data from the centroid may pull the centroid
awayfromtherealone.Heretheresultisofcircularclus
tershapedbecausebasedondistance.

The major problem faced during Kmeans clustering is

theefficientselectionofmeans.Itisquitedifficulttopre
dict the number of clusters k in prior. The k varies from
user to user.As a result, the clusters formed may not be
uptomark.Thefindingoutofexactlyhowmanyclusters
willhavetobeformedisaquitedifficulttask.Toperform
itefficientlytheusermusthavedetailedknowledgeofthe
domain.Againthedetailknowledgeofthesourcedatais
alsorequired.

2.2 Automatic Initialization of Means

The Automatic Initialization of Means (AIM) [12] has
beenproposedtomaketheKmeansalgorithmabitmore
efficient. The algorithm is able to detect the number of

176

total number of clusters automatically. This algorithm

also has made the selection process of the initial set of
meansautomatic.AIMappliesasimplestatisticalprocess
whichselectsthesetofinitialmeansautomaticallybased
on the dataset. The output of this algorithm can be ap
pliedtotheKmeansalgorithmasoneoftheinputs.

2.2.1 Background
Inprobabilitytheoryandstatistics,theGaussiandistribu
tionisacontinuousprobabilitydistributionthatdescribes
data that clusters around a mean or average. Assuming
Gaussiandistributionitisknownthat1contain67.5%
ofthepopulationandthussignificantvaluesconcentrate
aroundtheclustermean.Pointsbeyondthismayhave
tendency of belonging to other clusters. We could have
taken2insteadof1,butproblemwith2isthat
itwillcoverabout95%ofthepopulationandasaresultit
mayleadtoimproperclustering.Somepointsthatarenot
sorelevanttotheclustermayalsobeincludedintheclus
ter.

2.2.2 Description
LetusassumethatdatasetDas{xi,i=1,2N}whichcon
sistsofNdataobjectsx1,x2,,xN.,whereeachobjecthas
M different attribute values corresponding to the M dif
ferentattributes.Thevalueofithobjectcanbegivenby:
Di={xi1,xi2,,xiM}
Againletusassumethattherelationxi=xkdoesnotmean
thatxiandxkarethesameobjectsintherealworlddata
base. It means that the two objects has equal values for
theattributesetA={al,a2,a3,,am}.Themainobjectiveof
the algorithm is to find out the value k automatically in
prior to partition the dataset into k disjoint subsets. For
distance calculation the distance measure sum of square
Euclidian distance is used in this algorithm. It aims at
minimizing the average square error criterion which is a
goodmeasureofthewithinclustervariationacrossallthe
partitions.Thustheaveragesquareerrorcriteriontriesto
makethekclustersascompactandseparatedaspossible.

Let us assume a set of means M={mj, j=1, 2, , K} which

consistsofinitialsetofmeansthathasbeengeneratedby
thealgorithmbasedonthedataset.Basedontheseinitial
meansthedatasetwillbegroupedintoKclusters.Letus
assumethesetofclustersasC=(cj,j=1,2,,M}.Inthenext
phasethemeanshastobeupdated.
Inthealgorithmthedistancethresholdhasbeentaken
as:
dx=1

(2)
where=
and=

Before the searching of initial means the original dataset

D will be copied to a temporary dataset T. This dataset
willbeusedonlyininitialsetofmeansgenerationproc
ess.Thealgorithmwillberepeatedforntimes(wherenis
the number of objects in the dataset). The algorithm will

JOURNAL OF COMPUTING, VOLUME 1, ISSUE 1, DECEMBER 2009, ISSN: 2151-9617

HTTPS://SITES.GOOGLE.COM/SITE/JOURNALOFCOMPUTING/

selectthefirstmeanoftheinitialmeansetrandomlyfrom
the dataset. Then the object selected as mean will be re
moved from the temporary dataset. The procedure Dis
tance_Threshold will compute the distance threshold as
givenineq.2.
Wheneveranewobjectisconsideredasthecandidatefor
a cluster mean, its average distance with existing means
willbecalculatedasgivenintheequationbelow.
Average_Distance

(3)

whereMisthesetofinitialmeans,i=1,2,,mandmn,
mcisthecandidatefornewclustermean.

Ifitsatisfiesthedistancethresholdthenitwillconsidered
as new mean and will be removed from the temporary
dataset.Thealgorithmisasfollows:

Input:
//setofobjects
D={x1,x2,,xn}
Output:

K//Totalnumberofclusterstobegenerated

M={m1,m2,,mk}//Thesetofinitialmeans
Algorithm:

CopyDtoatemporarydatasetT

CalculateDistance_ThresholdonT

Arbitrarilyselectxiasm1

Insertm1toM

RemovexifromT
Fori=1tonkdo//Checkforthenextmean
Begin:

Arbitrarilyselectxiasmc

SetL=0

Forj=1tokdo//Calculatetheavg.dist

Begin:

L=L+Distance(mc,M[j])

End

Average_Distance=L/k

IfAverage_Distance
Distance_Thresholdthen:

RemovexifromT

InsertmctoM

k=k+1
End

3 PERFORMANCE ANALYSIS
TheAIM is just the extension of Kmeans to provide the
numberofclusterstobegeneratedbytheKmeansalgo
rithm.ItalsoprovidestheinitialsetofmeanstoKmeans.
Therefore it has been decided to make a comperative
analysis of the clustering quality of AIMKmeans with
convensional Kmeans. The main difference between the
two algorithms is that in case of AIMKmeans it is not
necessary to provide the number of clusters to be gener
atedinpriorandforKmeans,usershavetoprovidethe
numberofclusterstobegenerated.
Inthisevaluationprocessthreedatasetshave beenused.
They have been fed to the algorithms according to the
increasing order of their size. The programs were devel

177

oped in C. To test the algorithms thoroughly, separate

programs were developed for AIM, AIMKmeans and
conventionalKmeans.Inthefirstphasethedatasetshave
beenfedtotheKmeanswiththeuserfedkvalue.Then
the AIMKmeans was applied to the same data sets
where the value of k means is provided internally by
AIM. Lastly the Kmeans algorithm is again applied to
the same datasets with the value of k as given by the
AIMKmeansmethod.Theresultsreavelsthat:
1. ThereisadifferenceinperformanceforKmeans
andAIMKmeansalgorithm.
2. But the difference reduces when we use K-means
algorithm with the value of k as given by the
AIM-K-means.

Figure 1: Comparision Based on Average SSE

Theabovecomparisonwasmadeonthebasisofaverage
sum of square error. From the study it has been found
that AIMKmeans is showing improvements in average
sumofsquare.Thisisbasicallybecauseoftheinitialsetof
cluster means provided to the algorithm. In case of K
meansthevalueofkhasbeenprovidedbasedontheout
put provided by AIM. But it is not possible to provide
initialsetofclustersinKmeans.

4 CONCLUSION
ThemostattractivepropertyoftheKmeansalgorithmin
data mining is its efficiency in clustering large data sets.
But the main disadvantage it is facing is the number of
clusters that is to be provided from the user. The algo
rithm AIM, which is an extension of Kmeans, can be
usedtoenhancetheefficiencyautomatingtheselectionof
theinitialmeans.Fromtheexperimentsithasbeenfound
that it can improve the cluster generation process of the
Kmeans algorithm, without diminishing the clustering
quality in most of the cases. The basic idea ofAIM is to
keep the simplicity and scalability of Kmeans, while
achievingautomaticity.

ACKNOWLEDGMENT
This work has been carried out as part of Research Pro
motionScheme(RPS)ProjectfundedbyAllIndiaCouncil
forTechnicalEducation,GovernmentofIndia;videsanc
tionorder8023/BOR/RID/RPS217/200708.

JOURNAL OF COMPUTING, VOLUME 1, ISSUE 1, DECEMBER 2009, ISSN: 2151-9617

HTTPS://SITES.GOOGLE.COM/SITE/JOURNALOFCOMPUTING/

REFERENCES
[1]

Efficient Classification Method for Large Dataset,

ShengYi Jiang School of Informatics, Guangdong
Univ.ofForeignStudies,Guangzhou
[2] CLUSTERING TECHNIQUESFORLARGES DATASETSFROM
THE PAST TO THE FUTURE, ALEXANDER HINNEBURG,
DANIELA.KEIM
[3] DataClustering:AReview,A.K.Jain(MichiganState
University),M.N.Murty(IndianInstituteofScience)
andP.J.Flynn(TheOhioStateUniversity)
[4] DataClustering,byLourdesPerez
[5] Data Clustering and Its Applications, Raza Ali, Us
manGhani,AasimSaeed.
[6] MacQueen, J. Some methods for classification and
analysisofmultivariateobservations.Proc:5thBerke
leySymp.Math.Statist,Prob,1:218297,1967.
[7] KmeansClustering:Anovelprobabilisticmodeling
withapplicationsDipakK.Dey(jointworkwithSa
miran Ghosh, IUPUI, Indianapolis), Department of
Statistics,UniversityofConnecticut
[8] Variations of Kmeans algorithm: A tudy for High
Dimentional Large Data Sets: Sanjay Garg, Ramesh
Ch.Jain,Dept.ofComputerEngineering,A.D.Patel
InstituteofTechnlogy,India.
[9] J.HanandM.Kamber,DataMining:Conceptsand
Techniques.MorganKaufmann,2000.
[10] MichelR.Anderberg.ClusterAnalysisforApplica
tions:AcademicPress,1973.
[11] Extensions to the KmeansAlgorithm for Clustering
Large Data Sets with Categorical Values ZHEXUE
HUANG ACSys CRC, CSIRO Mathematical and In
formation Sciences, GPO Box 664, Canberra, ACT
2601,Australia.
[12] Automatic Initialization of Means (AIM): A Pro
posed Extension to the Kmeans Algorithm by Sa
marjeet Borah, M. K. Ghose, accepted in Interna
tional Journal of Information Technology and
KnowledgeManagement(IJITKM).
Samarjeet Borah has obtained his M. Tech. degree in Information
Technology from Tezpur University, India in the year 2006. His major
field of study is data mining. He is a faculty member in the Department of Computer Science & Engineering in Sikkim Manipal Institute
of Technology, Sikkim, India. He is the principal investigator in a
research project sponsored by Government of India. Till date he has
published ten papers in various conferences and journals. Borah is a
member of the Computer Society of India, International Associations
of Engineers, Hong Kong and International Association of Computer
Science and Information Technology, Singapore. He has received an
award on excellency in research initiatives from Sikkim Manipal University of Health Medical & Technological Sciences.
Dr. Mrinal Kanti Ghose has obtained his Ph.D. from Dibrugarh University, Assam, India in 1981. He is currently working as the Professor and Head of the Department of Computer Science & Engineering
at Sikkim Manipal Institute of Technology, Mazitar, Sikkim, India.
Prior to this, Dr. Ghose worked in the internationally reputed R & D
organisation ISRO during 1981 to 1994 at Vikram Sarabhai Space
Centre, ISRO, Trivandrum in the areas of Mission simulation and
Quality & Reliability Analysis of ISRO Launch vehicles and Satellite
systems and during 1995 to 2006 at Regional Remote Sensing Service Centre, ISRO, IIT Campus, Kharagpur(WB), India in the areas
of RS & GIS techniques for the natural resources management. Dr.
Ghose has conducted quite a number of Seminars, Workshop and
Training programmes in the above areas and published around 35

178

technical papers in various national and international journals in

addition to presentation/ publication of 125 research papers in international/ national conferences. He has guided many M. Tech and
Ph.D projects and extended consultancy services to many reputed
institutes of the country. Dr. Ghose is the Life Member of Indian Association for Productivity, Quality & Reliability, Kolkata, National Institute of Quality & Reliability, Trivandrum, Society for R & D Managers
of India, Trivandrum and Indian Remote Sensing Society, IIRS, Dehradun.