()(
()(
) (3)
(4)
Where
v
i
is the particle velocity of the i
th
particle
x
i
is the i
th
, or current particle
i is the particles number
d is the dimension of searching space
rand ( ) is a random number in (0, 1)
c
1
is the individual factor
c
2
is the societal factor
pBest is the particle best
gBest is the global best
Both c
1
and c
2
are set to be 2 in all literature works analyzed and hence the same is adopted
here. The velocity v
i
of each particle is clamped to a maximum velocity v
max
which is
specified by the user. v
max
determines the resolution with which regions between the present
position and the target position are searched.
The pseudo code for PSO algorithm is given below
For each particle
Initialize particle position and velocity
END
Repeat
For each particle
Calculate fitness value
If the fitness value is better than its personal best
set current value as the new pBest
End
Choose the particle with the best fitness value of all as gBest
For each particle
Calculate particle velocity according equation (3)
Update particle position according equation (4)
End
Until maximum number of iterations or minimum error criteria
The initial population is selected based on fitness value. The velocity and position of all the
particles are set randomly. Based on the fitness function the importance of the particles is
evaluated. The fitness function designed is based on support and confidence of the
association rule. The objective of fitness function is maximization. The fitness function is
shown in equation 5.
() () (() () ) (5)
Fitness (k) is the fitness value of association rule type k, confidence (x) is the confidence of
association rule type k and support(x) is the actual support of association rule type k. When
the support and confidence values are larger, then larger is the fitness value meaning that it is
an important association rule.
2.3 Predictive Accuracy
Predictive accuracy measures the effectiveness of the rules mined. The mined rules must have
high predictive accuracy.
(6)
where |X&Y| is the number of records that satisfy both the antecedent X and consequent Y,
|X| is the number of rules satisfying the antecedent X.
3. PSO and its Variants
Particle swarm optimization is based on the intelligence. PSO has no overlapping and
mutation calculation. During the development of several generations, only the most optimist
particle can transmit information onto the other particles. The speed of the searching is very
fast and it occupies the bigger optimization ability, thereby completing easily.
The swarm behaviour varies between exploratory behaviour, that is, searching a broader
region of the search-space, and exploitative behaviour, that is, a locally oriented search so as
to get closer to a (possibly local) optimum. The PSO algorithm and its parameters must be
chosen properly to balance between exploration and exploitation to avoid premature
convergence to a local optimum and yet also ensures a good rate of convergence to the
optimum. To avoid premature convergence at local optima Particle swarm optimization
variants are proposed and tested for mining association rules.
Variations have been introduced in velocity updation function to ensure convergence towards
global optima rather than local optima.
3.1 Particle Swarm Optimization with Inertia Weight
Inertia weight is added to the velocity update function and the equation 3 is modified as
()(
()(
) (7)
where is the inertia weight factor. The inertia weight is employed to control the impact
of the previous history of velocities on the current velocity, thus to influence the trade-off
between global (wide-ranging) and local (nearby) exploration abilities of the "flying points".
A larger inertia weight facilitates global exploration (searching new areas) while a smaller
inertia weight tends to facilitate local exploration to fine-tune the current search area. Suitable
selection of the inertia weight can provide a balance between global and local exploration
abilities and thus require less iteration on average to find the optimum.
3.2. Chaotic Particle Swarm Optimization
The canonical PSO tends to struck at local optima and thereby leading to premature
convergence when applied for solving practical problems. To improve the global searching
capability and escape from local optima chaos is introduced in PSO [14]. Chaos is a
deterministic dynamic system which is very sensitive and dependent on its initial conditions
and parameters. The common method of generating chaotic behaviour is based on Zaslavskii
map[15]. This representation of map involves many variables. Setting right values for all
these variables involved increases the complexity of the system. Erroneous values might bring
down the accuracy of the system involved. Logistic map and tent map are also most
frequently used chaotic behaviour. The drawback of these maps is that the range of values
generated by both the maps after some iteration becomes fixed to a particular range. To
overcome this defect the tent map undisturbed by the logistic map [16] is introduced as the
chaotic behaviour. The new chaotic map model is proposed with the following equation.
(
)
(8)
{
(
( (
)
)
The initial value of u
0
and v
0
are set to 0.1. The slight tuning of initial values of u0 and v
0
creates wide range of values with good distribution. The chaotic operator chaotic_operator(k)
= v
k
is designed therefore to generate different chaotic operators by tuning u
0
and v
0
. The
value of u
0
is set to two different values for generating the chaotic operators 1 and 2.
The velocity updation equation based on chaotic PSO is given in equation 9.
) ()
3.3 Neighbourhood Selection in PSO
In the original PSO, two kinds of neighbourhoods are defined for PSO:
In the gBest swarm, all the particles are neighbours of each other; thus, the position of
the best overall particle in the swarm is used in the social term of the velocity update
equation. The gBest swarms converge fast, as all the particles are attracted
simultaneously to the best part of the search space. However, if the global optimum is
not close to the best particle, it may be impossible to the swarm to explore other areas;
this means that the swarm can be trapped in local optima.
In the lBest swarm, only a specific number of particles (neighbour count) affect the
velocity of a given particle. The swarm will converge slower but can locate the global
optimum with a greater chance.
As the local best (lBest) value leads to convergence at the global optima the lBest value is
selected from neighbourhood values rather than the particles best values so far. The
neighbourhood best (lBest) selection is done as follows;
Calculate the distance of the current particle from other particles by equation 10.
) (10)
Find the nearest m particles as the neighbour of the current particle based on distance
calculated
Choose the local optimum lBest among the neighbourhood in terms of fitness values
The number of neighbourhood particles m is set to 2. Velocity and position updation of
particles are based on equation 3 and 4. The velocity updation is restricted to maximum
velocity V
max
set by the user. The termination condition is set as fixed number of
generations.
3.4 Self Adaptive Particle Swarm Optimization (SAPSO1 and SAPSO2)
The original PSO has pretty good convergence ability, but suffers with the demerit of
premature convergence [11], due to the loss of diversity [12]. Improving the exploration
ability of PSO has been an active research topic in recent years. Thus, the proposed algorithm
introduces the concept of self-adaptation as the primary key to tune the two basic rules
velocity and position. Effectively, reinforcing a PSO implies improving the inertia weight
formulae and thereby maintaining diversity of population. The basic PSO, presented by
Eberhart and Kennedy in 1995 [3], has no Inertia Weight. In 1998, first time Shi and Eberhart
[13] presented the concept of Inertia Weight by introducing Constant Inertia Weight.
By looking at equation (3) more closely, it can be seen that the maximum velocity allowed
actually serves as a constraint that controls the maximum global exploration ability PSO can
have. By setting a too small maximum velocity allowed, maximum global exploration ability
is limited, and PSO will always favour a local search no matter what the inertia weight is. By
setting a large maximum velocity allowed, the PSO can have a large range of exploration
ability to select by selecting the inertia weight. Since the maximum velocity allowed affects
global exploration ability indirectly and the inertia weight affects it directly, it will generally
be better to control global exploration ability through inertia weight only. A way to do that is
to allow inertia weight itself to control exploration ability. Thus the inertia weight is made
self adaptive. Two self adaptive inertia weights are introduced for mining association rules in
this paper.
In order to linearly decrease the inertia weight as iteration progress the inertia weight is made
adaptive through the equation 11 in SAPSO1.
(11)
Where
and
(12)
Where ( ) is the inertia weight for the current generation, () is the inertia weight for
the previous generation,
and
min
Lenses 24 2 2 0.2 100 0.9 0.4
Car
Evaluation
700 2 2 0.4 100 0.9 0.4
Habermans
Survival
300 2 2 0.4 100 0.9 0.4
Post-
operative
Patient Care
87 2 2 0.3 100 0.9 0.4
Zoo 101 2 2 0.3 100 0.9 0.4
Balancing between exploration and exploitation is carried out using the variants of PSO
proposed and the results for the five datasets are plotted in figures 1 to 5
.
45
50
55
60
65
70
75
80
85
90
95
100
10 20 30 40 50 60 70 80 90 100
P
r
e
d
i
c
t
i
v
e
A
c
c
u
r
a
c
y
(
%
)
No. of Iterations
PSO
WPSO
CPSO
NPSO
SAPSO1
SAPSO2
SACPSO
Figure 1. Convergence of Predictive Accuracy for Lens Dataset
Figure 2. Convergence of Predictive Accuracy for Car Evaluation Dataset
Figure 3. Convergence of Predictive Accuracy for Habermans Survival Dataset
Figure4. Convergence of Predictive Accuracy for Post Operative Patient Care Dataset
84
86
88
90
92
94
96
98
100
102
10 20 30 40 50 60 70 80 90 100
P
r
e
d
i
c
t
i
v
e
A
c
c
u
r
a
c
y
No.of Iterations
PSO
WPSO
CPSO
NPSO
SAPSO1
SAPSO2
60
70
80
90
100
10 20 30 40 50 60 70 80 90 100
P
r
e
d
i
c
t
i
v
e
A
c
c
u
r
a
c
y
(
%
)
No. of Iterations
PSO
WPSO
CPSO
NPSO
SAPSO1
SAPSO2
SACPSO
40
50
60
70
80
90
100
10 20 30 40 50 60 70 80 90 100
P
r
e
d
i
c
t
i
v
e
A
c
c
u
r
a
c
y
(
%
)
No. of Iterations
PSO
WPSO
CPSO
NPSO
SAPSO1
SAPSO2
SACPSO
Figure 5. Convergence of Predictive Accuracy for Zoo Dataset
The Self adaptive variants SAPSO1, SAPSO2 and SACPSO give consistent performance
when compared to other variants throughout the generations. The predictive accuracy
achieved by applying these self adaptive methods for association rule mining is better when
compared to the normal variants. The traditional particle swam optimization method when
applied for AR mining converges at very early stage for all the datasets. The performance of
WPSO, CPSO and NPSO varies from dataset to dataset. It is consistent for Zoo and Post
operative patient care datasets while inconsistent for Lenses, Habermans survival and Car
evaluation datasets.
The scope of introducing the variants in PSO is to avoid premature convergence and in turn
increase the predictive accuracy of the mined rules. The predictive accuracy is plotted for the
variants of PSO for all the five datasets in figure 6.
Figure 6. Predictive Accuracy comparison for PSO Variants
The variants of PSO perform better when compared to traditional PSO for mining association
rules. In terms of predictive accuracy the self adaptive methods SAPSO1, SAPSO2 and
40
50
60
70
80
90
100
10 20 30 40 50 60 70 80 90 100
P
r
e
d
i
c
t
i
v
e
A
c
c
u
r
a
c
y
(
%
)
No. of Iterations
PSO
WPSO
CPSO
NPSO
SAPSO1
SAPSO2
SACPSO
75
80
85
90
95
100
Lenses Car Evaluation Habermans
Survival
Po-opert Care Zoo
P
r
e
d
i
c
t
i
v
e
A
c
c
u
r
a
c
y
(
%
)
PSO
CPSO
NPSO
WPSO
SAPSO2
SAPSO1
SACPSO
SACPSO perform better than the normal PSO variants CPSO, WPSO and NPSO. The
weighted PSO gives better performance for all the datasets among the chaotic PSO and
neighbourhood selection PSO.
The iteration at which maximum predictive accuracy attained for the five datasets by
applying the variants of PSO in association rule mining is shown in figure 7.
Figure 7. Convergence rate comparison for PSO variants
The convergence rate varies from dataset to dataset for all the methods. The method in which
the convergence at local optima is avoided generates association rules with maximum
accuracy. This could be noted from figures 6 and 7.
The variants of PSO attempt to avoid convergence at the local optima by balancing between
exploration and exploitation. The predictive accuracy achieved by the variants is also
enhanced for all the datasets. The inertia weight, chaotic operators, neighborhood selection
and adapting the inertia weight dynamically, introduced in velocity updation function
maintains the balancing of convergence at local optima and deviation from global optima.
The self adaptive methods perform better than other methods.
5. Conclusion
Association rule mining is one of the most important tasks in data mining community because
the data being generated and stored in databases are already enormous and continues to grow
very fast. Particle Swarm Optimization algorithm mimics the social behaviour instead of
survival of fitness used in most of evolution algorithms. This principle reduces the time
complexity of PSO when compared to other algorithms. The convergence at local optima also
tends to reduce the time complexity.
In this paper inertia weight, chaotic operators, Neighbourhood selection and two adaptive
methods for inertia weight are introduced in the velocity updation function. These variants
when applied for association rule mining results in increased predictive accuracy for all the
five datasets used. The shift in convergence rate is achieved by avoiding convergence at local
optima though the variants of PSO. This also enhances the efficiency of the rules mined.
0
10
20
30
40
50
60
70
80
90
100
Lenses Car Evaluation Habermans
Survival
Po-opert Care Zoo
I
t
e
r
a
t
i
o
n
PSO
WPSO
CPSO
NPSO
SAPSO1
SAPSO2
SACPSO
When compared to PSO the PSO variants perform better both in terms of predictive accuracy
and balancing between exploration and exploitation. The three self adaptive methods
SAPSO1, SAPSO2 and SACPSO exhibit consistent performance for all the datasets. The
inertia weight factor performs better among the other PSO variants. The behaviour of Chaotic
PSO and neighbourhood selection in PSO varies from dataset to dataset depending on the
attributes involved and its values.
Avoiding exploitation at global search and testing on more datasets could be taken up for
further exploration.
References
1. A.A.Freitas. A survey of evolutionary algorithms for data mining and knowledge
discovery, Advances in Evolutionary Computation. Springer-Verlag, 2001.
2. Torn, A. Zilinskas (Eds.), Global Optimization, Lecture Notes in Computer Science, vol.
350, Springer-Verlag, 1989.
3. J. Kennedy, R. Eberhart, Particle swarm optimization, International Conference on
Neural Networks, pp. 19421948, 1995.
4. R. Agrawal, T. Imielin ski, A. Swami, Mining association rules between sets of items in
large databases, ACM SIGMOD 22 (2), pp.207216, 1993.
5. Tiago Sousa, Ana Paula Neves F. da Silva, Arlindo Silva , Ernesto Costa, Particle Swarm
Based Data Mining Algorithms for Classification Tasks, Parallel Computing, 30, pp.
767-783, Elsevier, 2004
6. Yang Shi, Hongcheng Liu, Liang Gao, Guohui Zhang, Cellular particle swarm
optimization, Information Sciences, 181, pp.44604493,2011.
7. M. Clerc, J. Kennedy, The particle swarm-explosion, stability, and convergence in a
multidimensional complex space, IEEE Transactions on Evolutionary Computation,
pp.5873, 2002.
8. F. Van den Bergh, A.P. Engelbrecht, A study of particle swarm optimization particle
trajectories, Information Sciences, 176, pp.937971, 2006.
9. J. Kenndy, Small worlds and mega-minds: effects of neighborhood topology on particle
swarm performance, In: Proceedings of IEEE Congress on Evolutionary Computation,
pp. 19311938, 1999.
10. W.-C. Yeh, Novel swarm optimization for mining classification rules on thyroid gland
data, Inform. Sci. , doi:10.1016/j.ins.2012.02.009, 2012
11. Zhao Xinchao, A perturbed particle swarm algorithm for numerical optimization,
Applied Soft Computing 10 (1), pp. 119124, 2010.
12. Yuxin Zhao, Wei Zub, Haitao Zeng, A modified particle swarm optimization via particle
visual modeling analysis, Computers and Mathematics with Applications 57, pp. 2022
2029, 2009.
13. Y. Shi and R. Eberhart., A modified particle swarm optimizer, International Conference
on Evolutionary Computation Proceedings, IEEE, pp. 6973, 1998.
14. W.J. Kong, W.J. Cheng, J.L. Ding, T.Y. Chai, A Reliable and Efficient Hybrid PSO
Algorithm for Parameter Optimization of LS-SVM for Production Index Prediction
Model, Third International Symposium on Computational Intelligence and Design, vol.2,
pp.140-143, 2010.
15. Bilal Atlas, Erhan Akin, Multi-objective rule mining using a chaotic particle swarm
optimization algorithms, Knowledge based systems,23, pp. 455-460,2009.
16. lal Alatas, Erhan Akin, A. Bedri Ozer, Chaos embedded particle swarm optimization,
Chaos,Solitons&Fractals, vol. 40, no.4, pp. 1715 - 1734, 2009.
17. R.C. Eberhart and Y. Shi., Tracking and optimizing dynamic systems with particle
swarms, Proceedings of the 2001 Congress on Evolutionary Computation, volume 1, pp.
94100, IEEE, 2002
18. M.S. Arumugam and MVC Rao., On the performance of the particle swarm optimization
algorithm with various Inertia Weight variants for computing optimal control of a class
of hybrid systems, Discrete Dynamics in Nature and Society, 2006.
19. Y. Feng, G.F. Teng, A.X. Wang, and Y.M. Yao., Chaotic Inertia Weight in Particle
Swarm Optimization, Proceedings of the 2001 Congress on Innovative Computing,
Information and Control, pp. 475-481. IEEE, 2008.
20. D.B. Chen, C.X. Zhao, Particle swarm optimization with adaptive population size and
its application, Applied Soft Computing. Pp.3948, 2009.
21. Y.S. Jin, K. Joshua, H.M. Lu, Y.Z. Liang, B.K. Douglas, The landscape adaptive
particle swarm optimizer, Applied Soft Computing. 8, pp. 295 304, 2008.
22. A. Nickabadi, M.M. Ebadzadeh, R. Safabakhsh, A novel particle swarm optimization
algorithm with adaptive inertia weight, Appl. Soft Computing, 11,pp. 36583670, 2011.