Multi-core Parallelization in Clojure – a Case Study.
Johann M. Kraus
Institute of Neural Information Processing
University of Ulm
89069 Ulm
johann.kraus@uni-ulm.de
ABSTRACT
In recent years, the demand for computational power in data
mining applications has increased due to rapidly growing
data sets. As a consequence, standard algorithms need to
be parallelized for fast processing of the generated data sets.
Unfortunately, most approaches for parallelizing algorithms
require a careful software design and a deep knowledge about
thread-safe programming. As a consequence they are hardly
applicable for rapid prototyping of new algorithms. We out-
line the process of multi-core parallelization using Clojure,
a new functional programming language utilizing the Java
Virtual Machine (JVM) that does not require knowledge of
thread-safe programming. We provide some benchmark re-
sults for our multi-core algorithm to demonstrate its compu-
tational power. The rationale behind Clojure is combining
the industry-standard JVM with functional programming,
immutable data structures, and a built-in concurrency sup-
port via software transactional memory. This makes it a
suitable tool for parallelization and rapid prototyping in
many areas. In this case study we present a multi-core
parallel implementation of the k-means cluster algorithm.
The multi-core algorithm shows an increase in computation
speed up to a factor of 10 compared to R or network based
parallelization.
Categories and Subject Descriptors
D.3.2 [Programming Languages]: Concurrent, distributed,
and parallel languages; I.5.3 [Pattern Recognition]: Clus-
tering
General Terms
Algorithms, Experimentation, Performance
Keywords
ACM proceedings, Lisp, Parallel programming, Clustering
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
ELW’09, July 6, 2009 Genova, Italy
Copyright 2009 ACM 978-1-60558-539-0 ...$5.00.
Hans A. Kestler
Institute of Neural Information Processing
University of Ulm
89069 Ulm
and
Department of Internal Medicine I
University Hospital Ulm
89081 Ulm
hans.kestler@uni-ulm.de
1. CONCEPTS OF PARALLEL PROGRAM-
MING
Currently, large collections of data are accumulated in
many areas, such as customer reports or high-throughput
data in the life-sciences. This has increased the need for
computer-intensive applications to handle these data sets.
Most software is implemented as a series of tasks starting
with a main() function and sequentially running through a
set of instructions to compute a result. A task is defined here
as a small set of instructions. Parallel programs process sev-
eral tasks simultaneously. Developers have to identify those
tasks of a program that can be processed in parallel. These
tasks are mapped to processes (threads), which are handled
by the different hardware resources available.
There are different challenges when decomposing a linear
program into simultaneously working threads. For instance
repeatedly sending large data sets over a network of comput-
ers reduces the profit gained from parallelization. Also load
balancing can be required to use all hardware resources effi-
ciently. Furthermore, synchronization problems arise when
two simultaneously running threads have to access the same
data. In the following we describe concepts to handle the
challenges of parallel programming.
Granularity of task decomposition.
Different levels of granularity are used to decompose a
program into smaller tasks. A program can be broken down
to the instruction level where a compiler may identify single
instructions without data dependencies that can be paral-
lelized. Another level is given by data parallelism. The data
is split into independent sets and the same instructions are
run on all subsets in parallel. A similar solution to task
decomposition is the parallelization of loops. In some lan-
guages there are special statements available to mark those
loops which can be run simultaneously due to data inde-
pendence. In this case the compiler is able to parallelize
the code. The most important level of parallelism for mod-
ern computing languages is a decomposition into functional
blocks. This approach requires synchronization techniques
as data or control structures are usually shared between dif-
ferent tasks.
Local vs. distributed hardware.
To overcome the limitations of currently available com-
puter hardware, different software systems for parallelizing
software for distributed and local multi-processor hardware
8
have been proposed, such as the Message Passing Interface
(MPI) [8] and Open Multi-Processing (Open MP) [6]. The
idea of message passing is to parallelize independent tasks
using a network of one master and several slave comput-
ers. While there is no possibility for communication be-
tween the slaves, this approach best fits scenarios where the
same algorithm is started several times on different data sets
or different analyses are calculated in parallel on the same
data set [11]. In contrast to the network-based architectures,
there are other approaches to parallelize software that are
designed to run on a single multi-core computer. In a multi-
core parallelization setting like in OpenMP or Clojure, there
is no need for network communication, as all threads run on
the same computer. Furthermore, the available memory can
be shared between different threads. Therefore, developing
multi-threaded programs reduces the overhead of communi-
cating through a network. In contrast, as every thread has
access to all objects on the heap there is a need for con-
currency control [20]. If two threads simultaneously try to
change the state of data, they potentially compromise the
data integrity. Concurrency control ensures that correct re-
sults are produced by concurrent operations.
Explicit vs. implicit parallelization.
Programming languages can support explicit or implicit
parallelization. In a language where only explicit paralleliza-
tion is available, a developer has to explicitly define commu-
nication and synchronization details for each task. This is
the standard programming scheme for MPI or thread-based
programming languages like Java. With implicit definitions
there is no need for manually administrating threads. Func-
tional programming languages can support this type of par-
allelization. In these languages parallelization is implicitly
defined by parallel processing of a set of functions without
side-effects and an immutable data base.
Thread programming.
Multi-core programming is closely linked to thread pro-
gramming. A sequential program is decomposed into sev-
eral tasks, which are then processed as threads. The con-
cept of thread programming is available in many program-
ming languages like C (PThreads or OpenMP threads), Java
(JThreads), or Fortran (OpenMP threads). Threads are
refinements of a process that share the same memory and
can be separately and simultaneously processed. Due to the
shared memory concept, communication between threads is
much faster than the communication of processes through
sockets. Invoking threads is also faster than generating pro-
cesses. Therefore, thread programming offers a flexible al-
ternative to programming with cooperating processes. The
execution of threads is handled by a scheduler that man-
ages the available processing time. During its lifetime every
thread undergoes different states. Figure 1 shows a typical
state diagram for threads. A thread is started in the state
new. Then it is alternating between the states runnable,
running, and waiting until it ends in the state terminated.
A scheduler controls changes between runnable and run-
ning. Blocking due to I/O operations or concurrency leads
to waiting. Different concurrency control mechanisms can
be used to administrate blocking of threads. For instance
Java provides mutex (mutual exclusion) to avoid simultane-
ous access of threads to a common resource. In Java syn-
chronized access to data has to be encapsulated in a syn-
9
new
start runnable
awake
schedule
waiting
end block
terminated running
Figure 1: During its lifetime a thread undergoes
different states from creation to termination.
chronized block. The following code shows an example for
a parallelized counter implemented in Java.
public c l a s s Counter {
private i n t v a l u e = 0 ;
public synchronized void i n c r {
v a l u e = v a l u e +1;
}
}
Counter c o u n t e r = new Counter ( ) ;
counter . incr ( ) ;
Every object in Java has an implicit mutex variable that is
implicitly set when a thread enters a synchronized block. If a
mutex variable is already set, then this thread is blocked un-
til the concurrent thread exits the synchronized block. The
use of implicit mutex variables in Java handles synchronized
access to shared data. But the developer has still to declare
the synchronized blocks. Therefore, problems might occur
when using too few locks, too many locks, wrong locks, or
locks in the wrong order [1]. A typical example are dead-
locks, where two threads are each waiting for each other to
first release a resource (unlock).
Concurrency control.
Concurrency control ensures that software can be paral-
lelized without violating data integrity. Currently, the most
popular approach for managing concurrent programs is the
use of locks [6]. Locking and synchronizing ensures that
changes to the states of the data are coordinated. But im-
plementing thread-safe programs using locks can be fatally
error-prone [24].
Software transactional memory (STM) is a modern alter-
native to the lock-based concurrency control mechanism [26,
29]. The basic functionality of software transactional mem-
ory is analogous to controlling simultaneous access via trans-
actions in database management systems. The execution
of a task is encapsulated in a transaction, which ensures
that all actions on the data are atomic, consistent, and iso-
lated. Here, the term atomic means that either all changes
of a transaction to the data occur or none do. Consistent
means that the new data from the transaction is checked
for consistency before it is committed. Isolated means that
every transaction is encapsulated and cannot see effects of
any other transaction while it is running. Therefore, STM
allows for sharing changing states between threads in a syn-
chronous and coordinated manner. In contrast to lock-based
strategies, parallelization is possible without explicit thread
management and without explicit locking resources.
2. CLOJURE
Clojure as a new Lisp dialect uses a software transac-
tional memory system as a means of concurrency control.
In the following we highlight some features of the program-
ming language Clojure and its access to STM. For an in-
troduction to Clojure refer to the Rich Hickey’s website
(http://clojure.org) and to the book by Halloway [9].
Functional programming.
Clojure is a new dialect of Lisp. Modern programming
languages have already adopted many features of Lisp, but
the interchangeability of code and data and the macro sys-
tem still are unique to Lisp. Clojure is not designed to pro-
vide backward compatibility to ANSI Common Lisp. For
instance Clojure extends the s-expressions from lists to vec-
tors and maps. All of its basic data structures are immutable
by design. This allows for very efficient copying of data, and
is an important requirement for Clojure’s software transac-
tional memory system with multiversion concurrency con-
trol [4].
Java Virtual Machine.
Clojure is hosted on the Java Virtual Machine (JVM). The
JVM is an industry-standard platform used by many devel-
opers and provides access to numerous Java libraries. Its use
is not restricted to a specific operating system. Therefore it
is already used as an operational basis by several program-
ming languages like Groovy or Scala [17, 23]. Compared to
Haskell and its STM system, Clojure can additionally profit
from this Java basis [24]. This assures that the final parallel
program can be easily complemented by a Graphical User
Interface (GUI) and distributed as a platform-independent
Java application.
Realizing software transactional memory in Clojure.
On the one hand, functional programming languages can
help to eliminate concurrency issues while retaining thread-
safe code, as their built-in immutable data structures will
never change their state [30]. On the other hand, it is often
necessary to provide mutable states. Concurrency control
via software transactional memory enables an efficient man-
agement of mutable shared memory. This enables a fast and
parallel design of algorithms without the need for manually
administrating threads and concurrency issues. Different
variants of realizing software transactional memory systems
exist, such as locking / pessimistic, lock-free / optimistic,
and hybrids. Clojure uses a multiversion concurrency con-
trol (MVCC, Figure 2) [4]. Every thread has its own copy of
the data and will process all of its modifications to the shared
data without blocking other threads. Before committing the
changes, it is checked whether other threads have altered the
data in use. If so, the whole process of transaction has to be
retried until a consistent commit can be done. Unless the
number of failed transactions is small, this optimistic con-
currency control technique scales well with the number of
threads. Figure 2 shows the principles of Clojure’s software
10
transactional model. All changes to the state of data are en-
capsulated in transactions, i.e. every thread has a copy of its
working data and can change its value. During submission
of the changes to the shared memory the consistency of the
internal state is checked. If no interim changes occurred, the
submission is performed, see Figure 2 Thread 0 and Thread
n. If another thread working on another copy of the same
data meanwhile has submitted its changes, the transaction
is rejected and restarted with a new copy of the data, see
Figure 2 Thread 1 and Thread 2.
MVCC supplies every actor with its own snapshot of the
data. In Clojure Refs and Agents handle changing mutable
states to new values automatically. Here, all interactions
with Refs must be encapsulated within a transaction via the
dosync macro. Transactions support synchronous change to
multiple Refs. Every actor will see a consistent snapshot of
the world at a certain time point. Any changes to the data
made by an actor will not be seen by other actors until the
transaction has been committed. A transaction is automati-
cally restarted if the states to be changed have been changed
by another actor in the meantime. There are three methods
to change the value of a Ref. Using ref-set the value is just
set to a new value, with alter the current in-transaction-
value of the Ref is modified. The commute operation enables
simultaneous write access without checking for intermediate
state changes, e.g. commutative changes to a list. For in-
stance a parallelized counter is realized with Refs as follows:
( def c o u n t e r ( r e f 0 ) )
( dosync ( a l t e r c o u n t e r inc ) )
In contrast to Refs, Agents offer the possibility of asyn-
chronous changes to a single reference. Functions are sent
to an Agent which are asynchronously applied to an Agent’s
state and whose return value will become the Agent’s up-
dated state. The actions of all Agents are collected in a
thread pool and executed at some future point in time. Clo-
jure’s Agents are integrated with the software transactional
memory system, i.e. any changes made during an action
are delayed until the final commit. As with all of Clojure’s
concurrency support, no user-code locking is involved. The
following code shows a parallelized counter realized with
Agents in Clojure:
( def c o u n t e r ( a g e n t 0 ) )
( send c o u n t e r inc )
3. A CLUSTERING CASE STUDY
With our case study we wanted to inspect the capability
of Clojure being used to parallelize existing algorithms on a
multi-core desktop computer. We decided to implement an
algorithm that does only require some basic transactional
properties of Clojure to benchmark the potential benefit of
the parallelization. Also the final program should give ex-
actly the same results as the single core implementation.
The k-means cluster algorithm fits our requirements and is
also included in several parallel benchmark suites [22]. Most
of the runtime for large data sets is spent on vector calcula-
tions. Therefore, the parallel implementation was expected
to scale well with the number of available processors.
 Thread 0

changed Data k Data k (copy) changed Data 2

Thread 1

Data k Data 2 (copy)

Data 2

Data 1 Data 2 (copy)

Data 0
Thread 2

changed Data 0 Data 0 (copy) changed Data 2

Submission fails.
Data has changed.
Reload data.
Thread n

Figure 2: The design principle of software transactional memory as implemented in Clojure.

The k-means clustering algorithm.
Clustering is a classical example of unsupervised learning,
i.e. learning without a teacher. The term cluster analysis
summarizes a collection of methods for generating hypothe-
ses about the structure of the data by solely exploring pair-
wise distances or similarities in the data space. Clustering is
often applied as a first step in data analysis for the creation
of initial hypotheses. Let X = {x1 , . . . , xN } be a set of data
points with the feature vector xi ∈ Rd . Cluster analysis is
used to build a partition of a data set containing k clusters
such that data points within a cluster are more similar to
each other than points from different clusters. A partition
P (k) is a set of clusters {C1 , C2 , . . . , Ck } with 0 < k < N
and meets the following conditions:
[
k F u n c t i o n KMeans
Ci = X , Ci 6= ∅ (1a)
i=1
Ci ∩ Cj = ∅ , i 6= j (1b)
Figure 3 illustrates the idea of clustering data. The arti-
ficial two-dimensional data set is split into 9 clusters, where
each cluster centroid is placed in the center of gravity of the
cluster. The dotted lines mark the borders of the clusters,
which are equally distant to all neighboring clusters. The
basic clustering task can be formulated as an optimisation
problem:
Partitional cluster analysis: For a fixed number of
groups k find that partition P (k) of a data set X out of
the set of all possible partitions Φ(X, k) for which a chosen
objective function f : Φ(X, k) → R+ is optimized. For all
possible partitions with k clusters compute the value of the
objective function f . The partition with the best value is
the set of clusters sought.
This brute force method is computationally infeasible as
the cardinality of the set of all possible partitions is huge
even for small k and N . The cardinality of Φ(X, k) can be
computed by the Stirling numbers of the second kind [14]:
!
1 X
k
k k−i k N
|Φ(X, k)| = SN = (−1) i (2)
k! i=0 i
Existing algorithms provide different heuristics for this search
problem. K-means is probably one of the most popular of
these partitional cluster algorithms [21]. The following list-
ing shows the pseudocode for the k-means algorithm:
I n p u t : X = { x 1 , . . . , x n } ( Data t o be c l u s t e r e d )
k ( Number o f c l u s t e r s )
Output : C = { c 1 , . . . , c k } ( C l u s t e r c e n t r o i d s )
m: X−>C ( C l u s t e r a s s i g n m e n t s )
I n i t i a l i z e C ( e . g . random s e l e c t i o n from X)
While C h a s c h a n g e d
For e a c h x i i n X
m( x i ) = a r g m i n j d i s t a n c e ( x i , c j )
End
For e a c h c j i n C
c j = c e n t r o i d ( { x i | m( x i ) = j } )
End
End
Given a number k, the k-means algorithm splits a data set
X = {x1 . . . , xn } into k disjoint clusters. Hereby, the cluster
centroids µ1 , . . . , µk are placed in the center of gravity of the

11

Figure 3: An artificial two-dimensional dataset clus-
tered into 9 clusters. The centroids of the clusters
are marked by crosses. The dotted lines show the
border of the clusters where each neighboring cen-
troid is equally distant.
clusters C1 , . . . , Ck . The algorithm minimizes the objective
function:
X
k X
F (µj , Cj ) = ||xi − µj ||2 . (3)
j=1 xi ∈Cj
This amounts to minimizing the sum of squared distances of
data points to their respective cluster centroids. K-means is
implemented by iterating between two major steps, which
reassign data points to nearest cluster centroids and update
centroids (often also called prototypes) for the newly assem-
bled cluster. A centroid µj is updated by computing the
mean of all points in cluster Cj :
1 X
µj = xi . (4)
|Cj | x ∈C
i j
Parallel implementation of k-means in Clojure.
Using software transactional memory and Clojure’s built-
in thread management the parallel execution of code is im-
plicitly defined. In Clojure asynchronous state changes are
modeled with Agents. Agents independently perform ac-
tions and change their state according to the result of an
action. For the k-means algorithm the data set is split into
smaller pieces that are handled by these Agents. To reduce
the administrative overhead, several data points are handled
by one agent. Clusters are also implemented as agents to al-
low asynchronous changes to the cluster centroids. Because
every cluster has to know its members to update the cen-
troid additionally a list of members is included. This list is
decoupled from the cluster agents and accessed through the
STM system (via the commute function) in order to allow
simultaneous write access for the data subset agents without
need for synchronization. An overview of our parallel design
12
is given in Figure 4.
The following code shows the assignment step of the par-
allel k-means algorithm in Clojure:
( defn a s s i g n m e n t [ ]
(map #(send % update−d a t a a g e n t ) DataAgents )
( defn update−d a t a a g e n t [ d a t a p o i n t s ]
(map update−d a t a p o i n t d a t a p o i n t s ) )
( defn update−d a t a p o i n t [ d a t a p o i n t ]
( l e t [ newass ( n e a r e s t −c l u s t e r d a t a p o i n t ) ]
( dosync (commute ( nth MemberRefs newass )
conj ( : d a t a d a t a p o i n t ) ) )
( assoc d a t a p o i n t : a s s i g n m e n t newass ) ) )
During the assignment step (red lines) the DataAgents cal-
culate the nearest cluster centroid for each data point and
update their assignments. Additionally each data point is
written to the list of members (MemberRef) of its nearest
centroid. Simultaneous write access to these lists is pos-
sible through Clojure’s STM system. Each DataAgent re-
cieves a request to update the cluster assignments of its data
points via the assignment function. Each agent processes
the update-datapoint function over its list of data points.
This function searches for the nearest cluster, adds the data
point to the respective MemberRef and updates the assign-
ment of the data point. The following code shows the update
step of the parallel k-means algorithm in Clojure:
( defn update [ ]
(map #(send % update−c l u s t e r a g e n t ) C l u s t e r A g e n t s )
( defn update−c l u s t e r a g e n t [ a g t ]
( l e t [ members ( deref ( nth MemberRefs
( : number a g t ) ) )
newcen ( reduce c a l c u l a t e −c e n t e r members ) ]
( dosync ( r e f −s e t ( nth MemberRefs ( : number a g t ) )
nil ))
( assoc a g t : d a t a newcen ) ) )
During the update step (blue lines) each Cluster Agent cal-
culates its new centroid from its list of members. The up-
date function distributes the call for update to each cluster
agent. The update-clusteragent function collects all clus-
ter members from the n-th MemberRef and then calculates
their new center. The n-th element of the MemberRefs is
cleared for the next assignment step. Finally, the new cen-
ter is assigned to the cluster agent. The loop of assignment
updates and center updates is repeated until no changes to
the centers occur.
Cluster validation.
Cluster validation provides methods to evaluate the re-
sults of cluster analysis in an objective way. The validation
methods always measure a selected point of view, that is
one has to predefine an expected characteristic of the data.
Several validation methods have been proposed. Following
Jain et al. [14] they are grouped into three types of criteria:
Internal criteria Measure the overlap between cluster struc-
ture and information inherent in data, for example sil-
houette, inter-cluster similarity.
External criteria Compare different partitions, for exam-
ple Rand-Index, Jaccard-Index, Fowlkes and Mallows.
Relative criteria Decide which of two structures is better
in some sense, for example quantifying the difference
between single-linkage or complete-linkage.
 Data Data Data Data Data
Agent 0 Agent 1 Agent 2 Agent 3 Agent n

Member
Cluster Ref 0
Agent 0

Cluster Member
Agent 1 Ref 1

Cluster Member
Agent k Ref k
read

write

Figure 4: Basic design of the multi-core k-means algorithm in Clojure. Each Data Agent calculates its new
cluster assignment and adds its value to the list of cluster members. The Cluster Agents calculate their new
centers from the list of cluster members.

To analyze cluster algorithms they are often applied on a-

priori labeled data sets and compared to the labels with the Table 1: External cluster indices describe the agree-
use of an external criterion. In the following we shortly in- ment of two cluster results. All of the following in-
troduce the basic concept of evaluating cluster results using dices can be derived from the contingency table of
an external cluster indices. An external index describes to two partitions.
which degree two partitions of N objects agree. Given a set Name Formula
of N objects X = {x1 , x2 , . . . , xN } and two different parti- Pr
i=1
Ps 2 Pr
j=1 nij −( “i=1 n2
Ps 2
i. + j=1 n.j )/2
Rand [28] 1+ ”
tions P = {C1 , C2 , . . . , Cr } and Q = {D1 , D2 , . . . , Ds } into n
2
r and s clusters respectively. A contingency table is defined Pr Ps 2
to compare these two partitions. Here nij denotes the num- i=1 j=1 nij −n
Jaccard [13] Pr P P Ps
n2 + s n2 − r n2 −n
ber of objects that are both in clusters Ci and Dj , ni. and i=1 i. j=1 .j i=1 j=1 ij

n.j denote the total number of objects in cluster Ci and Dj Pr

i=1
Ps
n2 −n
Fowlkes and Mallows [7] Pr “ ” j=1 ij“n ”
ni. Ps
respectively. Hubert and Arabie [12] give some indices based 2(
i=1 2 j=1
.j )1/2
2
on the contingency table of two partitions, see Table 1. As
our artificial data sets have been designed to reflect a spe-
cific clustering, we also decided to use the Rand-index [28]
as an external cluster validity index. of data points, dimensions, and clusters to simulate differ-
ent aspects of the k-means cluster analysis. The results are
summarized in the following paragraphs.
4. EXPERIMENTS AND RESULTS
We compared the performance of our multi-core algorithm Large data sets without cluster structure.
to ParaKMeans [18] and the k-means function implemented These data sets are included to benchmark the runtime of
in R [25]. Our test hardware was a Dell Precision T7400 the algorithms neglecting the effect of the random initializa-
with dual quad-core Intel Xeon 3.2 GHz and 32 GB RAM. tion of the centroids. Each feature is uniformly distributed
ParaKMeans was tested on the web-interface at over the interval [0, 1], i.e. there is apparently no cluster
http://bioanalysis.genomics.mcg.edu/parakmeans , structure hidden in the data set. The performance of clus-
but could not process our larger test data set because the tering these simulated data sets into 5 and 20 clusters is
master computer did not provide enough memory. We gen- summarized in Figure 5. In case of the small data set with
erated several artificial data sets having different numbers 10.000 data points the computational overhead of the paral-

13
 10.000 cases, 100 dimensions 1.000.000 cases, 200 dimensions R Clojure
150 20 Cluster 20 Cluster

450

600
runtime (seconds)
runtime (seconds)

runtime (minutes)
100

300

400
150
50

200
0

0
ParaKMeans R Clojure R Clojure
200 / 10 200 / 20 500 / 10 500 / 20 200 / 10 200 / 20 500 / 10 500 / 20

Number of samples / Number of clusters

Figure 5: Benchmark results for the simulated data Figure 7: Benchmark results for the simulated data
sets without clustering structure comparing the run- sets with 50.000 samples, 200 / 500 dimensions, and
time of ParaKMeans, R, and Clojure. For the 10 / 20 clusters. Runtimes for R are given on the
smaller data set (left) the computational overhead left, runtimes for Clojure on the right.
of the parallelization negatively effects the runtime.
For the large data (right) set an improvement of the
R Clojure
runtime by a factor of 10 can be observed.

800
runtime (seconds)
R Clojure

600
250

400
200
runtime (seconds)
150

200
100

200 / 10 200 / 20 500 / 10 500 / 20 200 / 10 200 / 20 500 / 10 500 / 20

50

Number of samples / Number of clusters

0

200 / 10 200 / 20 500 / 10 500 / 20
Number of samples / Number of clusters
Figure 6: Benchmark results for the simulated data
sets with 20.000 samples, 200 / 500 dimensions, and
10 / 20 clusters. Runtimes for R are given on the
left, runtimes for Clojure on the right.
lelization negatively effects the runtime. For the extremely
large data set having 1.000.000 data points an improvement
of the runtime by a factor of 10 can be observed, show-
ing that our implementation scales well with the number of
computer cores.
Data sets with cluster structure.
We simulated clustered data sets using separate multivari-
ate normal distributions as the basis for each cluster. We
used a combination of different conditions for the number
of data points (n = 20.000, 50.000, 100.000), number of di-
mensions (d = 200, 500), and number of clusters present
in the data set (k = 10, 20) to generate a total of 12 data
sets. We generated the artificial data sets to reflect a specific
cluster structure. Figure 6, Figure 7, and Figure 8 show the
benchmark results for these data sets.
Stability of large data sets.
To further illustrate the need for a high computational
speed of cluster algorithms we performed simulations to in-
fer the number of clusters inherent in a data set. This is an
often discussed issue in cluster analysis and additionally con-
cerns the reliability of the computed clustering / partition.
Evaluating the stability of cluster results have been widely
200 / 10 200 / 20 500 / 10 500 / 20 Figure 8: Benchmark results for the simulated data
sets with 100.000 samples, 200 / 500 dimensions,
and 10 / 20 clusters. Runtimes for R are given on
the left, runtimes for Clojure on the right.
discussed in the literature [10]. It has been shown that a re-
peated cluster analysis with different methods, parameters,
feature sets or sample sizes can help to reveal the underly-
ing data structure. For instance, the bootstrap technique
is used for estimating the number of clusters [15]. If the
fluctuations among the partitions are small the clustering is
called robust. Although there are few theoretical findings
on the stability property of clusterings, this methodology
has proven to work well in practice [5, 16, 19]. Recently it
was shown that for large data sets determining the cluster
number via robustness analysis may be problematic [2,3,27].
Specifically, it was shown that considering theoretical prop-
erties of k-means like clustering in large sample size set-
tings, stability is not connected to the number of clusters,
but solely controlled by properties of the algorithm’s under-
lying objective function. Independent of the parameter k
the stability measure always converges to 1 if there is an
unique global optimal minimum for the input data set [3].
We generated two large data sets to simulate this behavior.
The stability is measured by comparing the agreement be-
tween the different results of running k-means on subsets of
the data. The agreement is measured with the Rand-index.
A higher value indicates increased stability. Results from
two examples are shown in Figure 9 and Figure 10. Here,
we simulated clustered data sets using separate multivari-
ate normal distributions as the basis for each cluster. We
generated two data sets with 200 and 100.000 samples each

14
 1.0
0.8
Rand index
Rand index
0.6
0.4
0.2
0.0
2 3 4 5 6 7
number of clusters
Figure 9: Stability resampling and clustering a small
data set (n=200, d =100).
containing 3 √
clusters. Each data set was resampled 10 times
leaving out n data points. The effect of resampling on
the stability of the clustering can be reproduced on both
data sets. Both experiments correctly predict a most stable
clustering into 3 clusters. Total running time was 0.56 sec
for the small data set and 49.26 min for the data set with
100000 samples. In each simulation 70 separate clusterings
were performed.
5. DISCUSSION AND CONCLUSIONS
Developing an algorithm usually includes continual rounds
of implementation, tests, bug fixing, and code refactoring.
The family of Lisp programming languages strongly sup-
ports this process by the read-eval-print loop. Immutable
data structures reduce the amount of bugs produced during
the implementation phase. We presented a case study for
multi-core parallelization of a data mining algorithm imple-
mented in Clojure a new Lisp dialect. In contrast to other
programming languages targeting the JVM, such as Scala,
Clojure integrates a software transactional memory system.
The design principle of the STM reduces the additional over-
head required by the design of parallel algorithms. In con-
trast to parallelization of software using MPI or OpenMP,
our approach only requires minimal additional effort in de-
signing the software.
The benchmark results show that for large data sets a sig-
nificant improvement in the runtime can be achieved by a
parallel multi-core implementation. This is especially im-
portant if simulations are performed repeatedly to infer the
inherent cluster number and to quantify the stability and
robustness of the predictions. To summarize, we ended up
with about 100 lines of code running a multi-core parallel
algorithm that is fast, easy to maintain, and yet ready to re-
lease as a platform independent standalone application. As
multi-core computers are the standard on the next gener-
15
1.0
0.8
0.6
0.4
0.2
0.0
2 3 4 5 6 7
number of clusters
Figure 10: Stability resampling and clustering a
large data set (n=100000, d =100).
ation desktop, implementing parallel multi-core algorithms
will be a highly efficient alternative to compute clusters.
6. ACKNOWLEDGEMENTS
This work is supported by the German Science Foundation
(SFB 518, Project C5), the Stifterverband für die Deutsche
Wissenschaft (HAK), and the Graduate School of Mathe-
matical Analysis of Evolution, Information, and Complexity
at the University of Ulm (JMK).
7. REFERENCES
[1] A.-R. Adl-Tabatabai, C. Kozyrakis, and B. Saha.
Unlocking concurrency. ACM Queue, 4(10):24–33,
2006.
[2] S. Ben-David, D. Pál, and H. U. Simon. Stability of
k-means clustering. In N. H. Bshouty and C. Gentile,
editors, Conference on Learning Theory, volume 4539
of Lecture Notes in Artificial Intelligence, pages 20–34,
Berlin, 2007. Springer.
[3] S. Ben-David, U. von Luxburg, and D. Pál. A sober
look at clustering stability. In J. G. Carbonell and
J. Siekmann, editors, Conference on Learning Theory,
volume 4005 of Lecture Notes in Artificial Intelligence,
pages 5–19, Berlin, 2006. Springer.
[4] P. A. Bernstein and N. Goodman. Concurrency
control in distributed database systems. ACM
Computing Surveys, 13(2):185–221, 1981.
[5] A. Bertoni and G. Valentini. Random projections for
assessing gene expression cluster stability. In
Proceedings of the IEEE-International Joint
Conference on Neural Networks (IJCNN), volume 1,
pages 149–154. IEEE Computer Society, 2005.
[6] B. Chapman, G. Jost, and R. van der Pas. Using
OpenMP: Portable Shared Memory Parallel
Programming. MIT Press, Cambridge, 2007.
 [7] E. Fowlkes and C. Mallows. A method for comparing O’Reilly, Sebastopol, 2007.
two hierarchical clusterings. Journal of the American [25] R Development Core Team. R: A Language and
Statistical Association, 78(383):553–569, 1983. Environment for Statistical Computing. R Foundation
[8] W. Gropp, E. Lusk, and A. Skjellum. Using MPI: for Statistical Computing, Vienna, 2009. ISBN
Portable Parallel Programming with the Message 3-900051-07-0.
Passing Interface. MIT Press, Cambridge, 1999. [26] R. Rajwar and J. Goodman. Transactional execution:
[9] S. Halloway. Programming Clojure. Pragmatic Toward reliable, high-performance multithreading.
Programmers, Raleigh, 2009. IEEE Micro, 23(6):117–125, 2003.
[10] J. Handl, J. Knowles, and D. Kell. Computational [27] A. Rakhlin and A. Caponnetto. Stability of k-means
cluster validation in post-genomic data analysis. clustering. In B. Schölkopf, J. C. Platt, and
Bioinformatics, 21(15):3201–3212, 2005. T. Hoffman, editors, Advances in Neural Information
[11] J. Hill, M. Hambley, T. Forster, M. Mewissen, T. M. Processing Systems 19, pages 1121–128. MIT Press,
Sloan, F. Scharinger, A. Trew, and P. Ghazal. Sprint: Cambridge, 2007.
A new parallel framework for r. BMC Bioinformatics, [28] W. Rand. Objective criteria for the evaluation of
9(558), 2008. clustering methods. Journal of the American
[12] L. Hubert and P. Arabie. Comparing partitions. Statistical Association, 66:846–850, 1971.
Journal of Mathematical Classification, 2:193–218, [29] N. Shavit and D. Touitou. Software transactional
1985. memory. In Proceedings of the 14th ACM Symposium
[13] P. Jaccard. Nouvelles recherches sur la distribution on Principles of Distributed Computing, pages
florale. Bulletin de la Société Vaudoise des sciences 204–213, New York, 1995. ACM Press.
naturelles, 44:223–270, 1908. [30] S. Thompson. Haskell: The Craft of Functional
[14] A. Jain and R. Dubes. Algorithms for Clustering Data. Programming. Addison Wesley, Boston, 2nd edition,
Prentice Hall, New Jersey, 1988. 1999.
[15] A. K. Jain and J. V. Moreau. Bootstrap technique in
cluster analysis. Pattern Recognition, 20(5):547–568, APPENDIX
1987.
[16] H. A. Kestler, A. Müller, F. Schwenker, T. Gress, A. SOURCE CODE
T. Mattfeldt, and G. Palm. Cluster analysis of Multi-core implementation of k-means in Clojure.
comparative genomic hybridization data. Lecture ; set global constants 1
Notes NATO ASI: Aritificial Intelligence and ( d e f d a t a s e t ( l o a d −tab− f i l e ” c l u s t e r e x a m p l e . ta b ”) ) 2
( def k 2) 3
Heuristic Methods for Bioinformatics, pages S–40, ( def maxiter 1000) 4
2001. Abstract. ( d e f numagents 8 ) 5
[17] D. Koenig, A. Glover, P. King, G. Laforge, and ( def stop ( r e f ’ ( ) ) 6
( d e f n check−s t o p [ c u r i t e r ] 7
J. Skeet. Groovy in Action. Manning Publications Co., ( o r (= c u r i t e r m a x i t e r ) ( e v e r y ? #(= t r u e %) 8
Greenwich, 2007. ( deref stop ) ) ) ) 9
[18] P. Kraj, A. Sharma, N. Garge, R. Podolsky, and R. A. ( d e f s t r u c t c l u s t e r : number : d a t a ) 10
( d e f n i n i t −c l u s t e r −a g e n t [ number d a t a ] 11
McIndoe. Parakmeans: Implementation of a ( a g e n t ( s t r u c t c l u s t e r number d a t a ) ) ) 12
parallelized k-means algorithm suitable for general ( d e f s t r u c t d a t a p o i n t : number : d a t a : a s s i g n m e n t ) 13
laboratory use. BMC Bioinformatics, 9(200), 2008. ( d e f n i n i t −data−a g e n t [ s u b s t a r t subend ] 14
( a g e n t ( d o a l l (map #( s t r u c t d a t a p o i n t %1 %2) 15
[19] T. Lange, V. Roth, M. L. Braun, and J. M. Buhmann. ( i t e r a t e inc substart ) 16
Stability-based validation of clustering solutions. ( drop s u b s t a r t 17
Neural Computation, 16(6):1299–1323, 2004. ( t a k e subend d a t a s e t ) ) ) ) ) ) 18
19
[20] D. Lea. Concurrent Programming in Java: Design ; f u n c t i o n s needed f o r t h e init step 20
Principles and Patterns. Addison Wesley, Boston, 2nd ( d e f n i n i t −member−r e f [ ] 21
edition, 2000. ( ref ’())) 22
23
[21] J. MacQueen. Some methods for classification and ( d e f n i n i t −c e n t e r s [ d a t a k ] 24
analysis of multivariate observations. In J. Neyman (map #(nth d a t a %) ( sample ( c o u n t d a t a ) k ) ) ) 25
26
and L. L. Cam, editors, Proceedings of the 5th ; i n i t i a l i z a t i o n s t e p o f k−means 27
Berkeley Symposium on Math, Statistics and ( defn i n i t [ ] 28
Probability, volume 1, pages 281–297, Berkely, 1967. ( d o s y n c ( r e f −s e t s t o p ’ ( ) ) ) 29
University of California Press. ( d e f data−a g e n t s (map #( i n i t −data−a g e n t %1 %2) 30
( t a k e numagents ( i t e r a t e #(+ ( i n t ( . Math ( c e i l 31
[22] C. C. Minh, J. Chung, C. Kozyrakis, and ( / ( c o u n t d a t a s e t ) numagents ) ) ) ) %) 0 ) ) 32
K. Olukotun. Stamp: Stanford transactional ( t a k e numagents ( i t e r a t e #(+ ( i n t ( . Math ( c e i l 33
( / ( c o u n t d a t a s e t ) numagents ) ) ) ) %) 34
applications for multi-processing. In IISWC ’08: (+ ( i n t ( . Math ( c e i l ( / ( c o u n t d a t a s e t ) 35
Proceedings of the IEEE International Symposium on numagents ) ) ) ) ) ) ) ) ) 36
Workload Characterization, pages 35–46, Los ( d e f c l u s t e r −a g e n t s (map #( i n i t −c l u s t e r −a g e n t 37
Alamitos, 2008. IEEE Computer Society. %1 %2) ( i t e r a t e i n c 0 ) 38
( i n i t −c e n t e r s d a t a s e t k ) ) ) 39
[23] M. Odersky, L. Spoon, and B. Venners. Programming ( d e f member−r e f s (map ( f n [ x ] ( i n i t −member−r e f ) ) 40
in Scala. Artima, Mountain View, 2008. ( range k ) ) ) 41
true ) 42
[24] S. Peyton-Jones. Beautiful concurrency. In A. Oram 43
and G. Wilson, editors, Beautiful code, chapter 24. ; f u n c t i o n s used f o r the assignment s t e p 44
( d e f n whichmin [#ˆ d o u b l e s x s ] 45

16
 ( areduce xs i r e t 0 46
( i f (< ( a g e t x s i ) ( a g e t x s r e t ) ) 47
i 48
ret ))) 49
50
( d e f n d i s t a n c e [#ˆ d o u b l e s a s #ˆd o u b l e s bs ] 51
( areduce as i r e t ( double 0) 52
(+ r e t ( ∗ (− ( a g e t a s i ) ( a g e t bs i ) ) 53
(− ( a g e t a s i ) ( a g e t bs i ) ) ) ) ) ) 54
55
( d e f n s e a r c h −b e s t −c l u s t e r [ p o i n t ] 56
( l e t [ d i s t ( do u b l e−a r r a y 57
(map #( d i s t a n c e ( : d a t a p o i n t ) 58
( : d a t a ( d e r e f %))) c l u s t e r −a g e n t s ) ) ] 59
( whichmin d i s t ) ) ) 60
61
( d e f n update−d a t a p o i n t [ p o i n t ] 62
( l e t [ newass ( s e a r c h −b e s t −c l u s t e r p o i n t ) ] 63
( d o s y nc ( commute s t o p c o n j 64
(= ( : a s s i g n m e n t p o i n t ) newass ) ) ) 65
( d o s y nc ( commute ( nth member−r e f s newass ) 66
c o n j ( : data p o i n t ) ) ) 67
( a s s o c p o i n t : a s s i g n m e n t newass ) ) ) 68
69
( d e f n update−data−a g e n t [ d a t a l i s t ] 70
( d o a l l ( pmap update−d a t a p o i n t d a t a l i s t ) ) ) 71
72
; a s s i g n m e n t s t e p o f k−means 73
( defn assignment [ ] 74
( do s y n c ( r e f −s e t s t o p ’ ( ) ) ) 75
( dorun (map #(s e n d % update−data−a g e n t ) 76
data−a g e n t s ) ) 77
( a p p l y a w a i t data−a g e n t s ) ) 78
79
; f u n c t i o n s u s e d i n t h e update s t e p 80
( d e f n da+ [#ˆ d o u b l e s a s #ˆd o u b l e s bs ] 81
( amap a s i r e t 82
(+ ( a g e t a s i ) ( a g e t bs i ) ) ) ) 83
84
( d e f n update−c e n t e r p o i n t [ members ] 85
( l e t [ s c a l e ( d o u b l e ( c o u n t members ) ) 86
#ˆd o u b l e s newcen ( r e d u c e da+ members ) ] 87
( amap newcen i r e t 88
( / ( a g e t newcen i ) s c a l e ) ) ) ) 89
90
( d e f n update−c l u s t e r −a g e n t [ c l u s ] 91
( l e t [mem ( d e r e f ( nth member−r e f s 92
( : number c l u s ) ) ) 93
newcen ( update−c e n t e r p o i n t mem ) ] 94
( d o s y nc ( r e f −s e t ( nth member−r e f s 95
( : number c l u s ) ) ’ ( ) ) ) 96
( a s s o c c l u s : d a t a newcen ) ) ) 97
98
; update s t e p o f k−means 99
( d e f n update [ ] 100
( dorun (map #(s e n d % update−c l u s t e r −a g e n t ) 101
c l u s t e r −a g e n t s ) ) 102
( a p p l y a w a i t c l u s t e r −a g e n t s ) ) 103
104
; t h e k−means f u n c t i o n i t s e l f 105
( d e f n kmeans [ ] 106
( init ) 107
( loop [ c u r i t e r 0] 108
( assignment ) 109
( update ) 110
( i f ( check−s t o p c u r i t e r ) 111
curiter 112
( recur ( inc curiter ) ) ) ) ) 113

17