Architecture Optimizations for Synchronization and Communication on Chip
Multiprocessors
Sevin Fide and Stephen Jenks
Department of Electrical Engineering and Computer Science
University of California, Irvine, USA
{sevin.fide, stephen.jenks}@uci.edu
Abstract
Chip multiprocessors (CMPs) enable concurrent execu-
tion of multiple threads using several cores on a die. Cur-
rent CMPs behave much like symmetric multiprocessors
and do not take advantage of the proximity between cores to
improve synchronization and communication between con-
current threads. Thread synchronization and communica-
tion instead use memory/cache interactions. We propose
two architectural enhancements to support fine grain syn-
chronization and communication between threads that re-
duce overhead and memory/cache contention. Register-
Based Synchronization exploits the proximity between cores
to provide low-latency shared registers for synchronization.
This approach can save significant power over spin waiting
when blocking events that suspend the core are used. Pre-
pushing provides software controlled data forwarding be-
tween caches to reduce coherence traffic and improve cache
latency and hit rates. We explore the behavior of these
approaches, and evaluate their effectiveness at improving
synchronization and communication performance on CMPs
with private caches. Our simulation results show significant
reduction in inter-core traffic, latencies, and miss rates.
1. Introduction
The recent trend to increase microprocessor performance
by adding cores rather than increasing clock frequency or
instruction level parallelism has resulted in powerful chip
multiprocessors (CMPs). CMPs are built by incorporating
several processor cores on a single die, enabling concurrent
execution of multiple threads. To date, these cores have
been very similar to their uniprocessor counterparts, though
heterogeneous multi-core processor architectures, like the
Cell [6], exist or are proposed. Most CMPs share one
or more levels of the memory hierarchy among the cores,
with some sharing cache levels and others sharing the mem-
978-1-4244-1694-3/08/$25.00 ©2008 IEEE
Authorized licensed use limited to: Hewlett-Packard via the HP Labs Research Library. Downloaded on May 25, 2009 at 05:31 from IEEE Xplore. Restrictions apply.
ory interface. To achieve concurrent execution of multiple
threads on CMPs, applications must be explicitly restruc-
tured to exploit thread level parallelism (TLP), either by the
programmer or compiler. It is difficult to fully exploit per-
formance offered by today’s CMPs, especially when run-
ning applications with multiple threads that frequently com-
municate and synchronize with each other. Even though the
memory hierarchy is shared among the processors, there is
no explicit synchronization and communication support for
multithreaded applications to take advantage of the proxim-
ity between cores.
A parallel execution approach, called the Synchronized
Pipelined Parallelism Model (SPPM) [17], was introduced
to reduce the demand on the memory bus by restructuring
applications into producer-consumer pairs that communi-
cate through the cache rather than the memory. The pro-
ducer fetches data from the memory and modifies it. While
the data is still in the cache, the consumer accesses it. How-
ever, the producer and consumer need to be synchronized
to prevent the producer from getting far ahead of the con-
sumer. Therefore, SPPM enforces tight synchronization be-
tween the producer and consumer to make the producer’s
data available in the cache for the consumer’s access. This
approach reduces or eliminates the consumer’s need to fetch
data from the memory, resulting in better performance.
Spin waits are often used to provide synchronization be-
tween multiple threads, and they may be costly if the threads
are asymmetric or there are many synchronization points in
the application. Several past machines employed hardware
registers to facilitate synchronization among multiple pro-
cessors. We believe the employment of similar registers in
CMPs, along with blocking instructions that can temporar-
ily suspend a waiting core, will result in better performance
and power savings than spin waits.
Memory bandwidth is a potential bottleneck for CMPs
and may lead to poor performance. As each thread works
independently on its own data, the CMP’s shared memory
interface may be overwhelmed by many simultaneous ac-
cesses. In addition, the threads may cause cache pollution,
 evicting each other’s data from any shared cache. Com-
munication between multiple threads on CMPs takes place
via the shared memory hierarchy, such as a shared cache
or main memory. Due to demand-based data transfers and
cache coherence mechanisms, the requesting thread stalls
until the data is delivered. We believe a mechanism to ex-
ploit the cores’ proximity and allow fast communications
between cores is needed. Previous research has shown the
benefits of data forwarding to reduce communication la-
tency in distributed shared memory multiprocessors. We
propose employing a similar data movement concept within
the chip on CMPs with private caches that will extend the
communication support offered by today’s CMPs, and re-
sult in reduced communication latency and miss rates.
The hardware optimizations proposed here are applica-
ble to many types of communicating threads, but were mo-
tivated by SPPM’s behavior on current CMPs, particularly
those with private caches. It is shown in [18] that commu-
nicating through the cache coherence mechanism is slower
than communicating through memory for some common
CMPs. Using Register-Based Synchronization (RBS) will
reduce the spin waiting of the threads and cut cache con-
tention and overhead. In addition, our approach to move
data as it becomes available (called prepushing), reduces
the latency experienced by the consumer thread as cache
misses are turned into hits.
Similar ideas have been studied in the literature for large-
scale machines, but application of RBS and prepushing to
CMPs is new. Our work also focuses on SPPM applica-
tions, which provide better memory performance on CMPs
than regular parallel applications. Our implementation does
not rely on directory cache coherence protocols, which has
been the case in previous studies, but uses broadcast cache
coherence protocol similar to that of AMD’s Opteron [8].
Furthermore, our study targets CMPs with private L1 and
L2 caches, which is very different from previous studies.
Finally, prepushing forwards data based on hints received
from the application, and hence is software controlled to
reduce complexity and provide flexibility.
Our contributions include: (1) We present significant
architecture optimizations to improve the synchronization
and communication support for multithreaded producer-
consumer applications running on CMPs. (2) We evalu-
ate the performance of a well-established synchronization
mechanism, hardware registers, on today’s CMPs. (3) We
introduce a data forwarding approach, prepushing, tailored
to CMPs with private caches. (4) We evaluate performance
of four different prepushing models, including shared and
exclusive prepushing and prepushing to different cache lev-
els. (5) RBS and prepushing show promising results as
architecture optimizations to support synchronization and
communication on CMPs.
The rest of the paper is organized as follows: Section
Authorized licensed use limited to: Hewlett-Packard via the HP Labs Research Library. Downloaded on May 25, 2009 at 05:31 from IEEE Xplore. Restrictions apply.
2 outlines prior related research. Section 3 describes our
architecture optimizations to improve synchronization and
communication support. Section 4 presents our benchmark
applications, while Section 5 shows the results of our sim-
ulations and compares performance with and without our
techniques. Finally, Section 6 summarizes our contributions
and outlines future work to be done to further improve syn-
chronization and communication support on CMPs.
2. Related Research
Several machines used hardware registers to provide
synchronization and communication among multiple pro-
cessors. The Cray X-MP [2] had several clusters of shared
registers for interprocess communication and synchroniza-
tion. The operating system could assign a cluster to several
processors, to enable the use of the cluster for communi-
cation and synchronization. The Cray T3E’s [15] memory
interface was improved with a set of explicitly-managed ex-
ternal registers. All remote communication and synchro-
nization was done between these registers and memory. The
M-Machine [4] also provided synchronization through reg-
isters. For register synchronization, full/empty bits were
used to determine when register values were valid. This ap-
proach allowed the waiting thread to stall rather than spin.
The experiment results show that register-based communi-
cation and synchronization perform better than memory-
based communication and synchronization.
A hardware-based synchronization approach for simul-
taneous multithreaded (SMT) processors is presented in
[16]. The presented approach, blocking acquire and re-
lease, is a hardware implementation of traditional soft-
ware synchronization approaches. Synchronization through
full/empty registers is also presented. The success in prior
work on hardware-based synchronization leads us to em-
ploy full/empty registers to extend the synchronization sup-
port offered by today’s CMPs.
Data forwarding has been studied on several platforms.
The Stanford Dash Multiprocessor [11] provides operations
that allow the producer to send data directly to consumers.
The update-write operation sends data to all processors that
have the data in their caches, while the deliver operation
sends the data to specified processors. The KSR1 [5] pro-
vides programmers with a poststore operation. When a vari-
able is updated, using poststore causes a copy of the variable
to be sent to all caches which contain a copy of that variable.
The experimental study of the poststore is presented in [14].
A compiler algorithm framework for data forwarding is
presented in [10]. Write-and-Forward assembly instruction
is inserted by the compiler replacing ordinary write instruc-
tions. The same data forwarding approach and prefetch-
ing are compared in [9]. Similarly, another study shows
that prefetching is insufficient for producer-consumer shar-
 ing patterns and migratory sharing patterns in [1]. A range
of consumer- and producer-initiated mechanisms and their
performance on a set of benchmarks is studied in [3]. Store-
ORDered Streams (SORDS) [19] is a memory streaming
approach to send data from producers to consumers in dis-
tributed shared memory multiprocessors. This approach is
based on the producer-consumer temporal address correla-
tion, which claims that shared values are consumed in ap-
proximately the same order that they were produced.
Although all these studies on data forwarding fall into
the same category as ours, the implementation techniques
and target areas are different. First, our implementation
does not rely on directory cache coherence protocols, but
uses broadcast cache coherence protocol similar to AMD
Opteron’s cache coherence protocol. Second, our study is
not tailored to distributed shared multiprocessors, but tar-
gets CMPs with private L1 and L2 caches. Data forward-
ing on CMPs is new, because CMPs are new. Third, our
approach forwards data based on hints received from the
application, and hence it is software controlled to reduce
complexity and provide flexibility.
3. CMP Architecture Optimizations
3.1. Register-Based Synchronization (RBS)
In earlier systems, hardware registers were used to pro-
vide synchronization and communication among proces-
sors. The success of this prior work led us to employ
full/empty registers to extend the synchronization support
offered by today’s CMPs. Even though the concept of using
hardware registers for synchronization and communication
is not new, it has not been applied to CMPs before, yet is
sorely needed. As discussed in Section 2, research shows
that RBS performs better than memory-based communica-
tion and synchronization.
In SPPM applications, the consumer is usually faster
than the producer because it finds its data in the cache rather
than the memory. Thus, when the consumer is done, it
has to spin wait until the producer updates the next data
block and grants access. Figure 1 shows our RBS scheme
for CMPs. The shared register has a full/empty status bit.
When the producer reaches the synchronization point, it sets
the register status to full, which wakes the producer up.
When the consumer is done, instead of spin waiting and
consuming system resources, it goes into idle mode to save
power. Upon register status change, the consumer wakes up
and the process repeats. We consider the synchronization
registers to be shared between cores.
To evaluate RBS, we modified GEMS [13] Ruby mem-
ory model, which is based on the Simics [12] full system
simulation platform. Instead of adding hardware registers
to the system, we used memory mapped locations to track
Authorized licensed use limited to: Hewlett-Packard via the HP Labs Research Library. Downloaded on May 25, 2009 at 05:31 from IEEE Xplore. Restrictions apply.
Figure 1. RBS for SPPM Applications
down the accesses to the synchronization variables and eval-
uated the gain of using hardware registers for SPPM appli-
cations on CMPs. To dedicate a set of memory locations
for the evaluation of hardware registers, we wrote a device
driver and mapped it to the memory. Then, we accessed
the device as if we were accessing the hardware registers.
The following pseudo-code shows a simple consumer im-
plementation in SPPM applications. The data is partitioned
into blocks; a data block is read in each loop iteration.
Before reading a data block, the consumer has to check
whether the synchronization window has been violated or
not. The synchronization window is the interval between
the producer’s current position and consumer’s current po-
sition in the shared data array.
for i = 0 to N do
while synchronization window for dataBlock[i] is violated
do
wait
end while
read dataBlock[i]
generate results
end for
The condition to check the synchronization window vi-
olation includes accessing the synchronization variables.
Since the data block size consumed per loop may exceed the
L1 cache size, the synchronization variables will be evicted
from the L1 cache under LRU eviction mechanism. There-
fore, we assume an optimal miss rate of 15% and estimate
the average access time to the synchronization variables to
be 3 cycles on a CMP system with L1 cache of 1-cycle la-
tency and L2 cache of 12-cycle latency, as shown below:
average access time = L1 latency + miss rate
∗ (L1 latency + L2 latency)
= 1 + 0.15 ∗ (1 + 12)
≈3
The cost of implementing RBS is the addition of shared
hardware registers with full/empty bits to the system. The
latency of accessing those registers would be the same as
 Figure 2. Execution Behaviors of Conventional Approach and Prepushing
accessing any register on the system. In SPPM applica-
tions, the number of hardware registers needed to replace
synchronization variables is three; two for the producer and
consumer positions, and one for the flag that resets the posi-
tions when both threads are done at the end of each iteration.
3.2. Data Communications via Prepushing
Data forwarding has been shown to reduce communica-
tion latency and miss rates, as discussed in Section 2. The
previous studies mostly involve distributed shared memory
multiprocessors and directory cache coherence protocols.
In this paper, we present prepushing to facilitate data trans-
fers between a single producer-consumer pair. Our work
brings the data forwarding idea (i.e. sending data before it
is requested) to CMPs with private L1 and L2 caches, and
studies the effects of several prepushing models to improve
the performance of producer-consumer applications.
Figure 2 illustrates the execution behavior of a single
producer-consumer pair, comparing the conventional ap-
proach to prepushing. Normally, data is pulled by the con-
sumer rather than pushed by the producer. Each data block
consists of several cache lines. As the consumer accesses
a data block, it issues cache line requests and must wait or
stall while each block is retrieved from the remote cache. If
each cache line is consumed in less time than it takes to get
the next one, prefetching will not be fully effective at mask-
ing the remote fetch latency. Furthermore, prior prefetched
cache lines may still have been in use by the producer, so a
prefetcher may add to the coherence traffic overhead rather
than reduce it. On the other hand, prepushing allows the
producer to send the data as soon as it is done with it. There-
fore, the consumer receives the data by the time it is needed.
In the conventional approach, two operations per cache line
are seen on the bus, while only one operation per cache line
is seen with prepushing. The prepushing approach reduces
the number of data requests, the number of misses, and the
communication latency seen by the consumer.
We do not consider process/thread migration due to a
context switch, because SPPM applications assign each
Authorized licensed use limited to: Hewlett-Packard via the HP Labs Research Library. Downloaded on May 25, 2009 at 05:31 from IEEE Xplore. Restrictions apply.
thread to a specific processor. This is done to prevent nec-
essary data from being evicted from the cache. Our simu-
lations take place on a Solaris 9 platform, so we use pro-
cessor bind function to assign each thread to a specific pro-
cessor. To evaluate the prepushing approach, we extended
the GEMS Ruby memory model. We added a prepusher as
a hardware component integrated with the cache controller,
and experimented with MOESI broadcast cache coherence
protocol. There is a prepush queue that contains informa-
tion about data that will be forwarded from the producer to
the consumer. In addition, new events, actions, and state
transitions are added to the cache controller to describe the
prepushing behavior. In general, the cost of adding a pre-
pusher to the system is considerably small, as it would be
implemented by extending the cache coherence protocol.
The prepushing approach is well suited to SPPM appli-
cations with tightly synchronized producer-consumer pairs.
Software hints are inserted into the SPPM applications,
and hence the prepushing approach is software controlled.
When the producer is done with a block of data, it signals
the prepusher to initiate data movement by providing in-
formation such as data block address, data block size, and
destination. The prepusher determines which cache lines
comprise the data block and calculates the physical address
of each cache line. Then, messages containing information
about cache lines are placed into the prepush queue. The
cache controller reads the prepush queue and handles the
data transfers in shared or in exclusive mode, based on the
prepush request type. The data is prepushed to the L1 cache
or the L2 cache of the consumer, depending on the prepush-
ing model. On the consumer side, if a tag exists for a par-
ticular prepushed cache line, it is directly written to the ex-
isting location. Otherwise, it is written to a newly allocated
location, which may require a cache line replacement. We
explored four different prepushing models:
PUSH-S-L1: The prepush policy is to send each cache
line to the consumer in shared state as soon as it is pro-
duced, while the producer’s cache line assumes owned state.
This sharing behavior allows the producer to reuse the data,
if necessary. The placement policy puts the prepushed
 Figure 3. Red Black Solver Synchronization Window
cache line into the L1 cache, which may include evicting
cache lines from the L1 to the L2 cache.
PUSH-X-L1: The prepush policy is to send each cache
line to the consumer in exclusive state as soon as the pro-
ducer no longer needs it, thus invalidating it in the pro-
ducer’s cache. This behavior is similar to migratory shar-
ing [7]. It improves performance when the consumer needs
to write to the cache line, as it eliminates the invalidate re-
quest to make the cache line exclusive. The placement pol-
icy is the same as PUSH-S-L1.
PUSH-S-L2: The prepush policy is the same as PUSH-
S-L1. The placement policy is to write the prepushed cache
line to the L1 cache if it is not full, otherwise, writing the
cache line to the L2 cache, evicting cache lines, if needed.
PUSH-X-L2: The prepush policy is the same as PUSH-
X-L1, while the placement policy is the same as PUSH-S-
L2.
The implementation must also handle the race condition
when the consumer issues an explicit cache line request be-
fore the prepushing of that particular cache line takes place.
After the prepusher calculates the cache line address, it
checks the global request table to see if the corresponding
cache line has been explicitly requested. If so, prepush-
ing of that cache line gets canceled. If this condition is not
handled, the cache line will be sent to the remote cache,
incurring extra network traffic.
4. Benchmarks
We used three hand-coded producer-consumer applica-
tions as benchmarks for our research. Temporal parallelism
is exploited by running the producer and consumer concur-
rently. The producer fetches data it needs from the mem-
ory, generates its results, and updates the data in the cache.
While the data is still in the cache, it is fetched and used by
the consumer. The producer and consumer need to be syn-
chronized to prevent the producer from getting far ahead of
the consumer. Therefore, there is a synchronization window
between the producer and consumer to make producer’s
data available in the cache for consumer’s access.
Authorized licensed use limited to: Hewlett-Packard via the HP Labs Research Library. Downloaded on May 25, 2009 at 05:31 from IEEE Xplore. Restrictions apply.
Red Black (RB) Solver: The RB solver solves a partial
differential equation using a finite differencing method. It
uses a two-dimensional grid, where each point is updated
by combining its present value with those of its four neigh-
bors. To avoid dependences, alternate points are labeled
red or black, and only one type is updated at a time. The
values are updated until convergence is reached. The appli-
cation is restructured into a producer-consumer pair where
the producer does the red computation while the consumer
does the black.
Figure 3 illustrates the RB solver grid structure and the
producer’s and consumer’s positions in shared and exclu-
sive prepushing models. For each point in the current row,
the previous and next rows are needed. In exclusive pre-
pushing models, the synchronization window between the
producer and consumer is larger. The producer cannot
prepush the row it has just finished processing (i.e. cur-
rent row), because exclusive prepushing invalidates the data
needed for the next row, so, the producer has to prepush
the previous row. This larger synchronization window does
not degrade performance; in fact, it reduces the consumer’s
remote cache requests as discussed in Section 5.
Finite Difference Time Domain (FDTD): FDTD is an ex-
tremely memory-intensive electromagnetic simulation. It
uses six three-dimensional arrays, three of which constitute
the electric field and the other three constitute the magnetic
field. During each time step, the magnetic field is updated
using values of the electric field from the previous time step.
Then, the electric field is updated using values of the mag-
netic field computed in the current time step. The applica-
tion is restructured into a producer-consumer pair where the
producer does the magnetic field update while the consumer
does the electric field update.
ARC4 Stream Cipher (ARC4): ARC4 is the Alleged RC4,
a stream cipher commonly used in protocols such as the
Secure Sockets Layer (SSL) and Wired Equivalent Privacy
(WEP) in wireless networks. The encryption process uses a
pseudo-random number generator to generate a keystream
of pseudo-random bits. As each byte of the keystream is
generated, it is XORed with the corresponding byte of the
 CPU 2 GHz, UltraSPARC III+ processors
L1 Cache 64 KB, 2-way associative, 1 cycle
L2 Cache 1 MB, 16-way associative, 12 cycles
Cache Line Size 64 B
Memory 4 GB, 120 cycles
Operating System Solaris 9
Table 1. Simulation Environment
plaintext to generate a byte of the ciphertext. Because the
generation of each byte of the keystream is dependent on the
previous internal state, the process is inherently sequential
and, thus, non-parallelizable using conventional data paral-
lelism models. By treating the keystream generation and
the production of ciphertext as producer and consumer, we
exploit the inherent concurrency.
In our simulations, we used unmodified SPPM producer-
consumer code as our baseline. Since SPPM performs
better than prefetching in these applications, we compare
against SPPM. We added software hints to the applications
to command the prepusher to do data transfers. These in-
structions include the address of data to be prepushed, its
size, destination, and prepush mode (either shared or exclu-
sive). We use GEMS Ruby memory model as our simula-
tion platform. The GEMS Ruby simulator uses an in-order
processor model. The simulated system is a dual-processor
SPARC machine with characteristics shown in Table 1. The
prepush latency used in our simulations includes the latency
to fetch the data from the L1 cache and the network latency
to transfer the data to the destination’s cache. The system
did not use shared L2 cache because we modeled it after
AMD’s dual core machines and used the MOESI broadcast
cache coherence protocol like the Opteron.
5. Performance Results
Red Black (RB) Solver: The RB solver uses a two-
dimentional grid of doubles, so a grid size of 200x200 fits
in the L2 cache while 400x400 does not. Table 2 shows
the total execution time taken by a producer-consumer pair
in each iteration, the number of synchronization point ac-
cesses, and the estimated time taken for them. The aver-
age time to access synchronization variables is 3 cycles (see
Section 3). When we multiply average access time by num-
ber of accesses per iteration, we obtain estimated access
time per iteration. The ratio of estimated access time per it-
eration to the execution time per iteration gives us the gain
from RBS. The results show that the gain of using hardware
registers for synchronization is 2-5% per iteration.
All the results shown in our graphs are normalized to
SPPM. Figure 4 shows the total execution time for all three
benchmarks. In RB solver, there is a 20-39% improve-
ment in shared prepushing and 40-62% in exclusive pre-
Authorized licensed use limited to: Hewlett-Packard via the HP Labs Research Library. Downloaded on May 25, 2009 at 05:31 from IEEE Xplore. Restrictions apply.
Figure 4. Normalized Execution Time
pushing. The shared prepushing models take slightly more
time due to higher exclusive data requests as illustrated in
Figure 6(b). The reason is because the consumer needs
to receive the data exclusively, but, the shared prepushing
models leave the data in shared state. Therefore, the con-
sumer has to make explicit exclusive data requests.
Figure 5(a) illustrates the consumer’s L1D cache misses
for all three benchmarks. In RB solver, the number of con-
sumer L1D misses decreases by 47-48% when the data is
prepushed to the consumer’s L1 cache. There is no im-
provement in PUSH-S-L2 and PUSH-X-L2 because the
consumer cannot find the required data in its L1 cache and
has to retrieve it from its L2 cache. This is much better than
retrieving the data from either the remote cache or memory.
The number of consumer’s L2 misses decreases by 93-98%
in all prepushing models as shown in Figure 5(b).
Figure 6(a) and Figure 6(b) shows the consumer’s shared
and exclusive requests, respectively, for all three bench-
marks. In RB solver, the shared requests are reduced by 93-
98%. Since the consumer receives the data beforehand, the
number of requests is significantly reduced. As expected,
the number of exclusive requests is much lower in PUSH-X-
L1 and PUSH-X-L2, due to invalidations of the producer’s
data. The exclusive requests stay the same in the shared
prepushing models because the consumer has to issue in-
validation requests, thus causing a negligible increase in the
execution time in the shared prepushing models.
To avoid the race condition discussed in Section 3, the
prepusher checks to see whether the consumer has explic-
itly requested a cache line or not, before pushing it to the
consumer. If so, the prepusher cancels the prepushing of
that particular cache line. Our simulation results show that
the prepusher works very effectively in all benchmarks, al-
Grid Size 200 x 200 400 x 400
Exec. Time per Iter. (cycles) 2,056,725 13,905,891
Access Count per Iter. 13,531 211,933
Est. Access Time per Iter. 40,593 635,799
Table 2. Red Black - RBS Results
 (a) L1D Misses (b) L2 Misses

Figure 5. Consumer’s Normalized Cache Misses

(a) Shared Data Requests (b) Exclusive Data Requests

Figure 6. Consumer’s Normalized Remote Cache Requests

most achieving the ideal outcome. Due to space limitations,
we cannot provide our results here.
FDTD: The FDTD application uses six three-
dimensional grids of doubles, so the grid size of 20x20x20
fits in the L2 cache while the other grid sizes don’t. Table 3
shows the register-based synchronization results. The gain
of using hardware registers is 6-11% per iteration.
The execution time is reduced by 19-21% in the
20x20x20 case, in which the data is prepushed both ways
because the data size is so small that the caches do not get
polluted. In the other cases, there is only one-way prepush-
ing and the improvement is 26-29% in shared prepushing
and 46-49% in exclusive prepushing. The improvement in
exclusive prepushing is due to the reduced number of re-
mote cache requests, as illustrated in Figures 6(a) and 6(b).
The consumer’s L1D misses are reduced by 18% for
20x20x20, and 42-48% in larger data sizes (see Figure 5(a)).
Again, PUSH-S-L2 and PUSH-X-L2 provide no improve-
ment because the consumer always misses its L1 cache but
hits in its L2 cache. The number of consumer’s L2 misses
decreases by 69-73% in all the prepushing models.
In the 20x20x20 version, shared requests are reduced by
69-74%. There is an improvement of 72-74% in larger data
sizes (see Figure 6(a)). As for the exclusive data requests,
the 20x20x20 version and shared prepushing models do not
show any improvement. However, there is a 76% reduction
in the exclusive prepushing models because there is no need
to request the data in exclusive mode (see Figure 6(b)).
ARC4: Table 4 shows the register-based synchroniza-
tion results. Neither stream size fits in the L2 cache. The
gain of using this approach is negligible, as the producer
and consumer rarely wait for each other.
The execution time is reduced by 16-17% by prepush-
ing. Figure 5(a) shows the number of consumer L1D misses
decreasing by 15-17% when the data is prepushed to the
consumer’s L1 cache. Again, there is no improvement in

Grid Size 30x30x30 40x40x40 Stream Size 10 MB 50 MB

Exec. Time per Iter. (cycles) 5,741,480 8,124,822 Exec. Time per Iter. (cycles) 1,327,413 1,330,287
Access Count per Iter. 120,425 291,506 Access Count per Iter. 56 72
Est. Access Time per Iter. 361,275 874,518 Est. Access Time per Iter. 168 216

Table 3. FDTD - RBS Results Table 4. ARC4 - RBS Results

Authorized licensed use limited to: Hewlett-Packard via the HP Labs Research Library. Downloaded on May 25, 2009 at 05:31 from IEEE Xplore. Restrictions apply.
 PUSH-S-L2 and PUSH-X-L2 because the consumer cannot
find the required data in its L1 cache. The number of con-
sumer L2 misses decreases by 17% in all prepushing mod-
els. Figure 6(a) shows the consumer shared requests are
reduced by 20% by prepushing. However, there is no im-
provement in the consumer exclusive requests (Figure 6(b)).
6. Conclusion
Current CMPs provide no explicit synchronization and
communication support for multithreaded applications. The
register-based synchronization and prepushing techniques
proposed in this paper aim to improve that, particularly for
CMPs with private caches. The register-based synchro-
nization approach employs hardware registers to improve
performance and help power savings, while the prepush-
ing approach provides an efficient communications inter-
face where data can be moved/copied from one cache to
another before it is needed at the destination. Both ap-
proaches are particularly beneficial for producer-consumer
parallelism. Register-based synchronization shows promis-
ing results to reduce spin waits, improve resource utiliza-
tion, and help power savings. Prepushing significantly re-
duces cache misses and remote cache requests while elimi-
nating latency seen by the consumer processor. Our future
work will focus on register-based synchronization and pre-
pushing implementations on different architectures, as well
as on further architecture optimizations.
References
[1] H. Abdel-Shafi, J. Hall, S. V. Adve, and V. S. Adve. An Eval-
uation of Fine-Grain Producer-Initiated Communication in
Cache-Coherent Multiprocessors. In International Sympo-
sium on High-Performance Computer Architecture, Febru-
ary 1997.
[2] M. C. August, G. M. Brost, C. C. Hsiung, and A. J. Schif-
fleger. Cray X-MP: The Birth of a Supercomputer. IEEE
Computer, 22(1):45–52, 1989.
[3] G. T. Byrd and M. J. Flynn. Producer-Consumer Commu-
nication in Distributed Shared Memory Multiprocessors. In
Proceedings of the IEEE, March 1999.
[4] M. Fillo, S. W. Keckler, W. J. Dally, N. P. Carter, A. Chang,
Y. Gurevich, and W. S. Lee. The M-Machine Multicomputer.
In International Symposium on Microarchitecture, 1995.
[5] S. Frank, H. B. III, and J. Rothnie. The KSR1: Bridging the
Gap Between Shared Memory and MPPs. In IEEE Com-
puter Society Computer Conference, February 1993.
[6] M. Gschwind, H. P. Hofstee, B. Flachs, M. Hopkins,
Y. Watanabe, and T. Yamazaki. Synergistic Processing in
Cell’s Multicore Architecture. IEEE Micro, 26(2):10–24,
2006.
[7] A. Gupta and W.-D. Weber. Cache Invalidation Patterns
in Shared-Memory Multiprocessors. IEEE Transactions on
Computers, 41(7):794–810, 1992.
Authorized licensed use limited to: Hewlett-Packard via the HP Labs Research Library. Downloaded on May 25, 2009 at 05:31 from IEEE Xplore. Restrictions apply.
[8] C. N. Keltcher, K. J. McGrath, A. Ahmed, and P. Conway.
The AMD Opteron Processor for Multiprocessor Servers.
IEEE Micro, 23:66–76, 2003.
[9] D. Koufaty and J. Torrellas. Comparing Data Forward-
ing and Prefetching for Communication-Induced Misses in
Shared-Memory MPs. In International Conference on Su-
percomputing, 1998.
[10] D. A. Koufaty, X. Chen, D. K. Poulsen, and J. Torrellas.
Data Forwarding in Scalable Shared-Memory Multiproces-
sors. In International Conference on Supercomputing, July
1995.
[11] D. Lenoski, J. Laudon, K. Gharachorloo, W.-D. Weber,
A. Gupta, J. Hennessy, M. Horowitz, and M. S. Lam. The
Stanford DASH Multiprocessor. IEEE Computer, 25(3):63–
79, March 1992.
[12] P. S. Magnusson, M. Christensson, J. Eskilson, D. Fors-
gren, G. Hallberg, J. Hogberg, F. Larsson, A. Moestedt, and
B. Werner. Simics: A Full System Simulation Platform.
IEEE Computer, 35(2):50–58, February 2002.
[13] M. M. K. Martin, D. J. Sorin, B. M. Beckmann, M. R. Marty,
M. Xu, A. R. Alameldeen, K. E. Moore, M. D. Hill, and
D. A. Wood. Multifacet’s General Execution-driven Multi-
processor Simulator (GEMS) Toolset. In SIGARCH Com-
puter Architecture News, September 2005.
[14] E. Rosti, E. Smirni, T. D. Wagner, A. W. Apon, and L. W.
Dowdy. The KSR1: Experimentation and Modeling The
KSR1: Experimentation and Modeling of Poststore. In ACM
SIGMETRICS Conference on Measurement and Modeling of
Computer Systems, May 1993.
[15] S. L. Scott. Synchronization and Communication in the T3E
Multiprocessor. In International Conference on Architec-
tural Support for Programming Languages and Operating
Systems, 1996.
[16] D. M. Tullsen, J. L. Lo, S. J. Eggers, and H. M. Levy.
Supporting Fine-Grained Synchronization on a Simultane-
ous Multithreading Processor. In International Symposium
on High Performance Computer Architecture, January 1999.
[17] S. Vadlamani and S. Jenks. The Synchronized Pipelined Par-
allelism Model. In International Conference on Parallel and
Distributed Computing and Systems, November 2004.
[18] S. Vadlamani and S. Jenks. Architectural Considerations for
Efficient Software Execution on Parallel Microprocessors.
In International Parallel and Distributed Processing Sym-
posium, March 2007.
[19] T. F. Wenisch, S. Somogyi, N. Hardavellas, J. Kim, C. Gni-
ady, A. Ailamaki, and B. Falsafi. Store-Ordered Streaming
of Shared Memory. In International Conference on Parallel
Architectures and Compilation Techniques, 2005.