Energy- and Thermal-Aware Design of

Many-core Heterogeneous Data Centers

David Atienza, Luca Benini, Edouard Bugnion, Babak Falsafi, Marcel

Ledergerber, Fabrice Roudet, Patrick Segu, Lothar Thiele and John
Thome
 Computers Have Been Key in Our Society
From pure computing-intensive centric to data-centric
Information Age Era: connected and ubiquitous access
Communication
Era
Information
Age Era
1970s- 1990s 2000s Today+
Mainframes

PC Era
2
 Data Centers are Key in the Information
Age Era to Keep Up with Big Data Benefits
Data center consists of thousands of computing servers
Store, process, and serve user data on behalf of billions
Era of knowledge economy and Big Data in science
50% of economic value in developed countries [Economist]

Gene Financial
Sequencing Simulations

Weather Commerce
Prediction

Life data Medical

Data centers Analytics

New computing era to transform data into value for

society (but it must be at minimal energy cost) 3
 But Data Centers Energy Trends Are Not
Sustainable
280
240 Datacenter A modern data center (DC)
200 Electricity Demands
Billion Kilowatt

In the US
160 [Energy Star]
120
hour/year

80
[IDC]: Mega DCs is
40
70% of all DCs in
0
2018
Density per rack: 8kW-15kW
2001 2005 2009 2013
2017
Modern data centers increase power demands 20 MW
Power is beginning to clearly dominate costs in data center management
In Switzerland, 3-4% of all electricity, growing at >20%
Swiss industry is heavily based on services and requires significant IT
support
Change energy increase trend, how to flatten it?
4
 YINS Approach: System-Level Energy-Efficient Integration

Multi-scale system engineering to cover complete data center re-design

Balanced industry-academia participation: 3 large Swiss-based industrial partners
Evaluation of proposed approaches with experimental prototypes

TIK-ETHZ

DCSL-EPFL ESL-EPFL

PARSA-EPFL
LTCM-EPFL
IIS-ETHZ

5
 New Ultra-Adaptive Server Technology
Standard technologies + regular digital design do not enable energy-proportionality
today
Built first multi-core ever in RVT fully depleted Si-on-insulator (FD-SOI) [DATE15]
Characterization prototype in large range of frequencies (0.5 to 1.0) and temperatures (-40
to 120C)
log(mW) 28nm 28nm FD-SOI, 4-core low-power
server processor

40nm

1GOPS/mW

log(GOPS)

Goal: reduce bending up of More energy-efficient server (50% less energy), and
Servers energy curve at high able to compensate for performance variation at high
6
performance temperature
 Energy-Scalable Server Architectures and Networks

Coherence
core core ... core Coherence domain 2
domain 1

LLC NUMA
fabric
Memory Remote Memory
Controller Controller 300 ns round-trip latency
to remote memory

Scale-Out Non-Uniform Mem. Architect. (SO-NUMA):

Shared-memory architecture for racks
Low latency and bandwidth 2x faster
QoS Limit Application services not requiring peak
processing
Scale-out loads can operate at 200-500 MHz
Our new near-threshold (FDSOI) servers
VM loads can operate at less than 1 GHz
enable energy scalability and low network
enable energy scalability and low network
latency: no work done, no energy
consumed! 7
 Revolutionary Energy-Free Mini-Thermosyphon Cooling

Beyond state-of-the art: Active liquid cooling vs.

air fans
Built low-height thermosyphon prototype
15-cm prototype proves rack compatibility
Mini-thermosyphon demonstrator built
Effective use of gravity to remove heat
[ITHERM16]:
Validated approach for a 2U server blade,
design done for HP blade (donated by Yahoo):

80% less energy,

Temp. balanced
at server level
(less failures)

8
 Integrated Datacenter Control for Existing DCs
with Renewable Energy Sources
Multi-objective controller for geo-distributed data centers
Global phase
Global phase: Virtual Machines (VMs) clustering (VMs clustering)
Local phase: VMs allocated using CPU-load and data correlation
Green controller: on-line manager of renewable energy sources
Local phase Local phase
Significant improvements in cost, energy and perf. (VMs (VMs
Up to 65% in energy and 35% in costs ($$$) allocation) allocation)
DC 1 DC N

Green Green
controller controller
Scalable approach tested with more
than 12000 servers distributed in 3
data centers of Credit Suisse and Brain
Serve setups 9
 Rack Management and Smart PDU with Eaton: Clear Industrial Impact

Installation at Eaton facilities in Le Lieu

Real-time monitoring and management
Reduce peak temp. on multi-rack system
Developed rack management compatible
with Eatons servers and racks
products
Evaluations with larger IT setups on the way

Up to 80% energy savings in latest generation

of server racks in data centers
10
 Conclusion
Energy-efficient data centers are of strategic importance to Switzerland
75% of the Swiss economy (especially banking industry) are service-driven
3-4% of the Swiss energy is devoted to datacenters, growing at a rate of 20%/year

YINS: Developed a radically new thermal-aware design approach for

next generation energy-efficient data centers, key results:
Novel adaptive server technology (50% less energy at chip level)
Optimized server architecture and OS customization (energy-scalable servers)
Micro-thermosyphon cooling demonstrator (80% less energy in cooling, stable
temp.)
Experimental validation of new multi-rack and datacenter power management in
Credit Suisse datacenters with Eaton support (up to 80% energy savings in IT
overall )
11
Thank you for your
attention!
Questions?

12