CFD for Data Centers

Siddharth Premkumar Cad Technology Center

SM4760-V
With IT manufacturers improving the performance of datacom equipment year after year, the need for accurate design
of data centers have become crucial to ensure safe and efficient design of data centers. Based on the ASHRAE
publication for case studies on "High Density Data Centers" this class aims to look at some of the most common
ventilation strategies employed in actual existing data centers to understand how to model and set up these data
centers for CFD analysis and validate the results with actual published data. This class will also look at an example to
compare remodels of data centers based on the ASHRAE publication and would thereby help designers, engineers
and project managers gain more insight into the design and operation of data centers.

Learning Objectives
At the end of this class, you will be able to:

Understand the different ventilation strategies being employed in data centers

Compare results for a data center model with published data

Understand modeling techniques for raised access floor data centers

Compare different designs

About the Speaker

Siddharth Premkumar (Sid) is an Applications Engineer at CAD Technology Center, Inc. and provides training and
consulting to clients and helps them to get the most out of their software products. Sid has done his Masters in
Mechanical Engineering from Syracuse University and has experience with CFD and FEA programs.
siddharthp@cadtechnologycenter.com

CFD for Data Centers

Ventilation strategies being employed in data centers

The need for CFD Analyses on Data Centers
The following are some of the reasons to consider doing CFD analysis on a data center model:
IT manufacturers increasing performance at the cost of increasing heat dissipation
CFD analysis can reduce cost of retrofits
CFD can allow exploring design alternatives
CFD analysis can help locate hot spots
Let us now explore some of the common ventilation strategies employed in data centers as shown in the table below:
Ventilation Strategy
Raised access floor supply
Raised access floor supply with AHUs on the floor below
Raised access floor supply/ceiling return

Raised access floor supply with heat exchangers

Non raised access floor with or without ceiling return

Description
Air enters through perforated tiles located on a raised floor
and the racks are typically arranged in a hot aisle/cold
aisle layout.
In this strategy, the air handling units (AHUs) are located
on the floor below and so it allows all the mechanical
equipment to be grouped together.
This strategy reduces mixing of the warm and cold air and
the higher return air temperature encourages efficient
operation of the CRACs. There could also be a ducted
return used in this case which would further reduce the
chances of mixing.
This strategy can be employed to reduce the overall
temperature of the space by removing some of the heat
from the racks.
This ventilation technique has the CRACs and server
racks located on the same level

The figures below show images of these ventilation strategies as described in the ASHRAE book on High Density Data
Centers-Case Studies and Best Practices, 2008.

Raised access floor supply

Raised access floor supply with AHUs below

CFD for Data Centers

Raised access floor with ceiling return

Raised access floor with heat exchangers

Raised access floor with modular CRACs in hot aisle

Raised access floor with ducted return

Non-raised access floor with ceiling supply

CFD for Data Centers

Compare results for a data center model with published data

Non-raised access floor data center
In this case study, we will look at the case study for the Cedars-Sinai Medical Center data center as published in the
book High Density Data Centers-Case Studies and Best Practices, 2008 and look at the way it was modeled, meshed
and analyzed.

Material Assignment
The image below shows a snapshot of the case study. As indicated in this image, the non-raised access floor data
center has an area of approximately 600 sq.ft.. Also, the data center model has 13 fluorescent lighting fixtures which we
ignore in our analysis.

Image Courtesy of: ASHRAE, High Density Data Centers Case Studies and Best Practices, 2008

CFD for Data Centers

Referencing the image shown below, as published in the case study, we were able to create the model in Revit.

Image Courtesy of: ASHRAE, High Density Data Centers Case Studies and Best Practices, 2008
The figure below shows the model in Revit.

Also shown above is an add-in that can be used to launch the Revit model directly into Autodesk Simulation CFD.
Once this is done, the model can be assigned materials in Simulation CFD. The following assumptions were made for
the materials used:

CFD for Data Centers

Component
Walls
Ceiling
Floor
Server racks
Storage equipment
CRACs
Fire suppression equipment
PDU
UPS
BAT
Console and networking equipment

Material
Concrete
Concrete
Concrete
Aluminum
Aluminum
Aluminum
Aluminum
Aluminum
Aluminum
Aluminum
Aluminum

Shown in the figure above is the tab that can be used to assign materials. Also on the left of the image, notice that
aluminum and concrete is basically the material that is assumed for the various components of the data center. The air
volume is automatically generated as long as we have an air-tight volume in our Revit model. The properties for the air
volume can be set to constant or variable depending on how the model needs to be used. In our case, we have
assumed that this is an interior room and constant properties for the air volume would be used.

Boundary Conditions (BCs)

CFD for Data Centers

Once the material assignment is done, we apply BCs based

data provided for the Cedars- Sinai Medical Center data
The sources of heat generation were assigned a volume BC
the sources of air flow were assigned a surface BC. Also,
like the CRACs that did not contribute to the heat transfer
suppressed from the analysis to reduce the number of
generated during meshing and also so that the airflow BC can
applied correctly. We will use the labels as shown in the figure
identify the various equipment in the room.

Equipment Type
CRAC1
CRAC2
CRAC3
CRAC4
CRAC5
CRAC6
CRAC7
CRAC8
CRAC9
CRAC10
CRAC11
CRAC12
Servers
Storage
PDU
UPS
Networking
BAT
Console
Humidifier

Color

on the
center.
whereas
volumes
were
elements
be
below to

The images below show the heat dissipation and flow conditions as measured as part of the case study:

CFD for Data Centers

Image Courtesy of: ASHRAE, High Density Data Centers Case Studies and Best Practices, 2008
The table below lists the boundary conditions used from the information used above:
Volume
Boundary Condition
Storage/Networking equipment
2335.8333 W each
CRACs
253.333 W each
Servers
4444.5 W each
PDU/UPS
1337 W each
Surface
Exit Boundary Condition
CRAC 1
886.667 CFM, 63 F
CRAC 2
886.667 CFM, 65 F
CRAC 3
886.667 CFM, 61 F
CRAC 4
335 CFM, 62 F
CRAC 5
335 CFM, 61.5 F
CRAC 6
335 CFM, 62 F
CRAC 7
335 CFM, 62.5 F
CRAC 8
258 CFM, 63 F
CRAC 9
258 CFM, 68 F
CRAC 10
258 CFM, 63 F
CRAC 11
258 CFM, 62 F
CRAC 12
258 CFM, 61 F
Surface
Inlet Boundary Condition
CRAC 1
72 F
CRAC 2
72 F
CRAC 3
72 F, 0 gauge pressure
CRAC 4
72 F
CRAC 5
72 F
CRAC 6
72 F
CRAC 7
72 F
CRAC 8
72 F
CRAC 9
72 F
CRAC 10
72 F, 0 gauge pressure
CRAC 11
72 F
CRAC 12
72 F
The zero gauge pressure should ideally be applied to all the inlet surfaces; however, it was observed that doing this led
to divergence in the solution procedure. Hence, the zero pressure BC was only applied to two CRAC units.
Another assumption that has been made with regard to the boundary conditions, is that there is no need for a film
coefficient to be applied to the wall, roof and floor surfaces since the room is assumed to be completely surrounded by
other rooms that have similar thermal conditions and so there would not be any heat transfer across these surfaces.

CFD for Data Centers

Meshing
The next step is the meshing process. In this case study, we performed the automatic meshing process with the default
settings. The image below shows the meshed model.

As can be seen in the image above, the mesh is fairly coarse and you can see the nodes as the cyan colored points.

Solve
After the meshing is done, we can go into the solution module and define the physics we need to include
in our analysis. We are going to include flow and heat transfer in the model and also include the effects of
radiation. We perform the analysis over 300 iterations and the results are as shown below:

CFD for Data Centers

Results
As part of the case study, temperature measurements were taken at the rack locations as shown in the
image below:

Image Courtesy of: ASHRAE, High Density Data Centers Case Studies and Best Practices, 2008
Also, the image below shows the actual measurements of temperatures at the rack locations referenced
in the image above:

Image Courtesy of: ASHRAE, High Density Data Centers Case Studies and Best Practices, 2008

10

CFD for Data Centers

With this information, we can use planes in Simulation CFD to plot temperatures in front of the rack and
we can compare the temperatures between our model and the actual model as shown below:

Unit number

Actual temperature

1
2
3
4
5
6
7
8
9
10
11
12
13

63.7
69.3
67.3
62.6
63.1
65.2
68.2
64.2
64.2
63.8
64.9
64.8

Simulation CFD
temperature
69.36
68.15
62.31
62.65
62.44
62.6
64.61
68.45
68.04
67.48
65.72
62.31
62.27

Difference
-5.66
1.15
4.99
-0.05
0.66
0.59
-0.25
-3.84
-3.28
-1.92
2.59
2.53

11

CFD for Data Centers

The average difference between the CFD results and the actual measured results is 2.29 F.

12

CFD for Data Centers

Understand modeling techniques for raised access floor data

centers
Raised Access Floor
For the purpose of this case study, we will look at the data center located at The Georgia Institute of
Technology. This model was created in Revit with a total area of 1600 sq.ft based on the description for
the data center in the case study:

]
Image Courtesy of: ASHRAE, High Density Data Centers Case Studies and Best Practices, 2008

13

CFD for Data Centers

The image below gives an idea of how to model this data center. The 12 racks shown in the center portion
of the model are the server racks; the remaining 4 surrounding units represent the storage, management
and networking equipments.

Image Courtesy of: ASHRAE, High Density Data Centers Case Studies and Best Practices, 2008

14

CFD for Data Centers

The figure below shows the model as it was created in Revit.

The model mainly includes CRAC units and server racks. These units were modeled as shown in the
figure below:

15

CFD for Data Centers

Material Assignment
The image below shows the materials that were assigned to the various parts of the model at a glance.

The perforated floor was modeled as a resistance material with a 0.25 free area ratio since the case study
required the floors to have 25% open tiles. An air volume was used inside the CRAC unit for the boundary
conditions to be applied. For the server racks, an internal fan/blower material was used with a flow of
1530 CFM based on what was measured for the case study. Also a slip factor of 0.3 was assumed which
the recommended value for axial fans.

Image Courtesy of: ASHRAE, High Density Data Centers Case Studies and Best Practices, 2008

16

CFD for Data Centers

Boundary Conditions
Based on the planned cooling requirements for this data center, the following boundary conditions were
applied to the following pieces of equipment.

Image Courtesy of: ASHRAE, High Density Data Centers Case Studies and Best Practices, 2008

As shown in the figure above (on the left), a volumetric flow rate going into the CRAC units was applied to
simulate the return flow of air and a zero gauge pressure was applied to the third CRAC unit to again
simulate the return flow of air.
The image on the right shows a volumetric flow rate and temperature boundary condition that was applied
to the CRAC at its exit to simulate the air flowing into the room. The values for the flow rate and
temperatures were based on what was measured for the case study as indicated below:

17

CFD for Data Centers

Image Courtesy of: ASHRAE, High Density Data Centers Case Studies and Best Practices, 2008
The image above also shows that there are no boundary conditions applied on one of the CRAC units
(CRAC 38) and this is in accordance with the table shown above.

Meshing

The image above shows the results of the automatic mesh sizing process. Notice how the program
automatically determines areas where the mesh needs to be refined and areas where it can use a coarser
mesh. Another point to note is that the air volume inside the CRAC units (the volume to which the

18

CFD for Data Centers

boundary conditions have been applied) has been suppressed in order to omit them from the analysis. If
this were not done, the program does not recognize the inlets and outlets for the air flow.

Solution
Once the model has been assigned materials, boundary conditions and meshed, we are in a position to
run the simulation. Since we are interested in the flow and heat transfer behavior, we have both modes
turned on. Also we are interested in studying the effects of radiation. So with these conditions applied, we
run the simulation. With the default values set for automatic convergence assessment, we obtain
convergence in 737 iterations as shown in the convergence plot below:

19

CFD for Data Centers

Compare Different Designs

In the previous example, the data center was modeled to have a total surface area of 1600 sq.ft. However
due to excessive noise and air discomfort and large areas of under-utilized floor space, a decision was
taken to remodel the data center. The new design would incorporate a rear-door air-to-water heat
exchanger that would replace the rear panel of the computer racks as shown below:

Image Courtesy of: ASHRAE, High Density Data Centers Case Studies and Best Practices, 2008
So in order to compare the designs, we created a new design study in the same model and applied a heat
generation on the internal fan volume by taking into consideration the heat extracted by the heat
exchanger.

20

CFD for Data Centers

The images below show the remodeled design for the data center:

Image Courtesy of: ASHRAE, High Density Data Centers Case Studies and Best Practices, 2008
Also highlighted in the image above, are some of the benefits that were predicted for the new design. We
shall later see how we can validate these predictions.
The modeling technique including material assignment, boundary conditions and meshing is similar to the
previous example. The model is as shown below:

21

CFD for Data Centers

As can be seen from the figure above, the remodeled design (design 2) had a total surface area of 1000
sq.ft. In the remodeled design the 12 racks are arranged in the middle portion with the storage,
networking and management racks located along the corners.

22

CFD for Data Centers

Results
The images below show the velocity profile in the under-floor for the two designs.

In this case, the maximum velocity in the under-floor is about 762 CFM.

23

CFD for Data Centers

With the remodeled design, the maximum velocity in the under-floor reaches about 2272 CFM just as
predicted by the case study! Further, summary planes can be created to compare the two designs sideby-side and the static pressure in the under-floor could be compared. The static pressure for design 1
was 0.29 psf. as against 1.53 psf. for design 2, again in agreement with the prediction for the data center
remodel. Another consequence of the new design was that the temperature in the under-floor was higher
(69 F) compared to the previous design (55 F).

Another prediction made was that the new design would reduce the possibility of recirculation in the
model due to the presence of the wall. The image below for Design 1 shows that there is significant
amount of recirculation in this design.

24

CFD for Data Centers

We can also cut a plane above the raised floor and use a summary image to compare the temperature
distribution among the two designs:

The image above shows that the maximum temperature in Design 1 is about 137 F. For the same plane in
Design 2, we can see in the image below that the maximum temperature hovers in the late 70 F range.

25

CFD for Data Centers

Thus using CFD analysis, we were in a position to compare the two design alternatives for this data
center model and conclude that Design 2 is the better alternative due to:
Higher static pressures in the under-floor region
Higher velocity magnitudes in the under-floor region
Lower temperatures above the floor
Negligible recirculation of warm air above the perforated floor

26