Modeling Rack and Server Heat
Capacity in a Physics Based Dynamic
CFD Model of Data Centers
Sami Alkharabsheh, Bahgat Sammakia
10/28/2013
2
ES2 Vision
To create electronic systems that are self
sensing and regulating, and are optimized for
energy efficiency at any desired performance
level
Project Vision
Toward a full physics-based experimentally verified 3D
computational fluid dynamics model for data centers
3
Outline
 Introduction
 Physics Based Steady State Baseline Model
 CRAC model
 Server model
 Tile model
 Dynamic Model- Server Heat Capacity Effect
 Server level model
 Room level model
 Case studies
 Conclusions and Future Work
 EPA (2007):1.5 % of total U.S.
electricity consumption in 2006.
(Total cost of $4.5 billion)
 Datacenter Dynamics (2012)
Global Census : power
requirements grew by 63% globally
to 38 GW from 24 GW in 2011.
14%
12%
10%
8%
6%
4%
2%
0%
2.5<=
2.4-2.49
2.3-2.39
2.2-2.29
2.1-2.19
2.0-2.09
1.9-1.99
1.8-1.89
1.7-1.79
1.6-1.69
1.5-1.59
1.4-1.49
1.3-1.39
1.2-1.29
1.1-1.19
1.09>=
Introduction
Response percentage
4
PUE
M. Stansberry and J. Kudritzki, “Uptime Institute 2012 Data Center
Industry Survey,” Uptime Institute, 2013.
Others
 Uptime Institute 2012 Data Center
Industry Survey: PUE>1.8 for more
than 55% of data centers
HVAC
Cooling
IT
M. Iyengar and R. Schmidt, “Energy Consumption of Information
Technology Data Centers”, 2010.
5
Nature of Problem
Cooling
Power
Fromtimes.com
treehugger.com
• In real time, cooling is
difficult to control due to
long lag times
• Complexity of transport
in data centers
• Overprovisioning is
commonly used for safe
operation
Solutions for improving
the energy efficiency
in data centers have been
isolated
• Performance is not
proportional to power
• Server overprovisioning is
a common practice
System-level and holistic
solutions are a MUST
6
Bench Mark Numerical Model
CRAC
Raised Floor
Rack
Perforated tile
Parameter
Room size
Value
6.05 m x 13.42 m x 3.65 m
Plenum depth
0.6 m
Tile perforation ratio
50%
Perforated tiles area
0.61 m x 0.61 m
CRAC fan speed
100%
7
CRAC Model
 Based on manufacturer data
 Liebert 114D CW
3.5
3
2.5
CRAC internal resistance
 The CRAC model is calibrated
such that the flow rate can be
predicted accurately at
different operating pressures
Emersonnetworkpower.com
Static pressure (in. H2O)
 Operating fan curve is obtained
from the manufacturer, Liebert
Consulting
Calibrated operating point
2
1.5
1
Uncalibrated operating point
0.5
0
0
0.5
1
1.5
2
2.5
3
Flow rate (CFM)
* Alkharabsheh et al. “Utilizing Practical Fan Curves in CFD Modeling of Data Centers,” SEMITHERM2013.
3.5
4
4
x 10
8
Server Model
Flow bench
apparatus
 A standard testing procedure
following the AMCA 210-99
guidelines are used to measure
the pressure fan curves
2U server for
testing
 9 RU server simulators (load
banks) and a 2 RU commercial
server are tested
250
2 RU server
9 RU load bank
 The measured fan curves include
the internal resistance of the
server
Static pressure (Pa)
200
150
100
50
0
 The measured fan curve can be
imbedded directly into the CFD
-50
0
0.05
0.1
0.15
0.2
0.25
3
Flow rate (m /s)
0.3
0.35
0.4
0.45
9
Tile Model
 The CFD tile model is
validated using experimental
data in Schmidt et al.*
Computerfloorpros.com
0.1
0
1
0.1
Airflow rate (m3/s)
 The CFD tile model is
modified to compensate for
the momentum loss in the
CFD flow resistance model
-0.1
2
0
-0.1
0.1 1
2
4
5
6
7
8
9
10
11
12
13
14
15
3
4
5
6
7
8
9
10
11
12
13
14
15
4
5
6
7
8
9
10
11
12
13
14
15
4
5
6
7
8
Tile
9
10
11
12
13
14
15
Row C
2
0
-0.1
3
Row B
0
-0.1
0.1 1
 The CFD tile model is able to
capture the tile flow
distribution and can be used
in room level simulations
Row A
3
Row D
1
2
3
Solid line: experimental data, Dashed line: CFD results
*Experimental data: Schmidt et al, “Measurements and Predictions of The Flow
Distribution Through Perforated Tiles in Raised-Floor Data Center,” InterPACK2001
10
Steady State Room Level Simulations
 In addition to affecting the
power dissipation, the servers
power scenario also the
airflow pattern by operating
15 kW/ rack
 The room can be
underprovisioned/
overprovisioned based on the
servers power level
20 kW/ rack
* Alkharabsheh et al. “Numerical Steady State and Dynamic
Study in a Data Center Using Calibrated Fan Curves for CRACs
and Servers,” InterPACK2013
32 kW/ rack
15
17.5
20.5
24
28.1
32.9
38.5
45
Temperature
(C)
 Several parametric studies
can be conducted using this
model
11
Simple Dynamic Model
 Complete CRAC failure
simulated at 20 seconds
110
No backup power
Blower backup power
100
Inlet temperature (degC)
 The thermal capacity of the
equipment is not taken into
account
90
80
70
60
Critical temperature
50
Failure
40
30
20
 Supporting the CRAC blower
with backup power provides
the room with extra cooling
and time that can be
utilized in increasing the
reliability of operation
0
10
20
30
40
50
Time (s)
60
70
80
90
Unused plenum cold air
* Alkharabsheh et al. “Numerical Steady State and Dynamic Study in a Data
Center Using Calibrated Fan Curves for CRACs and Servers,” InterPACK2013
12
Server Heat Capacity
6
T server
 The server level CFD model is
developed based on the lumped
mass approximation
5.5
Exp. data [*]
CFD model
5
4.5
4
3.5
3
100
200
300
400
500
600
700
800
900
Time (s)
5
4.75
4.5
T server
 Experimental data is used to
calibrate and validate the server
level CFD model
0
Exp. data [*]
CFD model
4.25
4
3.75
3.5
3
0
100
200
300
400
500
4
3.5
3
700
800
No HC
1% Cap.
10% Cap.
50% Cap.
100% Cap.
120% Cap.
150% Cap.
4.5
*Ibrahim et al., "Thermal Mass Characterization of a Server at Different Fan Speeds,"
ITHERM2012.
600
Time (s)
5
T server
 An increase in the rate of change
in temperature is observed at low
values of heat capacity until
instantaneous change in
temperature is noticed when
server heat capacity is completely
neglected
3.25
0
100
200
Time (s)
300
400
13
Room Level Model
Blanking
panel
Mounting
rails
 The detailed rack model is
capable of hosting the server
model, blanking panels,
leakage through the mounting
rails, and internal supports
n=20
Server
n=2
n=1
Detailed rack model
 Each server consists of an
experimentally characterized
fan curve and thermally
calibrated heated mass
 Each rack is populated with
twenty of the 2 RU servers
CRAC
Raised
Floor
Rack
Perforated
tile
14
 It is assumed that all the servers
inside the modular data center are
shutdown at time 20 seconds
Power (kW/rack)
Case I: Servers Shutdown
 Three different room level models
are compared in this transient
analysis
0
0
20
Time (s)
1
Rack inlet tempeature
 Including the servers heat capacity
is crucial in dynamic modeling.
However, the heat capacity of the
rack chassis can be neglected
without affecting the accuracy of
the results and reducing the
computational time
20
No HC
Servers HC Only
Servers & Racks HC
0.8
0.6
0.4
0.2
0
0
500
Where:
1000
1500
Time (s)
T  Tss
Tˆ 
To  Tss
2000
2500
15
Case II: Server Power Short Pulses
15
Power (kW)
 Fluctuations in the
dissipated power is
simulated in the form of 30
second pulses
 The temperature increases
immediately in the model if
we ignore the heat capacity
30
120180
1000
Time (s)
1
Rack A1 inlet temperature
 The heat capacity damps
down the effect of short
duration power fluctuations
on the inlet temperatures
10
0.8
0.6
0.4
Temperature without HC
Temperature with HC
0.2
0
0
200
400
600
Time (s)
800
1000
16
Conclusions and Future Work
 Experimentally validated models of different data center
components are developed
 A steady state and dynamic, physics based, room level CFD
model for a bench mark data center is developed
 It is found that the heat capacity of the servers affects the
rate of change in temperature significantly
 The effect of rack frames heat capacity is found to be small
and can be neglected in room level simulations
 Future work will include adding cooling unit heat capacity
17
Acknowledgement
This material is based upon work supported by the National
Science Foundation under Grants No.1134867 and CNS-1040666