Session Title:
Demystifying Efficiency in the Data Center
Utilizing Airflow as a System
Presented By:
Jon deRidder
Enabled Energy
Learning Objectives:
• Identify how to improve your Power Usage
Effectiveness immediately
• Design an efficient airflow system in a data center
and apply to your own facilities
• Measure and verify the savings achieved in efficient
data centers
• Identify ASHRAE TC 9.9 and its effect on the
ecosystem of the data center
Background
Defining the terms:
•
Data center
–
The room (regardless of size, age, how anyone “feels” about it, the budget that you have [or had] to build or maintain it, or how
reliable it is / is not) that houses your computing equipment.
Background
Defining the terms:
•
Data center
•
Computing equipment
–
Server, network, or storage devices that compute, transport, and store information (data).
Background
Defining the terms:
•
Data center
•
Computing equipment
•
PUE
–
Power Usage Effectiveness. Taking the total facility power (feeding your data center) and dividing by your IT load (UPS load will
get you close) will give you your PUE. This PUE number will be greater than 1 (hopefully less than 3) and provides a “uniform”
way of calculating how much power is going to your IT load vs. how much power you are consuming to accomplish your compute
(the “tax”). A PUE of 2 is typical in a “legacy center”, while a PUE of 1.5 is “typical” for a new data center build (many are now
becoming very aggressive i.e. 1.1 and 1.2).
PUE = 3.0
IT Equipment
33%
Air Movement
16%
Electricity
Transformer/
UPS
14%
Lighting, etc.
4%
Cooling
33%
Courtesy of a very sad data center experience
Background
Defining the terms:
•
Data center
•
Computing equipment
•
PUE
•
A tax
–
Something you pay because you are forced to or because you are not aware of it.
Background
Defining the terms:
•
Data center
•
Computing equipment
•
PUE
•
A tax
•
ASHRAE TC 9.9
–
–
–
The American Society of Heating, Refrigeration, and Air Conditioning Engineers Technical Committee 9.9 brought together many
hardware manufacturers, locked them in a room, and came up with the latest “Thermal Guidelines for Data Centers”. This is a
GUIDELINE NOT A STANDARD.
Your equipment warrantee is provided by your equipment manufacturer and ultimately this is who gets to decide if you are or
are not “compliant” with housing the equipment in a “proper environment”. Examples: power quality; temperature and
humidity controls; particulate type; and size.
In the end data wins…he or she with the most information is likely going to be the person who controls how, when, who, where,
and why.
Background
Defining the terms:
•
Data center
•
Computing equipment
•
PUE
•
A tax
•
ASHRAE TC 9.9
•
Reliability
–
–
The resulting investment of many painstaking strategy sessions (brain cells) coupled with lots of redundant components (which
translates to big dollars) allowing for the concurrent maintainability of your entire infrastructure (planned maintenance to avoid
system downtime).
Hope is not a strategy!
Background
Defining the terms:
•
Data center
•
Computing equipment
•
PUE
•
A tax
•
ASHRAE TC 9.9
•
Reliability
•
Efficiency
–
–
–
–
An aggressive pursuit (and an exhausting effort after achieving the appropriate levels of redundancy) to achieve maximum
throughput with minimal restriction and waste.
This starts with doing the best you can with what you have, but working intently and diligently to make it better.
Please note that reliability is and must be first.
Walnuts can be opened with steamrollers, but they don’t need to be and the result isn’t pretty.
Background
Defining the terms:
•
Data center
•
Computing equipment
•
PUE
•
A tax
•
ASHRAE TC 9.9
•
Reliability
•
Efficiency
•
Problem
–
Opportunity
Self
Actualization
Esteem
Social
Safety
Physiological
Background
Defining the terms:
•
Data center
•
Computing equipment
•
PUE
•
A tax
•
ASHRAE TC 9.9
•
Reliability
•
Efficiency
•
Problem
–
Opportunity
Optimization
Efficiency
Reliability
Strategy
Communication
Airflow - A Systems Approach
Cause:
Effect:
Meaningful metrics are needed for the data center.
PUE and CUE are now metrics the industry is accepting as “standard” and yet
these are not universally understood or defined.
Cause:
Effect:
Delivery systems were developed around outdated guidelines.
Dramatic overcooling of IT equipment!
ASHRAE TC 9.9 published new thermal guideline for data centers
(~78.6˚F at the intake of compute equipment).
Cause:
Airflow delivery systems are generally unbalanced and full of air-mixing
opportunities.
Typical delivery systems have >50% “bypass” airflow.
Effect:
Airflow - A Systems Approach
It all starts with
 Organization
 Distribution
It falls apart with
 Poor communication
 Bad strategy
Discover
Your PUE
• Calculate how much you are spending now on the system and each part that creates
the total.
Your CRAC/CRAH efficiency
• Start with the intake temperature of your server, network & storage equipment.
• Then calculate the efficiency of your CRAC/CRAH units
–
(CFM * delta temperature) * .9 = BTUs of accomplished cooling.
The path for your airflow
• Supply path
–
–
–
•
Supply panels
Aisle layout (hot/cold)
Opportunity for recirculation
Return path
–
What is the path of least resistance?
Most Valuable Investment (MVI)
Bypass Airflow
Source: UpSite
Most Valuable Investment (MVI)
Forward-curved blades use blades that curve in the
direction of the fan wheel's rotation. It has 24 to 64
shallow blades with both the heel and tip curved
forward. Air leaves the impeller at velocities greater
than the impeller tip speed. Tip speed and primary
energy transferred to the air is the result of high impeller
velocities and operating most efficiently at lowest speed.
Backward-curved blades use blades that curve against the
direction of the fan wheel's rotation. The blades are single
thickness with 9 to 16 blades inclined away from the
direction of rotation. Air leaves the impeller at a velocity
less than its tip speed. Relatively deep blades provide
efficient expansion with the blade passages. The backward
curvature mimics that of an airfoil cross section and
provides good operating efficiency with relatively
economical construction techniques. Backward-curved
fans are much more energy efficient than forward curved
fans. The EC Fan design moves the air in more of a
straight line.
EC Fan
Forward Curved Fan
Reduced
AIR
FLOW
AREAS
17”
Most Valuable Investment (MVI)
Under-Floor Baffle
Most Valuable Investment (MVI)
Proportional Distribution Tiles
Sealing Cable Cutouts
Most Valuable Investment (MVI)
Blanking Openings in Cabinets
Most Valuable Investment (MVI)
Containment
Most Valuable Investment (MVI)
Ducting CRACs to Drop-Ceiling Air Space
Most Valuable Investment (MVI)
Optimized
Thank You!
7X24 Fall Meeting
Airflow secrets revealed
Tom Weiss
President C2
October 23, 2012
Presentation goals
 Provide baseline for analyzing effective airflow
cooling in a data center
 Provide financial measurements for airflow
efficiency
 Share common problems and their source
 Provide information on effective CFD
 Answer questions regarding airflow
Let’s start with the
science
Air cooling is a method of dissipating
Heat!
It works by making the object to be
cooled have a larger surface area or an
increased flow of air over its surface; or
both!
Problems with some
raised floor cooling!
Advice from Eaton
 “As much as 30 to 60 percent of the data center
utility bill goes to support cooling systems. If that
figure seems too high, it is.”
 “Poor airflow management reduces both the
efficiency and capacity of computer room cooling
equipment.”
An easy fix?
…. “optimize the existing cooling system
through mechanical and room layout
changes, using relatively inexpensive
devices to redirect and concentrate
available airflow”.
What can we fix?
 Cold Aisle / Hot Aisle Containment
 Curtains, blanking panels
 Close the holes in the floor
We still haven’t determined what happens to the
air once it leaves the floor.
So how does the cold aisle really work?
CFM vs.
Usable CFM
Traditional Measurements
 CFM – Cubic Feet Per Minute
 Static Pressure and open space
Performance questions
 How much air are we wasting?
 How does the air flow out of the tile?
Do we need more air or do we need to be more
efficient with the air we have?
How should it work?
The three components of cold aisle
airflow efficiency:
 Cool the upper servers
 Flow to the servers
 Come out of every section of the tile
 No back flow into the floor!
Stratification heat!
Industry white paper

Tile mixing!
Air is like water!
 Bypass the servers
 Entrainment
 Pollute the return air
 Room mixing
Stratification
Short cycle / Jet stream
950 CFM - Venturi
Wasted air = wasted
energy + hot spots
 “Only 28% of the air, in a traditional raised floor
system, cools the servers (72% does not cool)”
 “2.6 times more cooling than is necessary and yet we
still have hot spots”
 Lower set points
 More CRAC units
 Higher energy cost
Data from Uptime Institute Dr. Bob F. Sullivan and
Kenneth G. Brill – “24 by Forever”
How expensive?
Cooling costs are 50% of the total bill!
“Data center managers can save 4 percent in
energy costs for every degree of upward
change in the set point, according to Mark
Monroe, the Director of Sustainable Computing
at Sun Microsystems (JAVA)”
Data Center costs
 OPEX – Energy cost  40%!
 Asset refresh costs –
 Server failure 20% higher!
 CAPEX– Crac units, more devices!
 PUE – 2.0 or higher!
What is going on?
CEETHERM / NSF study
 Measure relationship of airflow to rack inlet
 Analyze air at the particle level
 Analyze CFD models
 Provide feedback to the industry
5
4
Tile/Rack Level Air Flow Modeling
05/04/2012
5
5
Rack and Tile Geometry
42 U
0.09m.
3 1/2"
0.44m.
1'-5 1/2"
10 U
0.44m.
1'-5 1/2"
10 U
Fan speed setting
dial
1.98m.
(6'-6")
10 U
0.44m.
1'-5 1/2"
10 U
0.44m.
1'-5 1/2"
Perforated floor
tiles with dampers
0.53m.
1'-9"
Server Simulator
Pressur
e Outlet
56
Details of Tile geometry
Pressur
e Inlet
Tile top
3030=900 pores,
0.5”0.5”,
Porosity=39%
88
Fan
Grill
Tile
4million
cells
Symmetr
y
Mass Flow
Inlet
Damper
s
49=36 pores,
1.25”5.25”,
Porosity=41%
2084
5
7
Details of Rack Geometry
12inch
18inch
Fan
18inch
Fan
18inch
Fan
18inch
Fan
Wall
Server
Simulator
1
Wall
Server
Simulator
2
Pressure outlet
Side View
5inch Wall
1inch
1ft
Symmetry
Front View
(aisle)
Front View (fan)
Front View
(grill)
0.5inch
3inch
6inch
open
area
1inch
wall
Grill
Wall
Server
Simulator
3
Wall
Server
Simulato
r4
Pressure outlet
Pressure
inlet
hub
Pressure
inlet
(K=10)
wall
fan
Aisle
Aisle
9inch
Tile top
1.5inch
Tile top
Dampers
Dampers
Plenum
24inch
Symmetry
Mass flow inlet
Plenum
Symmetry
24inc
h
Mass flow inlet
9.25inch
18inch
• Fan: Target mass
flow rate boundary
5
8
Current CFD Models for Tile Flows
• Porous jump model [1]
1
2
∆𝑃 = 𝐾 × 𝜌𝑉𝑖𝑛
1
1−𝐹
𝐹2
2
𝜌 = 𝑑𝑒𝑛𝑠𝑖𝑡𝑦
𝐾=
𝐹 = 𝑝𝑜𝑟𝑜𝑠𝑖𝑡𝑦 =
2
Low Pressure
Uniform Velocity
1
2
2
+ 1−𝐹
𝑂𝑝𝑒𝑛 𝐴𝑟𝑒𝑎
𝑇𝑜𝑡𝑎𝑙 𝐴𝑟𝑒𝑎
• Body force model [2]
 As mass is conserved hence
𝑉𝑖𝑛 momentum is under
∆𝑀 = 𝜌𝐴𝑡𝑖𝑙𝑒 𝑉𝑖𝑛 ×
− 𝑉𝑖𝑛
accounted:
𝐹
mass flow
𝐴
velocity velocity in
through porous jump
model
pores
𝑉

𝐹
𝑆𝑥 = 𝑆𝑜𝑢𝑟𝑐𝑒 𝑡𝑒𝑟𝑚 𝑓𝑜𝑟 "x" 𝑚𝑜𝑚𝑒𝑛𝑡𝑢𝑚 𝑒𝑞𝑢𝑎𝑡𝑖𝑜𝑛
 = 𝑉𝑜𝑙𝑢𝑚𝑒 𝑜𝑓 𝑚𝑜𝑚𝑒𝑛𝑡𝑢𝑚 𝑠𝑜𝑢𝑟𝑐𝑒 𝑟𝑒𝑔𝑖𝑜𝑛
𝐴𝑡𝑖𝑙𝑒 = 𝐴𝑟𝑒𝑎 𝑜𝑓 𝑡𝑖𝑙𝑒
P
Tile
(F)
Uniform Velocity
Vin
High
Pressure
Velocity Profile
S
x
X
𝑉
𝑡𝑖𝑙𝑒 𝑖𝑛
 Momentum
source 𝑖𝑛term
𝑆𝑥 =
− 𝑉𝑖𝑛specified just above the tile:
Uniform Velocity
[1] Patankar, S. V., Airflow and Cooling in a Data Center, Journal of Heat Transfer, 2010, Vol. 132, pp. 073001-1-17.
[2] Abdelmaksoud, W. A., Khalifa, H. E., Dang T. Q., Elhadidi, B., Schmidt, R. R., Iyengar, M., Experimental and Computational
Study of Perforated Floor Tile in Data Centers, Intersociety Conference on Thermal Phenomena (ITHERM), Jun 2-5, 2010, Las
Rack Flow = 2594 CFM
Tile Flow = 0 CFM
(0% of Rack Flow)
Tile Flow = 0% of Rack Flow
Aisle top
Velocity
(m/s)
Rack
Aisle center
Rack
Aisle center
Aisle top
Tile
Tile
PIV
CFD
[ref] Kumar, P., Joshi, Y., Experimental Investigations on the Effect of Perforated Tile Air Jet Velocity on Server Air Distribution in a
High Density Data Center, Intersociety Conference on Thermal Phenomena (ITHERM), Jun 2-5, 2010, Las Vegas, USA.
Tile Flow = 20% of Rack Flow
Aisle top
Velocity
(m/s)
Tile
PIV
Aisle center
Rack
Aisle center
Aisle top
Rack
Rack Flow = 2594 CFM
Tile Flow = 496 CFM
(~20% of Rack Flow)
Tile
CFD
[ref] Kumar, P., Joshi, Y., Experimental Investigations on the Effect of Perforated Tile Air Jet Velocity on Server Air Distribution in a
High Density Data Center, Intersociety Conference on Thermal Phenomena (ITHERM), Jun 2-5, 2010, Las Vegas, USA.
Tile Flow = 60% of Rack Flow
Aisle top
Tile
PIV
Aisle center
Rack
Aisle center
Aisle top
Rack
Rack Flow = 2594 CFM
Tile Flow = 1598 CFM
(~60% of Rack Flow)
Tile
CFD
[ref] Kumar, P., Joshi, Y., Experimental Investigations on the Effect of Perforated Tile Air Jet Velocity on Server Air Distribution in a
High Density Data Center, Intersociety Conference on Thermal Phenomena (ITHERM), Jun 2-5, 2010, Las Vegas, USA.
Tile Flow = 100% of Rack Flow
Aisle top
Rack
Aisle center
Aisle center
Aisle top
Rack
Rack Flow = 2594 CFM
Tile Flow = 2594 CFM
(100% of Rack Flow)
Tile
Tile
PIV
CFD
[ref] Kumar, P., Joshi, Y., Experimental Investigations on the Effect of Perforated Tile Air Jet Velocity on Server Air Distribution in a
High Density Data Center, Intersociety Conference on Thermal Phenomena (ITHERM), Jun 2-5, 2010, Las Vegas, USA.
Rack Flow = 2594 CFM
Tile Flow = 2594 CFM
(100% of Rack Flow)
Tile Flow = 100% of Rack Flow
Tile
Tile
Aisle center
Aisle top
Rack
Aisle center
Rack
Rack
Aisle center
Aisle center
Tile
Aisle top
Rack
Aisle top
Aisle top
Tile
Resolving
Body Force
Porous
Tile
Model
Jump
[ref] Kumar, P., Joshi, Y., Experimental InvestigationsGeometry
on the Effect of Perforated Tile Air
Jet Velocity on Server Air Distribution in a
Model
PIV
High Density Data Center, Intersociety Conference on Thermal Phenomena (ITHERM), Jun 2-5, 2010, Las Vegas, USA.
Let’s end with the
science
Air cooling is a method of dissipating Heat!
 Object to be cooled have a larger surface area –
 Freeze the room = $
 Or an increased flow of air over its surface –
 Flow to the servers = higher set point savings!
 Or both –
 Flow of cool, non – polluted air, through the rack!
Questions?
Thank you!
<Insert Picture
Here>
RMDC Cooling System Adjustments
Kevin Donnelly
Building Engineer
67
68
Example of Title
Extending to Two Lines
• First Level Bullet
– Second level bullet
• Third level bullet
– Fourth level bullet
• Fifth level bullet
69
Program Agenda Example
•
•
•
•
•
•
Our Understanding of XYZ
Capabilities and Value Drivers
Benefits & Assessments
Oracle Solutions
Oracle Credentials
Appendix
<Insert Picture Here>
70
<Insert Picture Here>
Appendix
71
Temperature and Humidity Adjustment Results
•
•
•
•
•
•
Invested $31K for installation of plenum return ductwork.
Server inlet temperature reduced from 80 to 70 degrees
Number of CRAC units humidifying from 11 to 1
Mechanical cooling energy reduced 155 KW (24%)
Annual energy savings of $53K
Payback 7 months
72
73
74
Other Adjustments
• Seal air gaps (Koldlok)
• CRAC temperature setpoint 72 to 76 degrees
• Condenser water loop setpoint 75 to 65
degrees
75
76
77
78
Questions?
79