COMPUTER ROOM AIR HANDLER (CRAH), SUB-FLOOR AIRFLOW ANALYSIS
FOR DATACENTERS
A Thesis
Presented to the faculty of the Department of Mechanical Engineering
California State University, Sacramento
Submitted in partial satisfaction of
the requirements for the degree of
MASTER OF SCIENCE
in
Mechanical Engineering
by
Brian Andrew Barrie
FALL
2013
© 2013
Brian Andrew Barrie
ALL RIGHTS RESERVED
ii
COMPUTER ROOM AIR HANDLER (CRAH), SUB-FLOOR AIRFLOW ANALYSIS
FOR DATACENTERS
A Thesis
by
Brian Andrew Barrie
Approved by:
__________________________________, Committee Chair
Dongmei Zhou
__________________________________, Second Reader
Ilhan Tuzcu
____________________________
Date
iii
Student: Brian Andrew Barrie
I certify that this student has met the requirements for format contained in the University format
manual, and that this thesis is suitable for shelving in the Library and credit is to be awarded for
the thesis.
__________________________, Graduate Coordinator ___________________
Akihiko Kumagai
Date
Department of Mechanical Engineering
iv
Abstract
of
COMPUTER ROOM AIR HANDLER (CRAH), SUB-FLOOR AIRFLOW ANALYSIS
FOR DATACENTERS
by
Brian Andrew Barrie
Data center energy efficiency is becoming a huge topic of research as more IT
infrastructure moves from local resources into the cloud. New data centers designed with energy
efficiency in mind have achieved impressive results, but legacy datacenters (10-20 years old)
make up a significant portion of the total data center space in the U.S. These sites can benefit the
most from having an independent energy audit done with the focus on using new technologies to
save substantial energy while maintaining reliability. One of these new technologies is wireless
mesh sensor networks. Wireless mesh sensor networks allow large amounts of sensors to be
deployed without running expensive power and communication cabling to the sensors. This
technology allows a very granular view of the pressure and temperature conditions in real time on
the data center floor. This large amount of real time data also provides a factor of safety when
changes are made to the cooling system in the quest for energy savings. Because IT loads in data
centers change frequently, it is difficult to match cooling capacity with the heat load in the room
and as a result, there is usually too much or too little cooling in the data center. Estimating how
much energy may be saved depends on determining the factors that affect the cooling efficiency
of the data center.
v
This study seeks to determine how physical variables such as floor depth, CRAH
arrangement, fan types, fan speed, perforated tile placement and IT equipment layout impact subfloor pressure distribution. If the subfloor pressure is evenly distributed the fan speeds can be run
lower, resulting in increased data center cooling efficiency.
In the study thousands of wireless sensors are installed throughout a data center as part of
an environmental monitoring and control solution for a large telecommunications company.
These wireless devices are capable of monitoring the sub-floor pressure, subfloor temperature,
CRAH supply and return temperatures and server rack inlet temperatures at three different
heights. When changes are made to the fan speed settings, or supply air temperature, the wireless
sensors will capture the changes to the sub-floor pressure and server rack inlet temperatures.
The data from the wireless sensors is time stamped and sent wirelessly every 5 minutes
back to a central server. This server uses a MySQL database to organize and store all the sensor
data. This data can then be accessed and exported for analysis. Once the impacts of these
variables have been established, optimal energy savings improvements can be made and are
provided in the study.
__________________________________, Committee Chair
Dongmei Zhou
__________________________________, Second Reader
Ilhan Tuzcu
____________________________
Date
vi
TABLE OF CONTENTS
Page
List of Tables ........................................................................................................................ viii
List of Figures ...........................................................................................................................ix
Chapter
1. INTRODUCTION TO DATACENTERS ... …………………………………………….. 1
1.1 Datacenter Equipment and Construction…………………………………………1
1.2 Purpose of the Experiments ................................................................................... 6
2. DATACENTER EFFICIENCY BACKGROUND ............................................................ 8
2.1 Review of Research ............................................................................................... 8
2.2 Experimental Method.............................................................................................. 9
3. ANALYSIS OF THE DATA ............................................................................................ 19
3.1 Verifying the Fan Law for Pressure………………………………………….….20
3.2 Verifying the Fan Law for Power…………………………………………….…22
3.3 Region of Influence of a CRAH Unit……………………………………..…….25
3.4 Uneven Pressure Distribution………………………………………………...…31
4. FINDINGS AND INTERPRETATIONS………………………………………………..35
5. CONCLUSIONS AND FUTURE WORK………………………………………………..38
Appendix A. Liebert CRAH Unit Data Sheets .................................................................... 39
References .............................................................................................................................. 40
vii
LIST OF TABLES
Tables
Page
3.1
Verifying the Pressure Fan Law…………………………………..……………... 21
3.2
Verifying the Power Fan Law ......................…………………………………..…. 23
3.3
Rounded Power Data ..........................………….…………………………………. 24
4.1
Typical CFM Vs. Pressure for a Common Datacenter Perforated Tile ………..37
viii
LIST OF FIGURES
Figures
Page
1.1
Typical Raised Floor Data Center Layout ……… .………………………………. 2
1.2
Inside a typical CRAH unit with Centrifugal supply fans .. ……………………. 3
2.1
Data Center Number One Floor Plan………..…….…………………………………. 11
2.2
Data Center Number Two, 1st Floor Plan……….……… ……………………....…… 13
2.3
Data Center Number Two, 2nd Floor Plan……………………………………………14
3.1
Datacenter 2 floor 1 - CRAH Units at 60% and Baseline Pressure Sensor……27
3.2
Datacenter 2 floor 2 - CRAH Units at 60% and Baseline Pressure Sensor.…..28
3.3
Change in Pressure Vs. Linear Distance……………………………………….30
3.4
% Change in Pressure Vs. Linear distance………………………………...….30
3.5
Uneven Pressure Distribution, Datacenter 1 Floor 1 ………………………………....33
3.6
Uneven Pressure Distribution, Datacenter 1 Floor 2 ………………………….…....…34
ix
1
1. INTRODUCTION TO DATACENTERS
In 2011 the EPA reported that datacenters consumed 3% of all electrical power generated
in the United States1. Many datacenters built in the last 5 years, especially those built by
large tech companies such as Google, Facebook and Yahoo, are incredibly energy
efficient and have attained LEED gold and platinum certification. Most datacenters are
over 10 years old and were not built with energy efficiency in mind. The EPA estimates
that over 23 billion kWh of electricity can be conserved, just by improved operation of
existing facilities1.
The goal of this study is to determine some energy savings opportunities in existing
legacy datacenters by taking advantage of state-of-the-art wireless sensor technology.
Wireless sensors have the advantage of being incredibly cheap to install, which makes it
possible to deploy thousands of sensors in a datacenter. This allows a very detailed look
at the environment in that datacenter and allows energy saving changes to be made
safely, without the risk of overheating IT equipment.
1.1 Datacenter Equipment and Construction
Modern datacenters come in many shapes and sizes and contain a wide assortment of
equipment serving many types of applications. At their most basic, datacenters can be
broken down into two types; raised floor cooling air delivery or overhead air delivery.
Figure (1) shows a typical raised floor data center layout, with a hot and cold aisle
arrangement of equipment.
2
Figure (1.1) Typical Raised Floor Data Center Layout
Source: Datacenterknowledge.com
This paper will be focusing on raised floor data centers with cold air supplied by
computer room air handlers (CRAHs). CRAHs have chilled water running through their
evaporator coils supplied by a central chiller plant. This is in contrast to a CRAC or DX
unit that generally uses some type of compressed refrigerant. A typical water cooled
CRAH with centrifugal fans is shown in figure (1.2).
3
Figure (1.2): Inside a typical CRAH unit with Centrifugal supply fans
Source: Liebert.com
The cold air supplied by the CRAHs makes its way to the servers by placing perforated
tiles in front of them, which allows the cold air into the room from the subfloor. A data
center that is older than 15 years and that is configured in this way is commonly referred
to as a ‘legacy’ datacenter and that is how this configuration will be addressed in this
paper.
4
There are two approaches to calculating the amount of cooling needed for a data
center. The first approach balances the IT load on the data center floor with the cooling
tonnage provided by the chiller plant. It also takes into account lighting load and losses
through the power delivery systems such as UPS's and PDU's. UPS stands for
uninterrupted power supply and they are typically located in their own rooms away from
the raised floor and have their own dedicated CRAH units for cooling. PDU’s (power
distribution systems) are typically located on the raised floor and are used to distribute
power to the individual racks of servers. PUE also assumes that all IT load, voltage
conversion losses and lighting are converted directly to heat. This approach is best
represented by a popular equation called PUE or Power Usage Effectiveness.
𝑃𝑈𝐸 =
𝐼𝑇 𝑃𝑜𝑤𝑒𝑟+𝐿𝑜𝑠𝑠𝑒𝑠+𝐶𝑜𝑜𝑙𝑖𝑛𝑔 𝑃𝑜𝑤𝑒𝑟+𝐿𝑖𝑔ℎ𝑡𝑖𝑛𝑔
𝐼𝑇 𝑃𝑜𝑤𝑒𝑟
(1.1)
PUE is a useful metric, but it is easy to game the system and achieve a low PUE without
actually saving any energy. For example, what if the server fans did the majority of the
work in creating the pressure difference between the room and the subfloor plenum
instead of the CRAH fans? This can happen in datacenters with cold aisle containment.
This would increase the IT load, while decreasing cooling load, resulting in a lower PUE
number while using the same or more energy.
PUE deals strictly with power and cooling tonnage and does not take into account how
this cooling is physically delivered to the equipment where it is needed. The lower the
5
PUE number the more efficiently the data center is using its cooling power. Some new
high efficiency data centers are achieving a PUE close to 1.2. Looking at equation (1.1),
a low PUE number means that the cooling power, losses and lighting are very small
compared to the IT power. However most legacy data centers fall into the 1.8-2.2 range,
which leaves a lot of room for improvement.
The second approach, which is the approach dealt with in this paper, is interested mostly
in the cold air and the variables that affect its delivery to the equipment. These variables
include square footage of the raised floor, raised floor height, floor pressure, types of air
handlers, number of perforated tiles on the floor, floor leakage and under floor
obstacles. All these factors affect how the cold air reaches the equipment and prevents
engineers from sizing cooling systems based on IT load alone. For example, two 20 ton
air handlers may be able to cool 20 servers spread out on a 10,000 sqft floor, but they
may not be able to cool that same number of servers effectively if they were spread out
over a 50,000 sqft area, even though the heat load is the same. This paper seeks to
quantify the area that a typical air handler can affect
Part of the challenge in trying to quantify the effects these variables have on airflow is
that data centers are mission critical environments and a few minutes of downtime can
cost hundreds of millions of dollars. In addition, no two data centers are alike and even
the equipment on the same floor can vary in age and design. Air handlers can vary from
their manufacturer specifications due to slipping belts, sheaves, worn bearings and
6
variable speed fan drives which adds to the complexity of quantifying how much air they
are capable of delivering. Replacing equipment is costly and often not necessary so
finding ways to improve the cold air delivery to the servers is the easiest and most cost
effective way to improve legacy, data center efficiency.
Because legacy datacenters are often neglected and not managed according to best
practices there are many steps that may be taken to save energy. Adding blanking panels
to server racks that are not full of equipment helps prevent hot aisle air from invading the
cold aisle. Sealing cable cutouts in the backs of equipment racks and removing
perforated tiles that are not in cold aisles or actively cooling equipment prevents the cold
air from being recirculated. Hot or cold aisle containment is costly, but is the best way to
prevent the mixing of hot and cold air. The most effective energy saving improvement
that is commonly made to legacy data centers is to add a variable frequency drive, or
VFD, to the fan motor inside the CRAH units that provide cold air to the sub floor
plenum8. CRAH fans are typically driven by 480V AC motors, which are only capable
of running at a fixed speed. VFDs allow the fan motors in the CRAHs to turn at any
speed in order to provide only the necessary amount of cooling to the datacenter.
1.2 PURPOSE OF THE EXPERIMENTS
This paper seeks to prove three fundamental properties of computer room air handlers.
First that the types of CRAHs commonly found in legacy datacenters behave according to
7
the ideal fan laws. If the fans in the units do not behave according to the fan laws, it will
be difficult to predict power consumption and subfloor pressure when the fan speeds are
reduced.
Second, this paper seeks to quantify the region of influence of a typical CRAH unit.
Because turning down the fan speed of a CRAH unit equipped with a VFD is a common
energy saving measure, it is important to understand how that reduction of fan speed
affects the sub floor pressure and air delivery to the servers.
Third, this paper also investigates how the centrifugal fans commonly used in older
CRAHs impart a horizontal velocity to the air, which causes uneven airflow under the
raised floor. In an ideal situation, the subfloor of a data center acts as a common plenum
and the pressure distribution should be uniform throughout. In real life however, this is
not the case. In addition to leaks in the floor, piles of abandoned cables and uneven
distribution of IT and perforated tiles, the CRAHs themselves impart a velocity to the
discharge air that can cause uneven distribution of pressure.
Once these properties have been verified, then more accurate energy savings estimates
can be made for legacy datacenters. Accurate energy savings estimates are extremely
important because utility rebates in the United States are rewarded on the actual amount
of energy saved. If a company wants to know how much money a VFD retrofit can save
them, then they need to be able to accurately predict what the utility rebate is going to be
and factor that into the cost of the project.
8
2. DATACENTER EFFICIENCY BACKGROUND
The best practice guidelines of datacenter operation; hot/cold aisle configuration, airside
economizers, perforated tile placement, sealed floor penetrations, blanking panels and
containment2,3 are common knowledge to datacenter operators in 2013. However, there
is no requirement that datacenters operators follow these guidelines and indeed many of
them do not. Unless there is an incentive for the operator to save energy, most operators
are more concerned about the reliability of their equipment and achieving 100% uptime,
even at the cost of energy efficiency.
2.1 Review of Research
Much research has been done on airflow in data centers. The two main approaches to
studying airflow in data centers have focused on CFD analysis, wireless environmental
monitoring and direct measurements of airflow using flow hoods, hand held pressure
meters and thermistors, or other similar equipment.
CFD analysis is a predictive technology and while it is very impressive in its granularity,
it is very difficult to verify the results with the same amount of detail. Modeling a system
as comlex as a large datacenter takes a considerable amount of setup and verification11.
The constantly changing nature of datacenters also reduces the effectiveness of CFD
analysis, because every time a rack of servers is added or removed from the datacenter a
new CFD report will have to be created and validated4. CFD is also costly and incredibly
complex and requires detailed modeling of all the equipment on the datacenter floor.
9
However it does have the advantage of showing directionality of airflow11 which is very
useful when deciding where to place new IT equipment. Most of the major datacenter
equipment suppliers (APC, Schneider, Emerson) offer some kind of CFD service to their
clients4.
Flow hoods, handheld pressure meters and laser thermometers are also great tools to
measure real conditions in the datacenter, and to verify CFD results, but they are
intrusive, only provide a single snapshot in time and cannot be left in place for real time
monitoring. This is where wireless sensors shine. They provide as much granularity as
needed, they are not intrusive, they are cost effective enough to be deployed in mass and
they provide real time monitoring. Wireless sensors can also be used as control inputs to
a building management system that actively controls the VFDs to meet the cooling
requirements on the floor8.
2.2 Experimental Method
This study makes extensive use of wireless sensors to gather data from thousands of
wireless sensors placed throughout the datacenter to measure environmental conditions in
real time. Having this much real time data available provides a factor of safety for the
datacenter operator instead of using excess cooling8.
Two different data centers were used for the experiments in this study. Data center
number one shown in figure (2.1) is a 25,000 square foot legacy data center with high
10
floor pressure and excessive cooling capacity compared to the small amount of IT load.
The 21 CRAH units are all located around the perimeter walls and the IT load is mostly
concentrated on the East side of the room. In figure (2.1) the CRAH units are shown
outlined in red and the pressure sensors are shown as green circles. The CRAH units in
this building are all Lieberts, but there is a range of models with different cooling
capacities. There are three FH-376C models with 5 horsepower fans that are rated at 17
tons of sensible cooling and the rest are FH-529C with 7.5 horse power motors rated at
23 tons of sensible cooling. See Appendix A for Liebert CRAH data sheets.
11
Figure (2.1) Data Center Number One Floor Plan
12
Data center number two is a 2-story facility, where the floors are completely separate
from each other. The first floor, shown in figure (2.2), is roughly 40,000 square feet and
features 20 CRAHs with VFDs. The second floor, shown in figure (2.3) is closer to
50,000 square feet and has 24 CRAHs with VFDs. Both floors have plenty of cooling
capacity compared to the heat load, but the IT equipment is very spread out and the
CRAH units are not operating efficiently. The CRAH units, outlined in red, are scattered
haphazardly around both of the rooms. Again, the CRAH units are outlined in red, while
the pressure sensors that were installed for this study are shown as green circles. The rest
of the objects shown on the floor plans are IT equipment of various shapes and sizes.
13
Figure (2.2) Data Center Number Two, 1st Floor Plan
14
Figure (2.3) Data Center Number Two, 2nd Floor Plan
15
At the beginning of the study the CRAH units in each data center had their VFDs running
at 100% fan speed. The VFDs allow for fan speed control of each individual CRAH unit
as well as monitoring of fan speed and kW being used by the fan motor. Each CRAH unit
was outfitted with a wireless sensor in the return and supply air plenum to measure the air
temperature entering and exiting the unit. This configuration allows the temperature
difference across the evaporator to be calculated. The manufacturer supplies the CFM of
the unit at 100% fan speed and if the fan follows the fan laws then it is possible to
calculate the CFM at any fan speed. Knowing the temperature delta ΔT and the CFM of
air flowing across the evaporator the amount of sensible cooling that the unit is doing is
calculated as
𝑞̇ = 𝑚̇𝑎 𝐶𝑝 (∆𝑇)
(2.1)
In datacenter one, twenty-nine pressure sensors were installed evenly throughout the floor
to measure to the static pressure differential between the sub floor plenum and the room.
This works out to less than 900 square feet being covered by each sensor. Or, to put it
another way, each sensor covers about a 16 foot radius. Figure (2.1) shows the layout of
the CRAHs (outlined in red) and the locations of the pressure sensors (green circles).
Every third server cabinet was also equipped with three temperature sensors, at different
heights, in the front to measure the server inlet temperatures, two sensors in the rear to
measure the server exhaust temperature, as well as one sensor under the raised floor to
measure the supply air temperature under the rack.
16
Data center number two has an identical deployment of temperature sensors with a
similar density of pressure sensors under the floor.
Both facilities are excellent examples of legacy data centers. While they generally follow
a hot aisle/cold aisle configuration, the rows are short and are not always oriented in the
same direction throughout the room. They have a wide assortment of equipment in an
even wider assortment of cabinet types. They also feature raised floor cooling with lots of
abandoned wiring under the floor and old leaky solid floor tiles that allow too much cold
air to escape where it is not needed to cool equipment.
The first goal of this study was to prove that the centrifugal fans found in the CRAH units
behaved according to the ideal fan laws as stated as follows.
𝑅𝑃𝑀
𝐶𝐹𝑀2 = 𝑅𝑃𝑀2 ∗ 𝐶𝐹𝑀1
1
(2.2)
2
𝑅𝑃𝑀
𝑆𝑃2 = (𝑅𝑃𝑀2 ) ∗ 𝑆𝑃1
1
𝑅𝑃𝑀
(2.3)
3
𝑃𝑜𝑤𝑒𝑟2 = (𝑅𝑃𝑀2 ) ∗ 𝑃𝑜𝑤𝑒𝑟1
1
(2.4)
Data center number one was used for this experiment because it is lightly loaded and the
floor pressure was significantly higher than the two floors of data center number two.
This allowed the speed of all of the fans to be reduced to around 50% without
jeopardizing the IT equipment.
17
Equation (2.2), the first fan law equation, states that the CFM supplied by the fan is
directly proportional to the speed of the fan. Unfortunately, in this experiment, this fan
law is impossible to verify because there is no means to measure the CFM of a CRAH
unit with any accuracy.
Equation (2.3) states that when the fan speed is reduced by half the static pressure drop
across the fan will be reduced by a factor of 4. This is significant in a raised floor data
center because the airflow to the servers depends heavily on static pressure. This
equation was verified by dropping the fan speeds and monitoring the changes to the sub
floor pressure with the wireless pressure sensors.
Equation (2.4) states that when the fan speed is reduced by half the power consumption
of the motor will be reduced by a factor of 8. This is the most important equation related
to energy savings because it is used to predict how much energy can be saved by
reducing the speed of the fans. This is where the energy savings benefits of a VFD
comes into play and it is important that this result be verified with real world data. This
result was verified by turning down the speed of the VFDs incrementally and observing
the resulting power draw from the fan motor. The VFD monitors the fan motor power to
the tenth of a kW.
18
The second goal of this study was to determine the region of influence of a CRAH unit.
In other words, over how large of an area can one unit reasonably affect cooling to the
servers? This was verified by reducing fan speeds in one area of the data center and
noting pressure changes throughout the sub floor. This experiment was performed in
both floors of data center two because the large size of the floors allowed the changes to
be verified farther away from the source.
A pressure sensor, centrally located to the CRAH units that had their fan speeds turned
down, was used as a baseline measurement. The pressure drop that the baseline sensor
saw was then compared to the pressure drop of other sensors throughout the floor.
The last goal of this study was to verify the hypothesis that the centrifugal fans in the
CRAHs impart a horizontal velocity to the flow as it exits the CRAH unit. This would
cause an uneven pressure distribution under the floor that should be seen in the sub floor
pressure data. This uneven flow would not be an issue for a data center that only had
CRAHs installed along the perimeter walls. This experiment was performed in both
floors of data center two where there are multiple units installed in the center of the floor.
19
3. ANALYSIS OF THE DATA
Datacenter 1 was used for the fan law experiments because the datacenter’s high subfloor
pressure allowed the fan speeds to be modulated across a larger range, making the results
more accurate. The fan speeds were changed in small increments from 50% to 100%.
Power readings were read directly from the display on the VFD unit. Pressure readings
were taken from the nearest subfloor pressure sensor. The goal was to verify that the
centrifugal fans followed the fan laws closely enough that the laws could be used in the
future to predict pressure drops and energy savings if fan speeds were reduced.
Datacenter 2 was used for the region of influence and pressure distribution studies
because it is a larger datacenter and because the CRAH units are arranged
asymmetrically. For the region of influence study the subfloor pressure sensors were
used to take pressure readings before and after the fan speeds were reduced to 60% with
the goal being to see what the pressure change was throughout the floor. The goal of the
pressure distribution study was to map the subfloor pressure and correlate the high and
low pressure regions to the direction of the centrifugal fans. This was also done using the
data from the subfloor sensors.
20
3.1 Verifying the Fan Law for Pressure
In order to verify that equation (2.3) holds true for computer room air handlers, the sub
floor pressure in data center 1 was recorded while the CRAH units were running at 100%
fan speed. This pressure data is recorded in column 2 of Table (3.1). Fan speeds were
then reduced to 56% on every unit in the data center and the sub floor pressure was
recorded again in column 3. Column 5 calculates the pressure that the fan law predicts
for a 56% fan speed based on what the pressure was at 100% and the last column shows
the percent error between the measured subfloor pressure at 56% and the predicted sub
floor pressure at 56% for each sensor. The error is significant for every sensor and
implies that subfloor pressure cannot be accurately predicted based on the ideal fan laws.
21
Table (3.1) Verifying the Pressure Fan Law
Fan Law
Measured 100% Measured 56% Ratio of fan
Pressure at
Absolute
Sensor
fan speed
fan speed
speeds
56% fan
Difference
Location
pressure
pressure
(56/100)2
speed
Error
% error
AF-48
0.113 in H2O
0.044 in H2O
0.3136
0.0354 in
0.0086
19.46%
AI-54
0.105
0.042
0.3136
H2O
0.0329
0.0091
21.60%
BH-67
0.117
0.045
0.3136
0.0367
0.0083
18.46%
BM-27
0.102
0.041
0.3136
0.0320
0.0090
21.98%
CS-57
0.087
0.044
0.3136
0.0273
0.0167
37.99%
BH-54
0.105
0.043
0.3136
0.0329
0.0101
23.42%
AL-24
0.113
0.046
0.3136
0.0354
0.0106
22.96%
CY-62
0.08
0.037
0.3136
0.0251
0.0119
32.19%
BY-20
0.101
0.041
0.3136
0.0317
0.0093
22.75%
CP-63
0.085
0.041
0.3136
0.0267
0.0143
34.99%
22
3.2 Verifying the Fan Law for Power
To verify equation (2.4), the ideal fan law describing power, fan speeds were varied from
50% to 100% and the fan power was recorded for each speed. The VFDs that these
CRAH units were retrofitted with only display fan power to the tenth of a kW. Table
(3.2) shows data from one CRAH unit, but the results were typical for all the units in
datacenter 1.
The data in table (3.2) shows that the centrifugal fans in these units do follow the fan
laws closely, especially in the 100-70% range, which is where most of the energy savings
is. However, why does the percent error increase as the fan speed decreases? This is
because the VFDs equipped to these units are only accurate to 0.1 kW. The fan power
changes so little between 70-50% that 0.1 kW is significant enough to cause a large
change in the error.
Table (3.3) shows the same data as Table (2) but with the fan power readings from the
VFD rounded down a tenth of a kW. With this small modification to the rounding error
caused by the inaccuracy of the VFDs the percentage error decreases dramatically and the
fan power follows the ideal fan law very closely. Even at 50% fan speed the actual fan
kW is only off 0.05 kW from the value predicted by the ideal fan law. This is more than
accurate enough to allow the ideal fan laws to be used for calculating energy savings.
For example, assuming these units run 24 hours a day, 365 days a year and electricity
costs are $0.1/kWhr, then reducing one unit to 50% fan speed would result in an annual
23
energy savings of over $4,100. The ideal fan law prediction at 50% speed would only be
off by $40 per unit, per year, compared to the measured values.
Table (3.2) Verifying the Power Fan Law
Ratio of Fan Speeds
% Fan Speed Fan Power kW
(x/100)3
Predicted kW
Delta
% error
100
5.2
1.00
5.2
0
0
80
2.8
0.51
2.66
0.14
4.9%
70
1.9
0.34
1.78
0.12
6.1%
65
1.6
0.27
1.43
0.17
10.7%
62
1.4
0.24
1.24
0.16
11.5%
60
1.3
0.22
1.12
0.18
13.6%
50
0.8
0.13
0.65
0.15
18.8%
24
Table (3.3) Rounded Power Data
% Fan
Fan Power kW
Ratio of Fan
Predicted kW
Delta
% error
Speeds (x/100)3
Speed
100
5.2
80
2.7
0.51
2.66
0.04
1.4%
70
1.8
0.34
1.78
0.02
0.9%
65
1.5
0.27
1.43
0.07
4.8%
62
1.3
0.24
1.24
0.06
4.7%
60
1.2
0.22
1.12
0.08
6.4%
50
0.7
0.13
0.65
0.05
7.1%
25
3.3 Region of Influence of a CRAH Unit
In order to lower fan speeds as much as possible, and achieve maximum energy savings,
it is important to understand what the effect of the reduced airflow will have on the subfloor pressure throughout the datacenter. In this experiment the fan speeds of 5 CRAH
units in datacenter 2, floor 1 were reduced to 60%. The goal of this was to cause
significant loss of floor pressure in the vicinity of the CRAH units in order to observe
how the pressure loss propagated across the floor. 60% was as low as the fan speeds
could be lowered and still safely provide adequate cooling to the room. The reason more
than one unit was turned down was to get a large enough pressure drop that the effect
could be more easily seen on the opposite side of the room.
A pressure sensor that was centrally located to the five units was chosen as a baseline
pressure reading, shown with a red arrow in figures (3.1, 3.2). This sensor should see the
largest drop in pressure when the fans are turned down. Additional sensors were chosen
along a diagonal line to the opposite corner of the floor and their readings were compared
to the value of the baseline pressure sensor. The goal was to see how the pressure loss
was distributed over distance.
The experiment was then repeated on the second floor of datacenter 2, except three units
were reduced to 60% fan speed instead of five. Fewer units were chosen in order to
compare whether the magnitude of the pressure loss near the units with the reduced fan
speeds determines how far away the effect is seen across the datacenter floor. Figures
26
(3.1) and (3.2) show the units that were chosen on both floors of datacenter 2. The
CRAH units are the red boxes and the red arrow points to the baseline pressure sensor.
27
Figure (3.1) Datacenter 2 floor 1 - CRAH Units at 60% and Baseline Pressure Sensor
28
Figure (3.2) Datacenter 2 floor 2 – CRAH Units at 60% and Baseline Pressure Sensor
29
After the fan speeds of the units were reduced to 60% the floor pressure was allowed to
stabilize, new pressure readings were recorded and pressure difference between 100% fan
speed and 60% fan speed was calculated. Figure (3.3) shows these pressure differences
versus the straight line distance away from the baseline pressure sensor.
Figures (3.3, 3.4) show that even as far as 270 feet away, there is still significant pressure
reduction due to the fan speeds being lowered. On the first floor the baseline sensor saw
a 40% reduction in pressure and the farthest sensor saw a 25% reduction. The pressure
drops across all the sensors on the second floor were much less than the first floor, but the
trend is the same, turning down units on one end of even a very large floor can still have
a significant effect on pressures on the far opposite end of the floor.
30
Figure (3.3) Change in Pressure Vs. Linear Distance
Figure (3.4) % Change in Pressure Vs. Linear distance
31
3.4 Uneven Pressure Distribution
Datacenter 2 was used for this study because the CRAH units on the two floors are
spaced unevenly and about half are deployed in the center of the room, instead of along
the perimeter walls. The goal was to see how this uneven arrangement effected the way
the pressure was distributed under the floor. For the experiment the fans speeds on the all
the units were set to 100% and readings were taken from all of the pressure sensors
installed in the datacenter. The pressure sensors read pressure in inches of water and are
accurate to 0.001 inches H2O. The locations of the pressure sensors can be seen in
figures (3.1) and (3.2).
On floor 1, the units in the center of the floor are all pointing in the same direction,
indicated by the red arrows, (West in figure (3.5)). On the second floor the units in the
center of the floor all point in different directions, but mostly down and to the left, of
figure (3.6) This combination of factors resulted in an uneven distribution of pressure
under the floor.
Figure (3.5) shows the pressure distribution under the floor of datacenter 1 floor 1. The
red areas are regions of higher pressure, about 0.06 inches H2O, and the blue regions are
lower pressure, about 0.035 inches H2O. It is clear that the red regions are concentrated
neat the top of the floor plan, which is in the direction that the CRAH units are facing.
This implies that the centrifugal fans in these units impart a significant amount of
horizontal velocity to the discharge air under the floor.
32
Figure (3.6) shows the pressure distribution under the raised floor of the second story of
datacenter1. The same type of pressure distribution can be seen on the second floor. The
CRAH units generally point downwards and to the left in figure (3.6). The green and
yellow coloring in the upper left of figure (3.6) clearly shows that there is more pressure
where the units are facing. Also, in the middle of the floor there are 4 units that are all
pointed toward each other, with a corresponding region of high pressure in the middle of
them.
33
Figure (3.5) Uneven Pressure Distribution, Datacenter 1 Floor 1
34
Figure (3.6) Uneven Pressure Distribution, Datacenter 1 Floor 2
35
4. FINDINGS AND INTERPRETATIONS
The fan power from the first experiment was conclusive in proving that the centrifugal
fans commonly used in older CRAH units follow the fan law predicting power usage with
good enough accuracy to predict potential energy savings. The VFDs were only accurate
to 0.1kW but this accuracy is more than adequate when potential savings are calculated
annually. The fan law equation for CFM was not proven, because there is no easy way to
measure the CFM of a running CRAH unit while it is actively cooling a datacenter.
Unfortunately, the same cannot be said for predicting subfloor pressure. The pressure
sensors routinely recorded a higher subfloor pressure than what was predicted by the fan
laws. This means that the fan laws cannot safely be used to predict subfloor pressures
when fan speeds are reduced. This inaccuracy is most likely due to the fact that there are
multiple other sources and sinks of pressure inside a datacenter. For example when the
VFD fan speeds ramp down, server fan speeds may ramp if the cooling is adequate. In
older datacenters especially, solid floor tiles often fit poorly and allow leakage
throughout the floor, and cable cutouts may be sealed improperly.
Perhaps the most surprising result was how large of an area of influence CRAH units
have. On the first floor, turning the fan speeds down on 5 units resulted in a 40%
decrease in pressure in the immediate vicinity of the units, but it also resulted in a 25%
decrease in pressure over 270 feet away. The results on the second floor showed the
36
same trend, but was not as dramatic because only 3 units were turned down due to safety
of the IT equipment.
This result is important because datacenter operators may think that because they have
adequate floor pressure in one area of the floor they can turn down (or turn off) units to
save energy. However, those actions may have unintended consequences for other areas
of the datacenter, especially if those areas already have low floor pressure. For example,
looking at figure (3.5) we see that the area around the 5 CRAH units that were turned
down to 60% has the highest floor pressure, which would suggest that it is safe to reduce
the fan speed in this area. However the area farthest away from the 5 CRAH units is also
the area with the lowest floor pressure. When the CRAH units were turned down this
area saw a 25% decrease in floor pressure, which may have put equipment in jeopardy.
Whether it is an automatic control system or a datacenter operator manually changing fan
speeds, this experiment shows that the area of influence of a CRAH unit may be much
larger than expected.
The pressure maps shown in figure (3.5) and figure (3.6) show conclusively that CRAH
units with centrifugal fans do impart a horizontal velocity to the supply air under the
floor, which causes uneven pressure. This shows that as long as the room isn’t too large
the best configuration is to install the CRAHs along the outside perimeter of the
datacenter. If the room is large and units need to be placed in the center of the room then
they should be placed back to back to ensure even pressure distribution.
37
All of these results are important for determining energy savings possibilities in a legacy
datacenter. When subfloor pressure is known, it is possible to determine the amount of
airflow through each floor tile. Table (4) shows the manufacturers specs for a typical
perforated floor tile in a datacenter. Most of the datacenter studies and server
requirements seem to agree that between 80-120 CFM is required per kW of server load5.
The results of this study can help a data center operator achieve even subfloor pressure to
his equipment and predict potential energy savings from fan speed reductions
The massive amounts of data available from the wireless sensor network allows the
datacenter operator to fine tune the subfloor pressure and CFM delivered by his CRAH
units to meet the demands of the servers, while the temperature data from the wireless
sensors provides a factor of safety by monitoring the inlet temperatures to the racks.
Table(4.1) Typical CFM Vs. Pressure for a Common Datacenter Perforated Tile
38
5. CONCLUSIONS AND FUTURE WORK
The results presented in this study are important for any datacenter operator looking to
save energy or add more IT load and/or cooling capacity to one’s datacenter. Even
distribution of subfloor pressure is essential to ensure adequate cooling to IT equipment
on the datacenter floor. CRAH units cannot just be deployed wherever there is space on
the floor, pressure and flow direction also need to be considered.
It is important to be able to predict energy savings in order to secure rebates from electric
utilities and also to know where savings can be achieved without putting equipment at
risk. Wireless sensor networks can provide a cheap and reliable way to monitor the
environment inside the datacenter and allow the operator to match cooling to IT load
safely and reliably.
One parameter that needs to be investigated further in the future is the effect of raised
floor height on subfloor pressure. Given the same cooling capacity, configuration and IT
load, how much less floor pressure would a 4 foot raised floor have compared to a 2 foot
raised floor? Would the larger volume help or hinder airflow by decreasing the effect of
airflow restrictions? One of the reasons this factor was omitted from this study is that
many datacenters will need to be used in order to get an adequate sample of floor heights
and because no two datacenters are alike in all respects besides raised floor height,
making it difficult to draw an accurate comparison.
39
APPENDIX A- Liebert CRAH Unit Data Sheets
40
REFERENCES
1. Energy Star Program, U.S Environmental Protection Agency. Report to Congress
on server and data center energy efficiency Public Law 109-431. Washington,
D.C. 2007.
2. Pacific Gas and Electric Company, High Performance Datacenters A Design
Guidelines Sourcebook, January 2006
3. Petschke, Benjamin, Datacenter Cooling Best Practice. Stulz White Paper, April
2008
4. Aperture Technologies White Paper, Computation Fluid Dynamics Modeling for
Operational Data Centers, Aperture Technologies Inc. 2009
http://www.emersonnetworkpower.com/documents/enus/brands/aperture/documents/apertureresourcelibrary/whitepapers/01/whitepaper
_cfd%20operationaldc.pdf
5. Moss, Davis, Guidelines for Assessing Power and Cooling Requirements in the
Data Center. Dell Power Solution 2005
6. VanGilder, James and Schmidt, Roger R. Airflow Uniformity Through Perforated
Tiles in a Raised Floor Data Center. White Paper #121 ASME InterPACK 2005
7. AMD. 2006. Power and Cooling in the Data Center. Advanced Micro Devices.
34246C. http://enterprise.amd.com/Downloads/34146A_PC_WP_en.pdf.
8. Bell. Geoffrey C. Demonstration of Datacenter Automation Software and
Hardware(DASH) at the California Franchise tax Board. Lawrence Berkeley
National Laboratory, December 2009
41
9. Herrlin, Magnus K. Rack Cooling Effectiveness in Data Centers and Telecom
Central Offices: The Rack Cooling Index (RCI), ASHRAE Transactions, Vol 111,
Part 2, 2005.
10. Radmehr, Amir; Schmidt, Roger R.; Karki, Kailash C. and Patankar, Suhas V.
Distributed Leakage Flow in Raised Floor Data Centers. ASME InterPACK 2005
11. Marshall, Liz and Bemis, Paul. Using CFD for Data Center Design and Analysis.
Applied Math Modeling White Paper. January, 2011