One Dell Way
R
Round
Rockk, Texas 78
8682
www.dell.com
DEL
LL ENTER
RPRISE WHITEPAP
W
PER
THERM
MAL DESIG
D
GN OF
O TH
HE
DEL
LL™ POWE
P
ERED
DGE™ M-S
SERIE
ES
K.C. Coxe
C
Enterp
prise Therm
mal Enginee
ering
THIS WHITE PAPER ISS FOR INFOR
RMATIONAL PURPOSES
P
ONLY, AND MAY
M
CONTAIN
N TYPOGRAPH
HICAL
ERRORS AND
A
TECHNIC
CAL INACCUR
RACIES. THE CONTENT ISS PROVIDED AS IS, WITH
HOUT EXPRESSS OR
IMPLIED WARRANTIES
W
S OF ANY KIND
D.
AMD and
d Opteron arre trademarkks of Advanced Micro Deevices, Inc. Intel and Xeeon are regisstered
trademarks of Intel Co
orporation. Other
O
tradem
marks and trad
de names maay be used in
n this documeent to
refer to either
e
the enttities claimingg the marks and
a names or
o their produ
ucts. Dell disclaims proprietary
interest in
n the marks and names of others.
For more information, contact Dell..
Informatio
on in this doccument is sub
bject to changge without no
otice.
2|Page
1 INT
TRODU
UCTION
N
With energy costs rising, energy efficiency
e
in IT equipmentt has quickly become a major
m
focus of
o Dell
customers. One of th
he key avenu
ues of addresssing energy efficiency att the platform
m level lies in the
developm
ment of a high
hly efficient th
hermal solution. System cooling
c
can co
onsume a siggnificant portiion of
overall system power, but it doesn’t need to.
The launcch of the De
ell™ PowerEd
dge™ M1000ee Modular Enclosure and
d PowerEdge M600 and M605
blades maarks an evolu
utionary leap in thermal deesign efficien
ncy by Dell. The
T thermal architecture
a
w
within
the system stretches from the layyout of each
h blade to Chassis Managgement Conttroller (CMC)) that
he fan modules that provvide the airflo
ow. This pap
per will walkk the user thrrough
manages cooling to th
some of the
t features of
o the thermal design and
d discuss the underlying principals
p
and
d design goalss that
are intend
ded to result in one of the most thermaally efficient servers
s
on thee market.
A reader seeking a mo
ore detailed description
d
off the featuress of the PoweerEdge M100
00e and assocciated
blades and peripheralss should read the Dell M-Series Architecture Whitep
paper availablle at dell.com
m.
2 DE
ESIGN GOALS
G
The Dell M1000e
M
is designed to reliiably operatee at ambient conditions
c
up
p to 35°C (95°°F) and at altitudes
of up to 10,000 feet (3,048m). In addition, the system supports
s
red
dundant cooling. The theermal
performance requirem
ments to meeet such condittions, however, are not tyypical of mosst deploymen
nts. A
more typ
pical data cen
nter could be supplying 15-17°C air from
f
the floor tiles, a much
m
more benign
b
operatingg condition. Even a dataa center opeerating at “higher” tempeeratures arou
und 25°C offfers a
significantt opportunityy for reducingg power-to-co
ool vs. a 35°C environmentt.
The challeenge that Dell faced in thee developing the
t M1000e was
w to develo
op a solution that is both:
• Reliable operation at worstt-case, redund
dancy lost conditions
• Optimized
O
for typical data center
c
conditions
w detail how
w Dell accom
mplished thiss in hardware design as well as in syystem
The sections below will
thermal management
m
through the Integrated
I
Deell Remote Acccess Controller (iDRAC) an
nd CMC.
3
CO
OOLING
G SOLU
UTION OVERV
VIEW
The sections below are intended to provide a general understanding of the major elements of
o the
thermal solution in the
e Dell PowerEEdge M1000ee Modular Encclosure. Thesse sections arre not all-inclusive,
but will ho
opefully provvide the readeer an appreciaation of the engineering
e
th
hat has gone into the design.
3|Page
Figure 1:: Front View of PowerEdg
ge M1000e wiith
1 Half-Heigh
16
ht Blade Encllosures
Fig
gure 2: Rear View
V
of PoweerEdge M10000e
3.1 What
W
prov
vides th
he coolin
ng
The PoweerEdge M1000
0e is cooled by
b 9 custom fan
f modules. The fan module was dessigned to meeet the
specific co
ooling require
ements of the M1000e, in
ncluding its airflow operatting point and
d power efficciency
requiremeents. Each fan
f is hot sw
wappable and
d can be indiividually speeed controlled
d to help opttimize
system co
ooling during normal run time or if redu
undancy is losst.
F
Figure
3: Dell M1000e Fan
n Module
Cooling iss managed byy a combinatiion of blade-level (iDRAC)) and enclosu
ure-level (CMC
C) controllerss (See
Figure 4). The thermal control algo
orithm runnin
ng on each blades’ iDRAC is
i an evolutio
onary step forrward
from the BMC that run
ns on the Delll PowerEdge 9G servers. It has the cap
pability to anaalyze the hard
dware
configurattion on each
h blade, the thermal cond
dition createed by softwarre load on th
he blade, and the
ambient temperature
t
ommunicate the
t blade’s sp
pecific cooling requiremen
nts to
of the blade and then co
the enclossure.
4|Page
The enclo
osure (CMC),, in turn, is designed to
o interpret each
e
of the 16 blades’ inputs, as well as
temperature sensors in the M1000
0e infrastructure (such as in the IO Modules and thee control pan
nel on
the front of the M100
00e), and set each fan to the
t lowest sp
peed possiblee – helping to
o minimize airflow
and poweer consumptio
on – to meett the cooling requirementss of the systeem. The CMC
C controls thee fans
in zones, with each zo
one mapped to
t the coolin
ng of a group of blades baased on locattion in the ch
hassis.
This desiggn allows the CMC to increease only thee fans needed
d to cool hottter blades orr blades with more
power-inttensive hardw
ware configurrations, whilee leaving the rest
r of the fan
ns at a lower power and airflow
saving levvel.
Fiigure 4: Fan Control
C
Block
k Diagram
3.2 Aiir Manag
gement
The M1000e makes use
u of parallel air paths to cool the blades, IO Modules,
M
and
d power sup
pplies.
h of the major subsystem
ms receives ambient
a
air to
t help avoid pre-heating the
Because of this, each
modules in the back of the system. Providing ambient air to the IO Modules and
d Power Sup
pplies,
despite th
heir locationss in the rear of the system, was a maajor focus of the early sysstem architectural
engineering, with the
e goal of allowing the modules
m
to consume lesss airflow while
w
still meeeting
ure requiremeents.
component temperatu
The serveer modules arre cooled with
h traditional front-to-backk cooling. As shown in Figgure 5, the fro
ont of
the system
m is dominatted by inlet area for the in
ndividual blad
des (green higghlighted areea). The air passes
p
through the server mo
odules, througgh vent holess in the midp
plane, and is then
t
drawn in
nto the fans which
w
t air from the chassis. There are plenums
p
both
h upstream of
o the midplane, between the
exhaust the
5|Page
midplane and the blades, and dow
wnstream of the midplan
ne, between the
t midplanee and the fan
ns, to
distributee the cooling potential
p
from
m the three columns of fan
ns across the server modu
ules.
Serrver Module
e Inlet
(M1
1000e frontt view)
Server Mo
odule Cooling Air Proffile
(M
M1000e side
e view)
Figure 5:
5 Dell Blade Server Modu
ule Cooling Path
P
The Poweer Supplies, lo
ocated in thee rear of the system, use basic front-tto-back coolin
ng, but draw their
inlet air frrom a duct located beneatth the server modules, as seen
s
in the green highlight in the left side of
Figure 6. This insuress that the power supplies receive amb
bient temperature air, wh
hich minimizees the
v
of airr needed to co
ool the power supply.
required volume
Power Sup
pply Inlet
((M1000e fro
ont view)
Power Sup
pply Cooling
g Air Profile
e
(M10
000e side view)
v
Figuree 6: M1000e Power
P
Supply
y Cooling Patth
The IO Modules use a bypass duct to draw amb
bient air from
m the front of
o the system to the IO Module
inlet, as seen
s
in Figure
e 7. This duct is located above the seerver modulees (again high
hlighted in grreen).
This cool air is then drawn
d
down through the IO Modules in a top-to-b
bottom flow path and intto the
b
the midplane
m
and
d fans. From this plenum,, the air is exxhausted from
m the system. The
plenum between
CMC and Integrated Ke
eyboard/Videeo/Mouse (iKVM) are also cooled by airr in this flow path.
p
6|Page
IO Module Inlet
(M
M1000e fron
nt view)
IOM Lo
ocations and
d Airflow Direction
(M1000e back
b
view)
IO Coo
oling Air Pro
ofile
(M100
00e side vie
ew)
Figu
ure 7: M1000ee IO Module Cooling
C
Path
h
4 DE
ESIGN CONSID
C
DERATIIONS AFFECT
A
TING PO
OWERTO
O-COOL
L
4.1 Im
mpedanc
ce, Airflo
ow, and Fan Effficiency
y
Developin
ng a fan to cool the M1
1000e and provide
p
scalability for thee next severral generatio
ons of
computing architecturre provided a unique challenge. The faan needed to
o meet an aggressive operrating
point, yett fall within a much tighter power budget than off-the-shelf
fans could provide.
o
p
Thiss was
accomplisshed by identtifying a fan efficiency
e
targget that requ
uired custom technology to be develop
ped to
fit the neeeds of the M1
1000e. This section
s
will prrovide a brieff overview of what fan effiiciency is, and
d why
it is criticaal to an efficie
ent thermal design.
d
There is somewhat
s
of a misconcepttion about ho
ow fans perfo
orm and are specified.
s
A fan
f purchased at a
local electtronics store for a desktop
p PC that is ad
dvertised as an
a “80 CFM” fan
f will not move
m
80 CFM of air
when insttalled in a PC.. This is called a “free air” measurement of the fan performancee, and is how much
7|Page
air the fan
n moves whe
en sitting on a table with nothing
n
occluding the inlett or outlet (su
uch as the intternal
components of a computer). This method of describing a faan is so comm
monplace thaat fan vendorrs will
even list this value on
n their datash
heets, even though
t
it is only
o
a partiall specification
n of a fan’s actual
a
airflow ou
utput.
The half of
o the fan equ
uation that isn’t often disccussed is impeedance. Impedance is thee opposition to
t the
flow of air,
a is measurred in pressu
ure, and in US
U customaryy units is rep
presented byy inches of water,
w
(Pascals in
n SI derived units).
u
The more
m
impedance that a faan is loaded with,
w
the lesss airflow it acctually
moves. A fan in free air has zero impedance. In an electrronics system
m, however, the
t fan runs at an
operatingg point of ‘airfflow at a giveen impedancee.’
The chartt below is typ
pical of a fan that might be
b found in a desktop com
mputer or tow
wer server. In this
chart, thee free-air ratin
ng is 116 CFM
M, but the acttual operatingg point is closser to 80 CFM
M at 0.19 inch
hes of
water (in.w.g.).
0.6
Fan Curve
Impedance (in.w.g.)
0.5
0.4
Operatting
Point
0.3
Freee Air
0.2
System Imp
pedance
Curve
0.1
0
0
20
40
60
80
100
120
140
Airflow (CFFM)
Figu
ure 8: Examplle Fan Perform
mance Curvee
Why is th
his importantt? A thermaal design enggineer knowss how much airflow is reequired to co
ool an
electroniccs enclosure and can measure the im
mpedance of the enclosurre to determ
mine the operrating
point required of the system
s
fan. The operatin
ng point of th
he system fan is a significcant driving factor
f
for how much
m
power is needed to cool the systtem. The op
perating pointt of a fan actu
ually describees the
amount of
o energy (wh
hich can be represented
r
as power) th
hat needs to be put into the air to fo
orce it
through a system. Wh
hen compared
d to the electtrical power required
r
to ru
un the fan, th
he efficiency of
o the
fan can bee determined
d.
8|Page
η=
Worrk out Operrating Flow
wrate × Operrating Presssure
=
Worrkin
Innput Powerr
Eq. 1
Fan efficieency is depen
ndent on how
w the fan is built
b
and dessigned, from the shape off the blades to
t the
quality off the motor, and
a can vary significantly,, even from one
o fan to an
nother in the same form faactor.
What doees all this meaan with respeect to cooling power? There are two im
mportant poin
nts:
1. A higher efficiency fan wiill consume proportionally
p
y less powerr than a low efficiency fa
an. A
higher efficien
ncy fan design
n, leading to a lower pow
wer fan, results in more power availab
ble for
co
omputing. Alternatively,
A
o
poweer consumptio
on, which caan be
it can result in lower overall
trranslated into
o cost savingss in the deployment. Thee higher efficciency may co
ome at the co
ost of
m
more
expensivve componen
nts to build the
t fan, so itt is critical th
hat a thermaal design enggineer
seelect a fan with a material cost approp
priate to the amount of en
nergy that may be saved in
i the
deeployment.
2. Iff a system ca
an be designeed with a low
wer work outp
put requirem
ment, the fan will consume less
po
ower. We have discussed how the operating point is determined byy the airflow
w and
im
mpedance of the system. While the volume of airflo
ow required to
t cool an eleectronics enclosure
iss somewhat flexible, it often
o
needs to be maintained at a minimum
m
levvel to contro
ol the
teemperature rise
r through the
t system (m
more on that later). Assuming a minim
mum fixed airflow,
im
mpedance is the
t other variiable that can
n affect the po
ower-to-cool.
These principals guide
ed not only development
d
of the M100
00e fan, but the
t close relaationship witth the
impedancce drove a lo
ow-impedance design as well. The effficiency of the M1000e fan
f is significcantly
higher thaan typical 92m
mm or 80mm
m form factor fans used in many
m
PCs currrently on thee market. Thiis was
achieved through opttimization off the aerodyynamics, mottor, and mottor controller design witthin a
custom fo
orm factor.
This fan iss coupled with a blade deesign with up
p to half the impedance of
o competitivve products on
o the
market an
nd a chassis midplane thaat underwentt several desiign iterationss to help optimize airflow. The
blade encclosure and layouts were studied exteensively priorr to building the first funcctional system
ms to
understan
nd what facto
ors contributted to the impedance of the
t module, and targetingg certain areaas for
improvem
ment. These layout studies resulted in
i componen
nt placements made for airflow,
a
as well
w as
subtle chaanges in the industrial dessign of the blaade, all to help drive lowerr impedance and
a a lower power
p
thermal design.
d
4.2 Aiirflow an
nd Temp
perature
e Rise
There aree numerous benefits
b
a cusstomer can reealize by loweering fan speeds and increeasing the exxhaust
air tempeerature from a server. To illustrate this effect, conssider the simplification off a Dell PowerEdge
1950 servver into basic thermal build
ding blocks: an
a airflow ressistance, a heeat source, a power
p
distrib
bution
system, and a fan (Figu
ures 9 and 10
0).
9|Page
Figure 9:: PowerEdge 1950
Figurre 10: Block-D
Diagram Simp
plification off a Server's Airflow
A
System
m
me the serverr is under a constant
c
load
d that requirees a 300W input. With th
he fan
In this exaample, assum
running at
a 9,000 RPM
M, there is an 8°C rise thro
ough the systtem. By deccreasing the fan
f speed to 3000
RPM, the exit air tem
mperature increases to 22°C above thee inlet tempeerature, but the
t system power
p
decreasess 55W!
10 | P a g e
Figure 11:
1 Comparison of Fan Speed, Power Draw,
D
and Exh
haust Tempeerature at Hig
gh (top) and Low
L
(botto
om) Fan Speed
ds
The reaso
on for the draamatic power decrease beetween the high
h
and low fan speeds in Figure 11 iss that
fan power and RPM do
o not have a linear
l
relation
nship. In factt, power chan
nges as a cubiic function off a fan
speed chaange, so small decreases in
i fan speed can have sign
nificant impacts on the po
ower consum
mption
of a fan (EEq 2).
Poweriniitial ⎛⎜ RPM
M initial
=
⎜ RPM
Powerfinal
M final
fi
⎝
⎞
⎟
⎟
⎠
3
(Eq 2)
As an example, and ussing (Eq 2), a 10% reductio
on in fan RPM
M can result in
n a 27% reduction in fan power
p
consumpttion! Because of this relattionship, there was a consscious effort made in the M1000e design to
minimize fan speed and increase exhaust air temperature,
t
, with the intended beneefit of significcantly
reducing system powe
er and airflow consumption. All of the
t componeents downstrream in the warm
w
airflow haave been desiigned to meet the high tem
mperature requirements of
o this design condition.
o the high teemperature rise
r that strettches beyond
d the enclosu
ure-level efficciency
An expectted benefit of
improvem
ments in the M1000e is th
hat the capaccity of the air conditionerrs in a data center
c
can acctually
increase with
w a higherr temperaturee rise as the CRAC (Computer Room Air
A Conditioneer) operates closer
c
to its opttimum return
n temperaturre. Care musst be taken when
w
selectin
ng air conditioning to pro
operly
match it to a datace
enter deployyment, but the
t
higher reeturn tempeerature to th
he CRAC (exxhaust
temperature from the system) can have benefitss beyond pow
wer-to-cool att the IT enclosure level.
11 | P a g e
4.3 Co
ompone
ent Selec
ction
With a ch
hassis designe
ed for therm
mal efficiency,, there are sttill several op
pportunities for a custom
mer to
lower pow
wer and airflo
ow consumption by selectiing easier-to--cool compon
nents and blad
de configurattions.
0, for examplle, is available with processors rangingg from 40W (Intel® Xeon® 5148LV) to 120W
1
The M600
(Intel Xeo
on X5355) the
ermal design power levelss. In addition to the pow
wer savings asssociated witth the
processorr power, the lower preheaat placed on downstream
m componentss by the 5148
8LV as oppossed to
the X5355
5 provides an
n opportunityy for airflow reductions ranging
r
from 5% to 40% at
a a 25°C am
mbient
temperature, depending on other components
c
in the blade.
Memory configuration
c
ns can also afffect cooling power and airflow
a
requirrements. Nottably, the M6
600 is
more efficient coolingg memory wh
hen 4 higher capacity DIM
MMS are useed in place off 8 lower cap
pacity
DIMMs to
o achieve a given memo
ory capacity.. Airflow reeductions of up to 25% can be achiieved,
depending on memorry capacity reequired and other
o
compo
onents in the blade. Thiss is a result of
o the
ed in every other
o
slot in the
t memory bank,
b
and thee resulting reeduction in DIMMDIMMs beeing populate
to-DIMM heating.
5 CA
ASE STU
UDY: POWERE
EDGE M600
M
V M60
VS.
05 VS.
PO
OWERED
DGE 19
950-III
What doees the custom
mer get for a well-designeed blade servver with cooliing efficiencyy designed-in from
the produ
uct’s inceptio
on? Considerr an M600 deeployment an
nd a deploym
ment of 16 PowerEdge 19
950 III
servers with similar hardware, particularly:
• (2
2) Intel Xeon 5130
5
processsors
• (4
4) 2GB Fully Buffered
B
DIMM
Ms
c
an M605
M
deploym
ment of 16 blades with
And also consider
• (2
2) AMD Opterron™ 2218 HE
H processorss
• (4
4) 2GB DDR2 2Rx4 DIMMs
12 | P a g e
Table 1: Compariso
on of Deploym
ment of 16 M600
M
Blades vs.
v M605 Blad
des
vs. 16 PowerrEdge 1950-IIII Servers
Num
mber of Servers
Racck-Space to Deploy
D
Serrver Density
Tottal AC Power Required1
Fan Power1, 2
Airfflow Consum
med by Deployyment3
Ave
erage Airflow
w per Server3
Po
owerEdge
19
950 III
16
6
16
6U
1.0
0 Servers/U
47
785W
53
30W
49
95 CFM
31
1 CFM
PowerEd
dge
M600
16
10U
1.6 Serveers/U
3835W
80W
370 CFM
23 CFM
Pow
werEdge
M60
05
16
10U
1.6 Servers/U
S
3660
0W
65W
W
340 CFM
21 CFM
Po
ower represen
nted in AC Watts
W
“at the wall” after power
p
supply
y efficiency iss considered
and
d assumes a SPECjbb
S
work
kload.
2 Fa
an power is in
ncluded in to
otal AC power, but is brok
ken out here for
f compariso
on of
cooling efficienccy.
3 Ai
irflow consum
mption is bassed on a 25°C
C ambient tem
mperature to the server an
nd will vary
as temperature varies.
v
1
From Tab
ble1, a deployyment of 16 M600 bladees deploymen
nt consumes 19% less po
ower and 29%
% less
airflow th
han the PowerEdge 1950 IIII deployment. Of the 845
5W in power savings, 450W
W is a result of
o the
efficiencyy in the coolin
ng solution.
The PoweerEdge M605 deployment shows additional power and
a airflow savings, madee possible by more
efficient cooling
c
in the
e parallel layo
out of the CP
PU and memo
ory in the airrflow, as welll as power saavings
associated
d with DDR2 as compared to FBD memory.
6 CO
ONCLUS
SIONS
Designingg the cooling solution forr IT equipment can be ass simple as selecting
s
a faan from a catalog,
installing it in a computer, and turrning it on. A basic appro
oach like that, however, will
w not resullt in a
solution that
t
is optim
mized for po
ower, airflow
w, or coolingg. The coolling solution deployed in
n the
PowerEdgge M1000e Modular
M
Enclosure and itts associated
d blades werre developed with the go
oal of
maximizin
ng energy effficiency and minimizing airflow requirements wh
hile still supp
porting worstt-case
operatingg conditions. Tools such as high efficciency fans, low-impedancce system deesign, advancces in
fan control, and
systems management
m
a low airflo
ow/high temperature rise are designed
d to help min
nimize
the poweer consumptio
on of the M1
1000e therm
mal solution and
a allow mo
ore of the customer’s avaailable
power to be dedicated
d to computin
ng, as opposed to cooling.
13 | P a g e