Applied Math Modeling White Paper
Using CFD for Data Center Design and
Analysis
By Liz Marshall and Paul Bemis, Applied Math Modeling Inc., Concord, NH
January, 2011
cialized functions. This white paper focuses
on how CFD is used for one such class of
rooms: data centers (Figure 1). It begins with
an overview of CFD basics and then describes how data center components can be
represented using numerical methods. The
chapter concludes with examples and suggested reading material.
Introduction
Computational fluid dynamics (CFD) is the
numerical simulation of fluid flow. It can be
used to predict fluid velocities, temperatures,
and many other variables of interest for a
wide variety of application areas. Over the
years it has been used to simulate the flow of
air over an airplane wing or water past a
ship’s hull; perform comparative
drag analyses of
automotive body
shapes; predict
the time needed to
mix two or more
liquids; and test
strategies for reducing coal plant
emissions. It is
routinely used to
model the flow
Figure 1: CFD can be used to illustrate
inside buildings
the flow of air, as in this example of hot
of all sizes and
aisle containment in a data center; flow
rooms with spepathlines are colored by temperature
© 2011 Applied Math Modeling Inc.
1
WP107
Computational Fluid Dynamics
straightforward. For data center applications,
the CFD software has objects available for
CRACs, racks, PDUs, perforated floor tiles
and other common components, which can
be positioned and sized easily within a room.
The goal of a CFD calculation is to predict
the highly detailed flow field in some region
of interest. The flow field is not just the motion of a fluid: a liquid or a gas or even a collection of small particles, such as sand. CFD
can also predict the distribution of temperature and chemicals. It can include other phenomena such as turbulence, timedependence, chemical reaction, evaporation,
melting, or freezing. Despite the range of
applications, the same general method is
used. Conservation equations are solved in
small cells that fill the region. The region
must therefore be divided into a large number of cells and the equations must be rewritten so that they can be solved in each one. In
the next few sections, these two processes
are described along with an overview of how
the solution is performed and how the results
can be viewed and interpreted.
Once the room and contained geometry are
created, the entire space needs to be broken
up into a large number of small cells where
the calculations will be performed and the
resulting information stored. The lines and
planes that separate these cells are often collectively referred to as the mesh or grid. The
cells can be of almost any shape; six-sided
hexahedra, five-sided pyramids or prisms,
and four-sided tetrahedra are the most common. The task of creating such a mesh can
be time-consuming for the CFD analyst, but
many automated or semi-automated tools are
available in software today. Data center
CFD software packages are among those that
offer this feature. A typical mesh in a data
center is shown in Figure 2. This mesh is
non-uniform, which means that the size of
the cells varies from region to region, depending on the detail required by the local
geometry.
The Computational Domain and Mesh
A CFD simulation begins with the creation
of the problem geometry. The region to be
modeled is often called the computational
domain, and the geometry consists of the
boundaries of that domain
and all of the objects contained within. While computer-aided design (CAD)
software focuses on the
solid objects, CFD focuses
on the space inside or in
between objects, where
the fluid is likely to flow.
Many application-specific
CFD packages are available nowadays that make
geometry creation
Figure 2: The mesh on a planar slice through a data center
© 2011 Applied Math Modeling Inc.
2
WP107
While the mesh in Figure 2 appears Cartesian, with lines at right angles, automatic
mesh generators often distort the elements to
fit around non-rectangular objects. One example is shown in Figure 3, where the mesh
is distorted near the circular object and actually changes from Cartesian to a different
style (called O-type) around some of the objects. Automatic meshing algorithms have
instructions to deviate from a standard Cartesian format based on the needs imposed by
the geometry. Often, a switch to another grid
style will result in a reduced cell count in the
domain, making the calculation run more
quickly.
ing. The process of translation is referred to
as convection, while the process of distortion
is related to the presence of gradients in the
velocity field and a process called diffusion.
In the simplest case, these processes govern
the evolution of the fluid from one state to
another. Other factors can also contribute to
changes in the fluid. For example, heat can
cause a gas to expand and rise. Conservation
equations include these effects and provide a
generalized description of fluid motion.
These equations track changes in the fluid
that result from convection, diffusion, and
sources of the conserved quantity. Conservation equations are coupled, meaning that
changes in one variable (say,
the temperature) can give rise
to changes in other variables
(say, the velocity). In most
CFD products targeted toward
a particular application, such as
data center airflow, control of
the equations during the solution process is shielded from
the software user. Thus while it
is important to have an appreciation of the equations behind
the scenes, it is usually not
necessary to understand the
various techniques and controls
Figure 3 : A Cartesian mesh is used for most of the region and
for solving them.
around the two elements in the middle, while O-type grids are
used for the rectangular and circular elements on the sides
Conservation Equations
If a small volume of fluid in motion is considered, two changes to its shape will usually
take place. First, the fluid element will translate or rotate in space, and second, it will become distorted, either by a simple stretching
along one or more axes, or by arbitrary twist© 2011 Applied Math Modeling Inc.
Continuity
The equation of continuity is a statement of
mass conservation. To understand its origin,
consider the flow of a fluid of density 
through the six faces of a rectangular block,
as shown in Figure 4. The block has sides
of length x, y, and z and velocity compo3
WP107
Momentum
Similar logic can be used to construct equations for the conservation of momentum in each of the
three coordinate directions. Momentum conservation is more
complicated than mass conservation, however. Momentum
sources such as gravity and pressure gradients play a role. In addition, a term representing the
Figure 4: A computational cell of volume xyz and vediffusion of momentum is inlocity components u, v, and w in the x, y, and z direccluded. This effect is attributed to
tions, respectively
gradients in the velocity field.
nents u, v, and w in each of the three coordiConsider, for example, how a jet of water
nate directions. To ensure conservation of
spreads (momentum diffusion) after being
mass, the sum of the mass flowing through
injected into a pool where the water is at rest
all six faces must be zero.
on either side of the jet. Taking these effects
into account, the momentum conservation
 u out  u in yz  
equation in the x-direction for constant v out  v in xz  
density fluids takes the form:
(1)
 w out  w in xy   0
 u
u
u
u 

u
v
w

t

x

y

z


 
Dividing through by (x y z), the equation
can be written as:

u
v
w


0
x
y
z
  2u  2u  2u 
p

   2  2  2 
xi
y
z 
 x
 g x  Fx
(2)
Here  is the fluid viscosity, g is the acceleration due to gravity, and F represents any
additional forces that might be present. The
term multiplied by  is the diffusion term,
described briefly above as one that arises
from gradients in the flow field. The three
momentum equations (for u, v, and w components of velocity) along with the continuity equation are collectively called the Navier
-Stokes equations. The four equations are
used to solve for the four unknown variables:
or, in differential form for the more generalized case where the density changes in time
(t) and space:
 


 ( u )  ( v)  ( w )  0
t x
y
z
© 2011 Applied Math Modeling Inc.
(4)
(3)
4
WP107
three velocity components and pressure, even
though the pressure appears only in the momentum equation. Numerical solution methods are used to link the equations through the
density.
in other words, thermal transport through the
fluid is more diffusive.
Solving the Equations
If each cell in the domain has a point at its
center, the task of a CFD solver is to obtain a
solution for all of the relevant conservation
equations at every such point. If the domain
is characterized by 100,000 cells and there
are five conservation equations being solved,
that means that 500,000 solutions must be
obtained! These numbers are actually modest by today’s standards, where models with
tens of millions of cells are common and the
physics requires at least seven separate equations to be solved.
Turbulence and Energy
In most applications, more than just the Navier-Stokes equations are needed to describe
the fluid behavior. If the Reynolds number is
high enough, turbulence plays a role. The
primary effect of turbulence is to increase
diffusion. In the momentum equations, the
viscosity effectively increases in the presence
of turbulence. Thus water tends to act more
like honey. Various approaches to include
turbulence in the momentum equations are
available. The simplest approach is to add a
correction to the viscosity based on local
flow and geometric parameters. More involved approaches require the solution of
turbulence transport equations. Timedependent calculations can also be done to
capture the random fluctuations that result
from turbulent behavior. Many enclosed
room applications, such as data centers, use
the simpler approaches to include the effects
of turbulence.
There are a number of numerical methods in
common use by CFD solvers. In short, the
differential equations are converted to algebraic equations. To compute how well those
equations are balanced in each cell, the values of all variables at the cell faces must be
computed. The manner in which those face
values are computed differs for different solution approaches. The numerical method,
the density of cells, the cell topology, and the
nature of the flow field combine to impact
the accuracy of the resulting solution.
In addition to turbulence, temperature can
play a role in the flow field if the fluid density varies with temperature (as is the case if
the ideal gas law is used). A conservation
equation for thermal energy (analogous to
that for mass and momentum) must also be
solved. If the flow is turbulent, the transport
of thermal energy is impacted through the
conductivity of the fluid. Turbulence acts to
increase the fluid’s ability to conduct heat, or
© 2011 Applied Math Modeling Inc.
Residuals
The set of coupled equations described above
cannot be solved exactly. Instead, an iterative solution technique is applied. Roughly
speaking, a guess is made for each variable at
each point in the domain and the error in
each conservation equation is tabulated.
That is, the code takes stock of how closely
the left side of each equation matches the
right side. The cumulative error for each
5
WP107
equation is called the residual. The goal of
the solver is to force the residuals to decrease
with each step in the iterative solution process (Figure 5). Once the residuals have fallen
below some preset value, the solution is considered converged. Only results from converged solutions are meaningful.
contrast, is a surface, usually not planar,
where one variable has a constant value. For
example, an iso-surface of constant temperature can be created and then contours of another variable, such as pressure, can be
drawn on that surface. This type of display
can convey more complex information than
contours of a variable
plotted on a flat plane.
Figure 5: Normalized residuals are shown to decrease during the iterative solution procedure
Pathlines are another
popular choice for examining results. These trace
the paths of imaginary
continuity
momentum
fluid elements that are
released from one surface
temperature
(say, where air enters a
room). Pathlines can be
colored by other problem
variables, such as temperature, and they can
usually be animated. In
Figure 6, a raised-floor
data center is shown that
contains two rows of high
density racks in the lower
left. The racks are shaded by temperature
Displaying the Results
contours and pathlines, also shaded by temOnce a solution is converged, the results can
perature, are released from the exhausts of
be examined in a number of ways. Consider,
these racks. The results show that one side
however, the fact that there may be thouof the high density racks runs hotter than the
sands or millions of points of data throughout
other side. As a result, the air returning to
the domain. How can one make sense of it
one of the CRACs is hotter than that returnall?
ing to the other, a situation that places an uneven burden on the cooling system. An
One of the simplest ways to examine the reanalysis such as this can illustrate imbalance
sults is to do so one plane at a time. Once
in a data center that can result from equipcreated, a plane can be used to illustrate conment positioning or the presence of undertours of a variable or velocity vectors. A
floor obstructions.
plane can be created at some x, y, or z location or at the surface of an object, such as the
inlet side of a rack row. An iso-surface, by
© 2011 Applied Math Modeling Inc.
6
WP107
Figure 6: A CFD model of a raised-floor
data center showing temperatures on the
surfaces of the rack rows and pathlines of
return air, shaded by temperature
Overview of the Modeling Process
One of the most important uses of CFD is as
a planning tool. In this mode, an existing
design is modeled and if the predictions are
in good agreement with measurements, the
assumptions are assumed reasonable and the
model is altered to reflect one or more prospective changes. In the automotive industry, for example, such studies are done to test
body shapes for aerodynamic characteristics
before any metal is bent. The method is ideally suited for data centers where ongoing
changes to equipment are routine. Using
CFD, a number of scenarios can be studied
before any equipment is moved, changed, or
added.
available) first. Simple models are a good
starting point. In a data center, a simplified
model may use average heat loads in the
racks, neglect small piping in the supply plenum, and make use of a coarse mesh, for example. A comparison of the simple model
results with measurements will usually point
to discrepancies, and these can be remedied
by modifications to the model on the next
pass. A refined model will therefore include
more geometric detail and make use of a
finer mesh. The choice of what detail to add
can best be made by doing the first comparison with a simple model. For example, suppose the simple model neglects all of the
small pipes in the supply plenum and treats
all of the racks with average heat loads. If
the flow distribution through the floor tiles is
in good agreement with measurements but
certain hot spots in the room are missed,
The success of this approach depends on the
accuracy of the CFD model. To test for accuracy, it is a good idea to build a model and
compare it against measurements (where
© 2011 Applied Math Modeling Inc.
7
WP107
tain good agreement with
data. In some cases there may
not be good agreement with
data after multiple refinements, and the root cause(s)
should be investigated. Poor
agreement could be the result
of inaccurate assumptions
about the heat loads and flow
rates, especially if the fans in
the equipment are variable
speed fans and the model does
not take that into account.
Measurements can also be at
fault, so analysts should always be aware of the error
recorded.
Techniques for
Modeling Data Center
Equipment
CFD is used to simulate many
different types of applications,
and for each, the components
that are common to that appliFigure 7: A flow chart showing the modeling process,
cation are usually treated in a
starting with a simple model and refining it as needed
unique way. For example, a
before using it for analyzing prospective scenarios
light source might be modeled
as a volumetric heat source in the geometric
chances are that neglecting the small pipes
shape of the light housing or bulb. A fan
was an acceptable approximation but nemight be modeled as a planar area through
glecting the server-level detail in the racks
which a certain flow of air is specified. In
was not.
data centers, similar approaches are applied
to model the components that are familiar to
Once a refined model is in good agreement
facilities engineers: CRACs, racks of equipwith data, that model can be used as the base
ment such as servers and switches, UPSs and
case for modifications to the room. A flow
PDUs, for example. In this section, methods
chart illustrating this process is presented in
for representing data center equipment are
Figure 7. Note that more than one refinereviewed.
ment to the model may be necessary to ob© 2011 Applied Math Modeling Inc.
8
WP107
CRACs
Computer room air conditioners (CRACs) or
air handlers (CRAHs) are large rectangular
objects that draw in warm exhaust air from
the heat generating equipment in the room,
cool and condition it, and then return it to the
room for reuse. Under normal operation,
they have a fixed flow rate and are capable of
extracting a certain amount of heat from the
air. A CFD model of a CRAC usually consists of a rectangular block with one or two
fans used to move the air (Figure 8). In one
scenario the volume inside the CRAC is not
meshed so is not part of the solution domain.
A return fan is used to draw ambient air out
of the room while a second supply fan operating at the same flow rate is used to supply
the cooled air back to the room. Some models of CRAC employ two or three supply
fans and a single return fan. In another scenario, the volume inside the CRAC is
meshed so that the inside air flow can be
simulated. A heat sink is associated with the
fluid inside the CRAC and a single fan is
used at the supply side, modeled using a fan
performance curve.
the cold supply air in the room. For nonraised floor data centers, topflow or upflow
CRACs are used to supply cold air from the
top of the unit, with the return air passing
through grilles on one side of the unit near
the floor. Upflow CRACs are sometimes
used along with downflow CRACs in raised
floor data centers. Frontflow or in-row coolers are typically positioned between racks in
a data center. They draw warm air from the
rack exhausts, often the hot aisle in a hotaisle/cold-aisle arrangement, and produce
cooled air out the opposite side, adjacent to
the rack intakes. In-row CRACs may be
used with or without other types of CRACs
in the data center. Overhead cooling, often
referred to as supplemental cooling, is available in overhead ceiling-mounted or rackmounted units. As for in-row coolers, these
units act locally, circulating and cooling the
air from nearby racks of equipment. Despite
the many types of CRACs, they can all be
modeled using a hollow block and pair of
fans with matched flow rates.
Fans
Downflow, Upflow, In-Row, and Overhead
The position of the return and supply fans
depends upon the nature of the room and
type of cooling application. Raised-floor
data centers usually use downflow CRACs,
with the supply fans directing the cooled air
into the supply plenum beneath the floor and
the return fans on the top of the unit. The
cooled air enters the room through perforated
tiles in the floor. If a ceiling plenum is used
for the return air, these CRACs usually extend up to the ceiling plenum so that none of
the warm return air has a chance to mix with
© 2011 Applied Math Modeling Inc.
Figure 8: A CRAC consists of a hollow block,
a return fan, and one or more supply fans
9
WP107
Plug Fans
While a great number of CRACs discharge
cooled air in a direction normal to the CRAC
boundary (straight down for downflow
CRACs, for example), plug fans are gaining
in popularity. For raised-floor data centers,
plug fans extend below the lower surface of
the CRAC into the supply plenum where
they discharge air radially. Such fans can be
modeled using a cylindrical fan in CFD.
That is, the fan surface is not a plane but the
wall of a cylinder. The volume inside the
cylindrical fan is not meshed and is not part
of the calculation domain.
(Q) as a function of return temperature. The
relationship in Equation (5) is used, but the
value for Q changes for different return temperature ranges.
Racks
Whereas CRACs extract heat from the air,
racks and cabinets full of equipment add heat
to the air that passes through them (Figure 9).
Servers are rated by their heat load. Assuming a certain temperature rise across a server
with a given power rating, Equation (5) can
be used to compute the flow rate required to
maintain that temperature difference. For
example, to maintain a 20°F temperature rise
across a server, about 150 CFM of cooling
air is needed for every kilowatt of power.
Racks can be modeled using their total heat
load or by modeling the individual components with blanks and/or gaps in between.
Whether a total heat load or equipment-level
detail is used, the volume inside a rack need
not be modeled. Instead, a matched set of
flow boundary conditions can be used that
ensures that the same flow rate enters and
leaves the rack (or equipment) and that between those matched boundaries a certain
temperature rise will occur. The temperature
rise can be specified directly or can be the
result of a specified heat load. Note that the
volume inside a rack can be meshed and
modeled, if desired, as in the case of the single-fan CRAC. This choice would be appropriate for analysts looking to model the temperature distribution inside individual servers
for a given set of external conditions, for example.
Cooling Methods
While it is a simple matter to specify the
temperature of CRAC supply air, most
CRACs operate in a more sophisticated manner. In practice, the heat exchanger may cycle on and off so that a certain thermostat
temperature is maintained at the CRAC return. In CFD, this can be accomplished using an iterative calculation of the following
equation:
Q  m cP (TR  TS )
(5)
Where Q is the amount of heat removed from
 is the mass
the air (in SI units of W), m
flow rate of air through the unit (kg/s), cP is
the specific heat of air (J/kg-K), and TR and
TS are the return and supply temperatures (in
K), respectively. An iterative CFD calculation periodically adjusts the supply temperature (TS) until the return air temperature
matches the thermostat temperature. For
many CRACs, the manufacturer supplies
performance data for the cooling capacity
© 2011 Applied Math Modeling Inc.
Racks can be modeled with air flow from
front to rear, front to top, bottom to top, or
10
WP107
side to side. Flow entering the front can also
be split to pass out the rear and top of the
unit. Flow exiting from the top of a rack is
an acceptable strategy if the rack is modeled
with an average heat load. For a rack modeled with individual servers, it is not good
modeling practice, because it is difficult to
estimate how the air from each of the individual servers contributes to the top exhaust
and the cooling of the other equipment.
However, if a channel of exhaust air from the
back of a rack of equipment is included in
the model, the front-to-top flow strategy can
be represented.
best to model the rack using an average heat
load. Otherwise, the air drawn up from the
plenum will mix with the cold air entering
the server closest to the floor only.
PDUs and UPSs
In addition to racks, PDUs and UPSs are heat
-generating components that require cooling.
In raised-floor data centers, the cooling air is
drawn through the floor into these units
(Figure 9). Alternatively, cold air is drawn
into the unit from side grilles. The heated
exhaust air exits through the top of the unit.
These components can be modeled like
racks, using a matched set of openings with
the same flow rate applied to both boundaries
with heat addition between. They can also
be meshed and given a volumetric heat load
in the interior. For this modeling method,
openings at the bottom and top will allow
flow to be drawn up and through the device
as a result of natural convection.
In raised-floor data centers, underfloor cable
trays that feed power to the racks do so
through cutouts in the floor, which are usually under the racks and characterized by
some percent open area. Openings in the
floor allow supply air to mix with the air entering through the rack inlets. In CFD simulations, if cable cutouts are important, it is
Figure 9: Racks (right) and PDUs (left) consist of hollow
blocks with openings (red or pink) for air flow and a heat
gain
© 2011 Applied Math Modeling Inc.
11
Perforated Tiles and Ceiling
Grilles
Perforated tiles (Figure 10)
allow the cold air in the supply
plenum to enter a raised-floor
data center where needed, usually in front of the rack rows.
These tiles are characterized by
some percent open area which
causes a drop in pressure.
They also serve to straighten
the flow as it passes through.
To accomplish these modeling
goals, tiles are typically modeled using a thin porous region.
The porous region (a volume)
is assigned loss coefficients for
WP107
of air can be uniform across the diffuser or
have some prescribed profile, such as high in
the center and low at the edges.
To model a duct using CFD, the volume inside the duct can, but need not be meshed
and included in the calculation. If the duct
volume is not included, assumptions can be
made to compute the properties of the air exiting from the diffusers. First, the air from
the participating CRACs can be assumed to
be well mixed, so the temperature at the diffusers can be computed using a weighted average of that produced by the CRACs. Second, the total mass flow produced by the
CRACs can be divided among the diffusers
and scaled to match the area of each. An assumption of equal flow rates is good if the
diffusers are of similar size and not positioned in the middle of recirculation zones
inside the ducts. Diffusers immediately following right-angle bends in the ductwork
might not satisfy this requirement as well as
those in the middle or at the end of a long
section of ductwork. By including the duct
volume in the calculation, the temperature
and flow rate distributions for all participating diffusers in a duct system will be computed as a result of the flow inside the ducts.
Figure 10: Perforated tiles (purple) are threedimensional objects that can straighten the
flow as it passes through
the three component directions. By making
the loss coefficient much smaller in the direction normal to the floor, flow from the
supply plenum will tend to align with that
direction as it passes through the region. Cable cutouts on the floor of a data center – not
underneath racks - are usually modeled in the
same manner.
Ceiling grilles are partially open tiles in the
ceiling. They cause a pressure drop for the
flow as it enters the ceiling plenum, but they
do not necessarily straighten the flow. Thus
they can be modeled by a zero-thickness
“face” or area that has a loss coefficient that
gives rise to the appropriate pressure drop.
What to Include in a Data Center Model
For the purpose of the CFD model, a data
center is a closed environment with very little leakage of air or heat into or out of the
room. This means that all heat produced by
equipment in the room must be removed by
the cooling units. All of the flow demands of
the equipment should (but need not) be met
by the cooling units as well. Thus when
choosing what items to include in the model,
Ducts and Diffusers
In non-raised floor data centers, and in some
raised-floor data centers, ductwork is used to
transport cold (or warm) air as needed. The
air is released to the room through diffusers,
which can be set to direct the air at an angle
or straight down into the room. The velocity
© 2011 Applied Math Modeling Inc.
12
WP107
the most important are the major heat generating and removing ones, and any objects
that contribute to the air circulation: all
CRACs, CRAHs, racks of equipment, power
generating stations, floor tiles, ceiling grilles,
ducts and diffusers.
top-sized control stations may be contributors to the heat load in the room, but their
role may be small compared to other equipment. If people are present most of the time
in the data center (seated at a desk), their
contribution to the heat may also be included. Finally, windows with minimal
shading may also play an important role.
The next level of importance consists of objects that impact either the flow or mixing of
air in the room. This group includes baffles
that are used for cold or hot aisle containment, turning vanes or other baffles in the
plenum used to direct the supply air, plenum
blockages and inert (non heat- or flowgenerating) objects in the room. While baffles can be set up to have some percent open
area, impermeable baffles should be used
with care. If baffles are used to section off
an office or closet that does not contribute to
the air flow in the room, the CFD solver may
have difficulty, since the solution domain
should be comprised of only a single connected volume of air. Side rooms that are
not participating in the overall flow in the
data center should be blocked off using solid
objects such as blocks or slab-to-slab columns. Overhead and underfloor cable
trays can impact the flow of air but may
be deemed unimportant by the analyst,
depending upon their size and position.
Of less importance are small objects,
especially in large data centers. Small
pipes in the plenum (less than 3” in diameter, for example) fall into this category. Tables and chairs, filing cabinets
and gas cylinders may be deemed too
small or too few to make a difference in
the flow field. Small heat generating
objects, such as overhead lights or lap© 2011 Applied Math Modeling Inc.
Examples
Two cases are presented in this section to
demonstrate how CFD can be used to assess
a data center. The first case is a raised-floor
data center and the second has a non-raised
floor and makes use of ducts and diffusers to
distribute the supply air.
Raised Floor Data Center
The 1800 sq. ft. raised-floor data center
shown in Figure 11 has an assortment of
racks (light gray in the middle of the room),
eight CRACs (dark gray along the walls),
five PDUs (dark gray but smaller than the
CRACs), and a ceiling plenum return. The
downflow CRACs are each modeled with a
cooling capacity and flow rate unique to the
Figure 11: An isometric view of a
raised-floor data
with a ceiling return
13
WP107
unit, and each is set with a thermostat temperature of 75°F.
Some of the racks are modeled
using an average heat load while
others cannot be approximated in
this manner so are represented as
a collection of servers, blanks and
gaps. Each component in the
racks is characterized by a heat
load, an anticipated temperature
rise, and a flow rate, controlled by
a fan. The PDUs are assumed to
draw air from the plenum and exhaust it out the top. Air is drawn
up into the room through perforated floor tiles, all of which have
a 25% open area.
Figure 12: Contours of temperature on rack and
server inlets for the raised-floor data center
The room does have some design flaws,
however. First, the total cooling capacity of
the CRACs is too close to the total heat load
in the room. Thus if one of the CRACs were
to fail, the others would have difficulty maintaining a 75°F thermostat temperature for the
return air. This problem would need to be
remedied by replacing one of the CRACs
with a different unit. CFD could be used to
determine the best choice of CRAC to replace. Second, the absence of underfloor obstructions gives rise to a large recirculation
of air in the plenum, as is shown in Figure
13. Often, recirculation patterns such as this
cause negative pressures in the “eye” and
result in the flow of air from the room into
the plenum. Had the power demands on the
racks in the middle of the room been greater,
enough supply air might not have been available. Baffles can be used to redirect supply
air in a plenum so that recirculation patterns,
especially large ones, don’t develop and adversely impact the distribution of air through
the perforated tiles.
A summary of the heat loads and cooling capacity in the room suggests that there is 6%
excess airflow and 3% excess cooling delivered by the CRACs. Despite the fact that the
cooling capacity cuts very close to the heat
load, the detailed CFD results indicate that
all racks pass the recommended ASHRAE
guidelines for inlet temperature falling below
80.6°F (Figure 12). There are a few reasons
for this success. First, all perforated tiles are
positioned directly in front of rack inlets.
Second, there are no underfloor obstructions
to divert the flow away from where it is
needed. Third, the tiles offer uniform resistance to the flow because they have the same
percent open area. The results support this
finding: the average flow rate through the
tiles is 681 CFM, with a maximum deviation
of 14% with most tiles falling well within
8%. While the CRACs are not all the same,
all are able to meet their target thermostat
temperature to within a few degrees.
© 2011 Applied Math Modeling Inc.
14
WP107
passes the recommended
guidelines (64.4°F to
80.6°F). Despite the fact
that all of the diffusers
introduce cooling air at
60°F, the average maximum temperature delivered to the rack inlets is
over 85°F. The reason
this design fails is because the exhaust air
Figure 13: Pathlines in the supply plenum, shaded by
from the racks circulates
speed; the darker lines represent the more slowly moving
over and around the rows
supply air, most of which is shown after it enters the room
and mixes with the supply air on the opposite
side. That is, even though there are cold
Non-Raised Floor Data Center
aisles below the diffusers, the aisles are short
A non-raised floor data center with four
and wide, allowing the rack exhaust air to
CRACs in a middle aisle is shown in Figure
infiltrate the cold zone and degrade the cool14. The data center is 800 sq. ft. with a heat
ing air. Furthermore, the CRAC returns are
density of about 81 W/sq. ft. The rack heat
visible to nearby supply ducts, a design that
loads vary from 500 W to 4 kW. The CRAC
encourages the cooling air to short-circuit
returns are on alternating sides of the units;
back to the CRAC returns before being fully
two face the right side of the room and two
face the left side. Supply air
mixes in a main duct above the
CRACs and is dispensed
through diffusers to cold aisles
between the rack rows. A summary of the equipment in the
room indicates that the CRACs
provide 26% excess air and
27% excess cooling than the
racks demand.
While the room appears to have
adequate cooling, the CFD results paint a very different picture. While all of the racks pass
the allowable ASHRAE thermal
guidelines (59°F to 90°F), none
© 2011 Applied Math Modeling Inc.
Figure 14: A non-raised
floor data center with a
center aisle of top-flow
CRACs and ducts and diffusers to deliver the supply air to the rack rows
15
WP107
Data Center Modeling Best Practices
utilized by the racks. In Figure 15, pathlines
released from the rack outlets show how the
exhaust air travels over the tops and around
the sides of the rack rows so that it can be
drawn in by the intake fans on the opposite
side. The mixing of exhaust air with the
cooler supply air at the inlets raises the temperature of the rack cooling air, especially
near the tops and sides of the rack rows.
In recent years, several articles have been
written on Best Practices for CFD modeling.
These recommendations are typically targeted toward CFD analysts who have full
control over the mesh generation and solver
options. Even so, users of applicationspecific software should also be mindful of
certain practices that ensure success. A short
summary follows.
It is interesting to note that a CRAC failure
analysis done on this room shows that the
performance of the CRACs is the same or
better when each one of the CRACs is turned
off. This unexpected result is due to the fact
that a disabled CRAC will not short-circuit
the supply air from the nearest diffuser directly to the CRAC return. This allows more
cold air to be available for the pair of rack
rows nearest the diffuser(s).
Avoid putting too much detail into a
model on the first pass. Get a general
sense of the flow in the room before
complicating the model with too many
objects or details. Remember that a CFD
model is different from a CAD model,
where more detail may be better.
 Avoid small gaps. If a CRAC is 2” from
a wall, there is probably no need to
model the air between the CRAC and
wall. Move the CRAC so that it is flush
against the wall in such
cases.
 Start with a coarse
mesh solution and progress to a fine mesh only
after confirming that the
results make sense.
 Avoid separated regions of air. If adjacent
rooms do not contribute
to the overall flow in the
data center, block them
out of the model using a
solid object.
 Understand how to
Figure 15: Return pathlines, shaded by temperature, show some
use the software and its
of the warmed exhaust air being entrained by the rack inlets, eflimits. Go through the

fectively raising the temperature of the cooling air
© 2011 Applied Math Modeling Inc.
16
WP107
recommended training before starting
to use the product.
centers as the loads vary. Dynamic CFD
simulations will be able to identify the cooling distribution among CRACs and air handling units as server loads rise and fall with
varying activity. In this manner, CFD will
contribute to the control system that governs
the overall facility cooling, activating certain
units while deactivating or simply ramping
down others. The potential feedback from
such a system will not be instantaneous, but
will be sufficient for the timescales typical of
these fluctuations.
By adhering to simple guidelines such as
these, most common modeling problems can
be avoided. While early CFD software products were most often used by specially
trained CFD analysts, today’s applicationspecific products can be used successfully by
engineers, designers, and technicians with a
wide range of backgrounds. When choosing
a product, a prospective user should consider
ease of use, capabilities to suit a given set of
needs, and availability of technical support.
For example, if rack-level detail will be an
important component of a planned analysis, a
product should be chosen that includes this
feature.
Suggested Reading
Two books provide excellent, readable summaries of CFD methodologies. Numerical
Heat Transfer and Fluid Flow by Suhas V.
Patankar (Hemisphere Publishing Corporation, 1980) is the early standard that has
helped guide the growth of the field. An Introduction to Computational Fluid Dynamics,
The Finite Volume Method, by H. K. Versteeg and W. Malalasekera (Prentice Hall,
1995, 2007) contains more recent examples
and techniques that have been developed
since Patankar’s book was published. While
Patankar focuses on Cartesian (or cylindrical) meshes in his examples, Versteeg and
Malalasekera describe other mesh topologies
as well. In addition, they provide discussions
of some sophisticated physical models such
as combustion and radiation.
Conclusions and Future Trends
With data center managers and designers
looking for innovative ways to cut energy
costs, CFD is quickly becoming an essential
tool of the trade. The flow field in a room
can expose problem areas that don’t show up
in simple heat and flow balance calculations.
In addition, trial layout modifications can be
quickly evaluated using CFD prior to moving
or replacing equipment. CFD is also useful
for illustrating the division of labor among
the cooling units in a data center and for illustrating the room conditions should one or
more of them fail.
In the future, as computing speeds increase
and hardware costs drop, CFD will become
more integrated in data center monitoring
systems. Rather than just be used for static
assessments of data centers, CFD will be
used in real time for dynamic models of data
© 2011 Applied Math Modeling Inc.
17
WP107