A Customer Success
from the Experts in
Business-Critical ContinuityTM.
Background
Sandia National
Laboratories
Sandia National Laboratories supports United States national
security by providing science-based solutions to address threats
from terrorism, cyber attacks, uncertainties over energy security
and infrastructure and other challenges. To support their extensive
research and development initiatives, Sandia’s IT managers are faced
with consistently updating and implementing the newest, fastest
and most reliable supercomputer technologies. When the laboratory
sought to deploy a new supercomputer, the Sandia’s IT Facility
Infrastructure and IT professionals convened to advance their approach
to high-density cooling to meet Sandia’s critical need for high
availability, while improving energy efficiency and processing speed.
Case Summary
Location: Albuquerque, New Mexico, USA
Products/Services:
• Liebert XDR rack door cooling module applied
as the Sun Cooling Door
• Liebert XDP refrigerant pumping units
Critical Need: Support high-performance computing needs
while maximizing energy efficiency and delivering maximum
availability.
Results
Sandia National Laboratories has
developed science-based technologies to
support U.S. national security since 1949.
This government-owned and contractoroperated facility supports the U.S.
Department of Energy’s National Nuclear
Security Administration and has major
research and development responsibilities
in national security, energy and
environmental technologies and economic
competitiveness.
Applied a rack door, refrigerant-based passive cooling system
capable of removing 90 percent of server heat load before
it leaves the rack.
Reduced energy costs by 77 percent by replacing 140 racks
operating at 13 kW per rack with 36 racks operating
at 32 kW per rack.
Minimized the organization’s environmental footprint by
incorporating innovative high-density cooling technologies.
The Situation
At a time when terrorism and other threats are top
concerns, Sandia National Laboratories, which develops
science-based technologies to support national security,
continuously works to implement cutting-edge data
center infrastructure strategies to support the IT systems
its laboratories and customers depend on. With data
center usage rising, Sandia needed to deploy a new
supercomputer. Sandia’s IT Facility Infrastructure and IT
professionals decided to use the opportunity to assess the
reliability, availability and efficiency of their cooling systems,
and employ new, advanced technologies to optimize energy
efficiency and further address the organization’s critical
need for High Performance Computing.
Sandia’s effective but aging supercomputer, named
“Thunderbird,” consisted of 140 racks operating at full
capacity of 13 kW per rack. The five-year-old system could
process data at a rate of 50 teraflops and relied on 518
tons of cooling, fed by 12.7 million gallons of water each
year. With limited floor space, the system had little room
for growth, and given the critical nature of its operations,
Sandia could allocate only a day and a half of planned
downtime throughout the year for scheduled maintenance.
“We were looking for a power and cooling infrastructure
that would support our customers’ needs as well as
have room to grow in the future,” said Dave Martinez,
Data Center Facilities Coordinator at Sandia National
Laboratories. “There are a lot of people throughout
the world who access our data center for their critical
applications, so we felt the need to implement a faster,
more reliable infrastructure to maintain the high level of
availability and to improve efficiencies at the same time.”
The Solution
IT Systems and IT Facility Infrastructure professionals worked
together to design and deploy an innovative, high-density
IT system and supporting data center infrastructure that
could fit within the existing facility and meet current and
future IT needs, while providing the flexibility to support
future growth. Named “Red Sky,” the new supercomputer
dramatically increased the density of the space by
consolidating 140 racks to just 36 racks, operating at 32
kW per rack. To support this virtualized and highly-dense
computing environment, the team needed a leading edge,
close-coupled cooling infrastructure to accommodate
extreme compute densities in a cost-effective manner.
Liebert XDP pumping units act as an isolating heat
exchanger between the building chilled water source
and the circulating refrigerant. The compressor-less
units operate above dew point, providing 100 percent
sensible cooling.
“Supercomputing is invaluable,” said Martinez. “That is why
we like to do innovative projects with a twist to support our
supercomputing needs. We reviewed bids from three or
four different vendors to identify the most energy-efficient
cooling solution, but the one that was the most attractive
was Emerson Network Power’s Liebert XDR solution, applied
as Sun Cooling Doors.”
The Liebert XDR solution was the first refrigerant-based, 100
percent passive cooling rack door system on the market.
It requires no fans, and utilizes a pumped refrigerant that
is 70 percent more efficient at removing heat from the IT
rack, while eliminating the possibility of equipment damage
caused by water leakage. The Sun Cooling Doors employed
by Sandia are a private-label Liebert XDR door, which easily
attaches to the back of most racks. This system provides 2.3
times the capacity of competitive passive cooling systems.
In the Red Sky deployment, air inside the racks reaches
temperatures as hot as 130 degrees Fahrenheit, but upon
passing through the Liebert XDR cooling doors, is discharged
into the next row at 72 degrees. The Liebert XDR solution
removes 90 percent of the IT heat load before it leaves the
rack. The doors leverage the server fans to move air across
the micro-channel coil, eliminating cooling unit fans and
reducing electricity costs.
Since the expelled air is cool enough to feed a cold aisle,
a laminar airflow configuration—circulating air in parallelflowing planes—is employed to further enhance cooling
efficiency. Perforated floor tiles are only required to support
the first row of racks, with the 72-degree air expelled from
the first row feeding into the front of the next, and cascading
through each rack row. Air discharged back into the room is
cooled to the point where the impact on the room is close to
room-neutral.
Liebert XDP pumping units act as an isolating heat exchanger
between the building chilled water source and the circulating
refrigerant. The compressor-less units operate above dew
point, providing 100 percent sensible cooling.
“From the Liebert XD units on the chilled water side, we took
the return water and supplied it to our CRAC units in the
room, raising the Delta-T up, also enabling us to bring the
sensible cooling up even more,” said Martinez. “So we were
already really efficient with the Liebert XD systems, and then
we raised the sensible cooling up in the room by delivering
warmer water to our CRAC units.”
The Results
Since deploying the Red Sky supercomputer, Sandia has
reduced both energy consumption and carbon footprint
associated with the supercomputer by more than 75
percent, thanks in large part to the energy and water
savings provided by the Liebert XDR solution. Annual
water consumption needed to support cooling for the
supercomputer has dropped by 5.7 million gallons, while
chiller plant consumption and cost have fallen 37 percent.
By cooling heat loads directly at the source—before the hot
air ever enters the room – Sandia has significantly enhanced
the efficiency of its cooling systems while simultaneously
accommodating even more computing power, within a
smaller data center footprint. While Sandia’s Thunderbird
supercomputer could handle 13 kW per rack at full
load, Red Sky can accommodate up to 32 kW per rack.
Simultaneously, whereas Thunderbird required 518 tons of
cooling to manage temperatures within the relatively highdensity system, the new Liebert XDR solution requires just
328 tons of cooling to accommodate nearly six times the
power capacity.
The Red Sky supercomputer uses a laminar airflow
configuration, with perforated floor tiles required only
in the first row of racks. The 72-degree air expelled
from the Liebert XDR cooling doors feeds into the front
of the next rack, and cascades through each rack row.
Efficiently cooled using the Liebert XDR solution, Sandia’s
Red Sky supercomputer ranks among the world’s top 15
fastest supercomputers. Previously, the entire Thunderbird
supercomputer could process 50 teraflops using all 140
of its racks, but Red Sky can process approximately 10
teraflops within each rack, running at 433 teraflops, or 433
trillion calculations per second.
Leveraging Red Sky’s speed and capacity, Sandia partnered
with the National Renewal Energy Laboratory (NREL), which
specializes in the research and development of renewable
energy and energy efficiency technologies. Using the Red
Sky supercomputer, NREL can now generate its research
results within four to six weeks, compared to previous
efforts, which consumed four to six months.
“Sandia National Laboratories are critical to our nation’s
path forward,” said Martinez. “We continue to make
innovative adjustments to the new infrastructure because
we have the capacity to do so, and because we want to
continue to be at the leading edge of the computing world.
For more information on Emerson Network Power and
Liebert solutions, visit: www.liebert.com
For more information on Sandia National Laboratories, visit:
www.sandia.gov/
Emerson Network Power.
The global leader in enabling Business-Critical ContinuityTM.
AC Power
Connectivity
Embedded Computing
Embedded Power
DC Power
Infrastructure Management & Monitoring
EmersonNetworkPower.com
Outside Plant
Power Switching & Controls
Precision Cooling
Racks & Integrated Cabinets
Services
Surge Protection
Emerson, Business-Critical Continuity and Emerson Network Power are trademarks of Emerson Electric Co. or one of its affiliated companies. ©2011 Emerson Electric Co.
CH-00026 (10/11)