Energy Efficient Strategies for
High Density Telecom Applications
Alan M. Lyons, David T. Neilson and Todd R. Salamon
Bell Laboratories
Why do we care about energy efficiency?
Transistor
Package
Circuit Pack
Shelf
Cabinet
Central Office
Environment
Increasing Thermal Energy
Greener Networks
 Overall power dissipation
 Network operator power dissipation is 1% of many countries energy
consumption
 Pressure to reduce network power consumption while still growing network
capacity and functionality
Scalability
 Power density in racks of communications equipment is reaching practical
limits.
 Makes cost-efficient scaling of telecom networks difficult
All Rights Reserved © Alcatel-Lucent 2008
Thermal Management: Motivation
Telecom equipment vendors face the toughest thermal challenge and must lead
the electronics industry.
1. Increase equipment functionality
and density

Meet future power density needs –
exceeding 15,000 watts/cabinet

Maintain high reliability and meet
acoustic noise limits.
2. Reduce carbon footprint and OPEX
for our customers


Telecom consumes 3% of U.S.
electricity
Major Telcos spend > $1 Billion USD
annually on electricity
3. Meet or exceed regulations

NEBS, ETSI & guidelines for energy
consumption and reduction
Source: ASHRAE Society Handbook, 2005
How can cooling technologies be extended to
meet future power density needs?
4. Differentiate ALU’s products
All Rights Reserved © Alcatel-Lucent 2008
What are we doing in Thermal Management Research?
Extending the Limits
of Air Cooling
Thermal
Management
Research
Thermal Interface
Materials
Increasing Cooling
Efficiency
Liquid Cooling Cabinet
Architectures (with CTO)
Heat Sinks &
Vortex
Generators
Waste Heat Recovery
Thermo-Electric Modules
Fan Reliability
Vapor Chambers
All Rights Reserved © Alcatel-Lucent 2008
Liquid Cooling Architectures for the Central Office
From: dispersing waste heat into
Central Office air
To: transferring waste heat into a liquid
coolant and piping outside the CO
Motivation:
Thermal Density:
 Heat capacity of air limits total heat that can be removed at air flow rates that meet
acoustic noise specifications.
 Liquids have 103 higher heat capacity compared to gases
Energy Efficiency:
 Pumping Power required for air is >> than that for water
 Hot air leaks into cold aisles. Air must be chilled below ambient temperature to insure
sufficient cooling capacity
Challenges:
All Rightsincrease
Reserved © Alcatel-Lucent
2008
Insure system reliability, minimize costs,
thermal
capacity from 15 to 50kW
Design Example:
Heat Exchangers Above Each Shelf
Approach:
Front View
Perspective Cut-Away
Cooling
Liquid
Supply
Heat
Exchangers
 Heat-pipe based heat exchangers above
electronics extract heat from air
 Vapor in heat pipe transports heat through
cabinet walls into water side heat exchanger
 Chilled water pumps heat to outdoor cooling
tower.
Advantages:
Cabinet Level:
 Insures that cooling air is at ambient
temperature
 Reduces number of fans
Equipment
shelves
 Isolates water from electronics
Room Level:
Ambient Airflow
Heated
water
return
 Minimizes room air handlers – eliminates
large blower motors
 Eliminates need to chill air below ambient
temperature. No opportunity to intake hot
room air
All Rights Reserved © Alcatel-Lucent 2008
Sealed Cabinet Advantages
 Higher thermal densities can be accommodated
both in the cabinet and Central Office/Data Center
Heat Pipe Architecture
(Open System)
Heat Pipe Architecture
(Closed Recirculating
Air System)
Cooling Water
Supply
 Improved real-estate utilization
 Closer spacing of cabinets
 Eliminates need for raised floors
Finned
Heat Pipes
 Higher reliability - ambient pollutants & dust not
continuously drawn across equipment
 Quieter operation at higher fan speeds. Closed
cabinet reflects noise and enables acoustic foam
installation.
 Energy savings
• Eliminates need to under-cool air. Cooling
is provided directly to the cabinet
• Large air volumes need not be moved
across large distances.
• Enables use of low-cost cooling towers and
potential off-peak cooling.
Hot Airflow
Cold Airflow
All Rights Reserved © Alcatel-Lucent 2008
(cut-away view)
Enhanced Cooling in a Sealed Cabinet Using an
Evaporating-Condensing Dielectric Mist
Acoustic foam
Recirculating air
Mist from collectors
Hot water
pumped to atomizer
Heat pipe
Mist condenses on
Shelf 2
heat pipes and falls by
Pump
gravity into collector
Chiller
unit
Cold water
Heat pipe
Shelf 1
Atomizer
∞ Fan ∞
Large droplets of mist from
atomizer directed into circuit packs
Cabinet Level
Circuit Pack Level
Heat Sink Level
 Objective: Limit temperature rise of air flowing through circuit packs by injecting
atomized mist to increase its effective sensible heat capacity.
 Approach: Inject atomized HFE7000 (environmentally benign dielectric fluid)
upstream of circuit packs and/or high power component’s heat sinks. Condense
vapor on finned heat pipes between shelves and recirculate it.
All Rights Reserved © Alcatel-Lucent 2008
Calculated Optimum Droplet Diameter for Complete Evaporation at
Outlet of a 32 mm Wide x 32 mm Long x 13 mm High Heat Sink and
Cooling Capacity and Associated Parameters as Function of Mist Loading
Parameter
Mist loading
= 0.01 % by
volume
Mist
loading =
0.1 % by
volume
Total mass flow rate (gm/s)
Length of side of unit cell (in terms of droplet
diameter)
0.435
17.3
0.871
8.04
Optimum droplet diameter (μm)
RH after complete evaporation (constant
volume, constant pressure) (%)
Residence time in heat sink (msec)
Acceleration pressure drop (Pa)
Evaporative cooling (W)
83.2
(2.8, 2.75)
74.9
(28.1, 24)
31.5
0.0355
6.87
22.5
0.67
68.7
 Enormous potential for cooling enhancement (about 70 W) for modest pressure
drop (< 1 Pa) and relative humidity (<25%) increases
All Rights Reserved © Alcatel-Lucent 2008
Extending the Limits of Air Cooling
Initial approach is to extend the limits of air cooling while maintaining full
NEBS/ETSI compliance…
…to allow thermal density increases without exceeding noise standards
Schematic of
Microprocessor
Cooling
Thermal
Resistance Stack
Heat Sink
Improvement Targets:
 IC Package to heat
sink interface
0.36 K/W
60%
TIM2
IHS
0.15 K/W
TIM1
Chip Carrier
Silicon Die
0.34 K/W
Thermal Resistance = (Tjunction – Tambient)/Power
All Rights Reserved © Alcatel-Lucent 2008
 Fin to air interface
 Heat Spreading in
the base
void
Thermal Interface Materials (TIMs)
conventional TIM
Conventional TIMs: polymer-metal composites
TIM test rig
 Limits:
 Thermal path limited by particle-particle point
contacts: typical values 2-10 W/mK
 Voids reduce reliability and achievable thermal
properties
TIMs research program
 Measurements:
 Designed and built a world-class test rig to verify
and compare commercial materials
 Maximize conductivity while minimizing assembly
force.
 First Results
 Thermal conductivity >4.5 W/mK with >60%
compression (>1.5mm compliance)
 Large experimental/modeling space to explore
All Rights Reserved © Alcatel-Lucent 2008
Pressure (MPa)
 Developing new micro-textured TIM with higher
thermal conductivity, compliance and reliability
Effective Thermal Conductivity (W/mK)
 Materials:
Effective conductivity and loading
pressure versus compression
Compression (mm)
Energy Harvesting Using Thermoelectrics
Goals:
 Reduce power required to cool
equipment by > 20%
 Convert waste heat into
electricity
Approach:
 Carnot Efficiency is inherently
too low for CMOS devices Target IC’s with junction
temperatures > 300oC for
adequate efficiency
Thermoelectric Module approach
die
die
Vapor Chamber
Hot
Thermoelectric Module
Cold
 examples: SiC, GaN
 Direct approach: Thermoelectric
Modules to convert waste heat
directly to electricity
Challenge:
 Efficiency of Thermoelectric
Module is low – research required
for new materials and designs.
All Rights Reserved © Alcatel-Lucent 2008
V
Electricity
generated
by Thermoelectric
Module
Network routers are responsible for much of ICT energy growth
Overall Network power growth
 In 2006 in Japan, IT and Communications equipment consumed about
45TWh or 4% of the total electricity generated or 1% of the country
total energy consumption.
 About ¼ of this i.e. 1% of power consumption is network operators
Information and Communications
Technologies Energy Usage
Packet switching
responsible for much
of growth rate
Energy
usage
[TWh]
Japan's Ministry of Internal
Affairs and Communications
Study Group on ICT System
and Network for Reducing
Environmental Impacts,
March 2007
Year
All Rights Reserved © Alcatel-Lucent 2008
Scalability:
Electronic Router Power Density
Historical trend of x2 capacity every 18
months
 Not sustainable because of thermal
density
 Tb/s router today with 10-15kW is at the
limit
 Future growth thermal density limited
 Shortfall of 30 fold in capacity by 2015
compared to historical trend for single
rack router
Where the power dissipation is in core
routers
 2/3 of power associated with layer 3
function
16%
Core Router
Power
Consumption by
Layer
L1+L2
L3
Switch
 Packet Forwarding Engine dominates :
–Processing IP headers for destination
and quality of service queuing
 Eliminating L3 function in core would
allow more scalability and lower overall
power
All Rights Reserved © Alcatel-Lucent 2008
22%
62%
Current Core Network - IP over DWDM
Legend
Core IP router
DWDM terminal
or fixed OADM
Core IP routers connected by point to point DWDM
 All switching at a node uses router
 Router capacity scales as node capacity
 Traffic has multi hop routing
 Requires total router capacity in network to be larger than traffic by the average number of
hops
All Rights Reserved © Alcatel-Lucent 2008
Network Evolution to Transport switching
Packet Transport switch
Core IP router Layer 3
OCh
TMPLS Layer 2 Switch
OCh
OCh
OCh
DWDM terminal,
ROADM Layer 1 Switch
Transparent flexible optical
switching
Reduce switching requirement for Layer 3 IP routers
 Layer 1 ROADM: Wavelength channels bypass nodes
reduce number of hops for electronic switches
 Layer 2 TMPLS: Packet switching network for transporting
packets
 1/3 of the power of Layer 3 switching
 Layer 3 Routers only handle service layer not transport
switching
 Primarily edge/service layer function not core switching
 Does not require rapid scaling of core IP routers
 Uses lower-power switching elements
All Rights Reserved © Alcatel-Lucent 2008
OCh
Architecture change is already
happening: Qwest runs it’s
Juniper T640’s as MPLS routers
in the core (Poll: OFC 2008
plenary)
However they are still
using core routers
Summary
Enhanced cooling using a
dielectric mist
Why do we care about energy efficiency?
 Financial and environmental cost of energy usage
 Practicality of cooling high-energy-density equipment
Thermal Management Solutions
 Liquid cooling solutions for cabinet architectures
 Extending the limits of air cooling
 Thermal interface materials (TIMs)
 Energy harvesting
Energy-efficient switching architecture
 Thermoelectric modules
Network Evolution to Reduce Load on Layer 3 Routers
 Combine Layer 1 ROADM and Layer 2 TMPLS to
reduce energy usage of Layer 3 Routers
 Expected reduction of 30 to 50 percent
All Rights Reserved © Alcatel-Lucent 2008
OCh
OCh
OCh
OCh
Acknowledgements
Vaihbav Bahadur
Martin Cleary
Cormac Eason
Ryan Enright
Marc Hodes
Domhnaill Hernon
Roger Kempers
Paul Kolodner
Shankar Krishnan
Wei Ling
Sal Messana
John Mullins
Paul Rominski
Patricia Scanlon
All Rights Reserved © Alcatel-Lucent 2008
Additional Material
All Rights Reserved © Alcatel-Lucent 2008
Example optimal design problem:
Determine heat exchanger fin height and thickness
adiabatic wall
Assumptions for liquid heat transfer
 Flow rate of water at 2 GPM and 10oC
•FIN THICKNESS
 Fin gap of >3 mm (maximize number of fins)
Air side
heat flux
into each
of three
heat pipes
 Area for water flow: 50mm x 50mm
Governing equations for steady laminar flow
 Mass conservation (incompressible fluid)
 v  0
 Momentum conservation
 v  v    p     ,
H
E Fixed
I 76 mm
G Stack of
H 3 Heat
T Pipes
where   v   v T
 Energy conservation
 C p v  T    k T 
•Fin gap
2 GPM water at 10oC
All Rights Reserved © Alcatel-Lucent 2008
Characteristics of optimal geometry:
Minimizing heat pipe temperature rise
Contours of Average Heat Pipe
delt_fixedpitch
Temperature
Rise (oC)
Temperature contours along middle
fin for optimal geometry
12 2
12.. 0
11. 8
11. 6
11. 4
11. 2
11. 0
2 .5
2 .0
10
.6
121.24. 6
local minimum
in design space
(20% improvement
relative to other
admissible
design choices)
1 .5
10
.4
ECHIP
finthick [mm]
fin thickness
3 .0
1 .0
80
90
100
110
120
130
140
150
fin height
[mm]
finheight
finheigh=135.61
finthic k=1.63
Value
Low Limit
High Limit
Physics-based
optimization
is applicable
to a
10.32
9.87
10.76
range of other heat transfer problems, e.g.,
 Transient enhancement of heat transfer
 Three-dimensional heat sinks
 Very large potential design space
Essential design tool to minimize time and expense
All Rights Reserved © Alcatel-Lucent 2008
Current Core Network Example
 Example of a core network
 800Gb/s capacity packet network from Paris to
Barcelona
 Power Consumption
 800Gb/s Optical transport on protected ring
between 8kW
 Terabit class routers in Paris and Barcelona 26kW
 Cooling and power supply equipment x2 = 68KW.
 Assume 30% link utilization this gives power
consumption of ~300W per Gb/s
 Of the core power 25% is transport, 75%
switching (33% layer 1 and layer 2, 66% is
layer 3 IP).
 Half of the network power consumption is
because of Layer 3 functionality
All Rights Reserved © Alcatel-Lucent 2008
Paris
Barcelona