Energy Efficient Strategies for High Density Telecom Applications Alan M. Lyons, David T. Neilson and Todd R. Salamon Bell Laboratories Why do we care about energy efficiency? Transistor Package Circuit Pack Shelf Cabinet Central Office Environment Increasing Thermal Energy Greener Networks Overall power dissipation Network operator power dissipation is 1% of many countries energy consumption Pressure to reduce network power consumption while still growing network capacity and functionality Scalability Power density in racks of communications equipment is reaching practical limits. Makes cost-efficient scaling of telecom networks difficult All Rights Reserved © Alcatel-Lucent 2008 Thermal Management: Motivation Telecom equipment vendors face the toughest thermal challenge and must lead the electronics industry. 1. Increase equipment functionality and density Meet future power density needs – exceeding 15,000 watts/cabinet Maintain high reliability and meet acoustic noise limits. 2. Reduce carbon footprint and OPEX for our customers Telecom consumes 3% of U.S. electricity Major Telcos spend > $1 Billion USD annually on electricity 3. Meet or exceed regulations NEBS, ETSI & guidelines for energy consumption and reduction Source: ASHRAE Society Handbook, 2005 How can cooling technologies be extended to meet future power density needs? 4. Differentiate ALU’s products All Rights Reserved © Alcatel-Lucent 2008 What are we doing in Thermal Management Research? Extending the Limits of Air Cooling Thermal Management Research Thermal Interface Materials Increasing Cooling Efficiency Liquid Cooling Cabinet Architectures (with CTO) Heat Sinks & Vortex Generators Waste Heat Recovery Thermo-Electric Modules Fan Reliability Vapor Chambers All Rights Reserved © Alcatel-Lucent 2008 Liquid Cooling Architectures for the Central Office From: dispersing waste heat into Central Office air To: transferring waste heat into a liquid coolant and piping outside the CO Motivation: Thermal Density: Heat capacity of air limits total heat that can be removed at air flow rates that meet acoustic noise specifications. Liquids have 103 higher heat capacity compared to gases Energy Efficiency: Pumping Power required for air is >> than that for water Hot air leaks into cold aisles. Air must be chilled below ambient temperature to insure sufficient cooling capacity Challenges: All Rightsincrease Reserved © Alcatel-Lucent 2008 Insure system reliability, minimize costs, thermal capacity from 15 to 50kW Design Example: Heat Exchangers Above Each Shelf Approach: Front View Perspective Cut-Away Cooling Liquid Supply Heat Exchangers Heat-pipe based heat exchangers above electronics extract heat from air Vapor in heat pipe transports heat through cabinet walls into water side heat exchanger Chilled water pumps heat to outdoor cooling tower. Advantages: Cabinet Level: Insures that cooling air is at ambient temperature Reduces number of fans Equipment shelves Isolates water from electronics Room Level: Ambient Airflow Heated water return Minimizes room air handlers – eliminates large blower motors Eliminates need to chill air below ambient temperature. No opportunity to intake hot room air All Rights Reserved © Alcatel-Lucent 2008 Sealed Cabinet Advantages Higher thermal densities can be accommodated both in the cabinet and Central Office/Data Center Heat Pipe Architecture (Open System) Heat Pipe Architecture (Closed Recirculating Air System) Cooling Water Supply Improved real-estate utilization Closer spacing of cabinets Eliminates need for raised floors Finned Heat Pipes Higher reliability - ambient pollutants & dust not continuously drawn across equipment Quieter operation at higher fan speeds. Closed cabinet reflects noise and enables acoustic foam installation. Energy savings • Eliminates need to under-cool air. Cooling is provided directly to the cabinet • Large air volumes need not be moved across large distances. • Enables use of low-cost cooling towers and potential off-peak cooling. Hot Airflow Cold Airflow All Rights Reserved © Alcatel-Lucent 2008 (cut-away view) Enhanced Cooling in a Sealed Cabinet Using an Evaporating-Condensing Dielectric Mist Acoustic foam Recirculating air Mist from collectors Hot water pumped to atomizer Heat pipe Mist condenses on Shelf 2 heat pipes and falls by Pump gravity into collector Chiller unit Cold water Heat pipe Shelf 1 Atomizer ∞ Fan ∞ Large droplets of mist from atomizer directed into circuit packs Cabinet Level Circuit Pack Level Heat Sink Level Objective: Limit temperature rise of air flowing through circuit packs by injecting atomized mist to increase its effective sensible heat capacity. Approach: Inject atomized HFE7000 (environmentally benign dielectric fluid) upstream of circuit packs and/or high power component’s heat sinks. Condense vapor on finned heat pipes between shelves and recirculate it. All Rights Reserved © Alcatel-Lucent 2008 Calculated Optimum Droplet Diameter for Complete Evaporation at Outlet of a 32 mm Wide x 32 mm Long x 13 mm High Heat Sink and Cooling Capacity and Associated Parameters as Function of Mist Loading Parameter Mist loading = 0.01 % by volume Mist loading = 0.1 % by volume Total mass flow rate (gm/s) Length of side of unit cell (in terms of droplet diameter) 0.435 17.3 0.871 8.04 Optimum droplet diameter (μm) RH after complete evaporation (constant volume, constant pressure) (%) Residence time in heat sink (msec) Acceleration pressure drop (Pa) Evaporative cooling (W) 83.2 (2.8, 2.75) 74.9 (28.1, 24) 31.5 0.0355 6.87 22.5 0.67 68.7 Enormous potential for cooling enhancement (about 70 W) for modest pressure drop (< 1 Pa) and relative humidity (<25%) increases All Rights Reserved © Alcatel-Lucent 2008 Extending the Limits of Air Cooling Initial approach is to extend the limits of air cooling while maintaining full NEBS/ETSI compliance… …to allow thermal density increases without exceeding noise standards Schematic of Microprocessor Cooling Thermal Resistance Stack Heat Sink Improvement Targets: IC Package to heat sink interface 0.36 K/W 60% TIM2 IHS 0.15 K/W TIM1 Chip Carrier Silicon Die 0.34 K/W Thermal Resistance = (Tjunction – Tambient)/Power All Rights Reserved © Alcatel-Lucent 2008 Fin to air interface Heat Spreading in the base void Thermal Interface Materials (TIMs) conventional TIM Conventional TIMs: polymer-metal composites TIM test rig Limits: Thermal path limited by particle-particle point contacts: typical values 2-10 W/mK Voids reduce reliability and achievable thermal properties TIMs research program Measurements: Designed and built a world-class test rig to verify and compare commercial materials Maximize conductivity while minimizing assembly force. First Results Thermal conductivity >4.5 W/mK with >60% compression (>1.5mm compliance) Large experimental/modeling space to explore All Rights Reserved © Alcatel-Lucent 2008 Pressure (MPa) Developing new micro-textured TIM with higher thermal conductivity, compliance and reliability Effective Thermal Conductivity (W/mK) Materials: Effective conductivity and loading pressure versus compression Compression (mm) Energy Harvesting Using Thermoelectrics Goals: Reduce power required to cool equipment by > 20% Convert waste heat into electricity Approach: Carnot Efficiency is inherently too low for CMOS devices Target IC’s with junction temperatures > 300oC for adequate efficiency Thermoelectric Module approach die die Vapor Chamber Hot Thermoelectric Module Cold examples: SiC, GaN Direct approach: Thermoelectric Modules to convert waste heat directly to electricity Challenge: Efficiency of Thermoelectric Module is low – research required for new materials and designs. All Rights Reserved © Alcatel-Lucent 2008 V Electricity generated by Thermoelectric Module Network routers are responsible for much of ICT energy growth Overall Network power growth In 2006 in Japan, IT and Communications equipment consumed about 45TWh or 4% of the total electricity generated or 1% of the country total energy consumption. About ¼ of this i.e. 1% of power consumption is network operators Information and Communications Technologies Energy Usage Packet switching responsible for much of growth rate Energy usage [TWh] Japan's Ministry of Internal Affairs and Communications Study Group on ICT System and Network for Reducing Environmental Impacts, March 2007 Year All Rights Reserved © Alcatel-Lucent 2008 Scalability: Electronic Router Power Density Historical trend of x2 capacity every 18 months Not sustainable because of thermal density Tb/s router today with 10-15kW is at the limit Future growth thermal density limited Shortfall of 30 fold in capacity by 2015 compared to historical trend for single rack router Where the power dissipation is in core routers 2/3 of power associated with layer 3 function 16% Core Router Power Consumption by Layer L1+L2 L3 Switch Packet Forwarding Engine dominates : –Processing IP headers for destination and quality of service queuing Eliminating L3 function in core would allow more scalability and lower overall power All Rights Reserved © Alcatel-Lucent 2008 22% 62% Current Core Network - IP over DWDM Legend Core IP router DWDM terminal or fixed OADM Core IP routers connected by point to point DWDM All switching at a node uses router Router capacity scales as node capacity Traffic has multi hop routing Requires total router capacity in network to be larger than traffic by the average number of hops All Rights Reserved © Alcatel-Lucent 2008 Network Evolution to Transport switching Packet Transport switch Core IP router Layer 3 OCh TMPLS Layer 2 Switch OCh OCh OCh DWDM terminal, ROADM Layer 1 Switch Transparent flexible optical switching Reduce switching requirement for Layer 3 IP routers Layer 1 ROADM: Wavelength channels bypass nodes reduce number of hops for electronic switches Layer 2 TMPLS: Packet switching network for transporting packets 1/3 of the power of Layer 3 switching Layer 3 Routers only handle service layer not transport switching Primarily edge/service layer function not core switching Does not require rapid scaling of core IP routers Uses lower-power switching elements All Rights Reserved © Alcatel-Lucent 2008 OCh Architecture change is already happening: Qwest runs it’s Juniper T640’s as MPLS routers in the core (Poll: OFC 2008 plenary) However they are still using core routers Summary Enhanced cooling using a dielectric mist Why do we care about energy efficiency? Financial and environmental cost of energy usage Practicality of cooling high-energy-density equipment Thermal Management Solutions Liquid cooling solutions for cabinet architectures Extending the limits of air cooling Thermal interface materials (TIMs) Energy harvesting Energy-efficient switching architecture Thermoelectric modules Network Evolution to Reduce Load on Layer 3 Routers Combine Layer 1 ROADM and Layer 2 TMPLS to reduce energy usage of Layer 3 Routers Expected reduction of 30 to 50 percent All Rights Reserved © Alcatel-Lucent 2008 OCh OCh OCh OCh Acknowledgements Vaihbav Bahadur Martin Cleary Cormac Eason Ryan Enright Marc Hodes Domhnaill Hernon Roger Kempers Paul Kolodner Shankar Krishnan Wei Ling Sal Messana John Mullins Paul Rominski Patricia Scanlon All Rights Reserved © Alcatel-Lucent 2008 Additional Material All Rights Reserved © Alcatel-Lucent 2008 Example optimal design problem: Determine heat exchanger fin height and thickness adiabatic wall Assumptions for liquid heat transfer Flow rate of water at 2 GPM and 10oC •FIN THICKNESS Fin gap of >3 mm (maximize number of fins) Air side heat flux into each of three heat pipes Area for water flow: 50mm x 50mm Governing equations for steady laminar flow Mass conservation (incompressible fluid) v 0 Momentum conservation v v p , H E Fixed I 76 mm G Stack of H 3 Heat T Pipes where v v T Energy conservation C p v T k T •Fin gap 2 GPM water at 10oC All Rights Reserved © Alcatel-Lucent 2008 Characteristics of optimal geometry: Minimizing heat pipe temperature rise Contours of Average Heat Pipe delt_fixedpitch Temperature Rise (oC) Temperature contours along middle fin for optimal geometry 12 2 12.. 0 11. 8 11. 6 11. 4 11. 2 11. 0 2 .5 2 .0 10 .6 121.24. 6 local minimum in design space (20% improvement relative to other admissible design choices) 1 .5 10 .4 ECHIP finthick [mm] fin thickness 3 .0 1 .0 80 90 100 110 120 130 140 150 fin height [mm] finheight finheigh=135.61 finthic k=1.63 Value Low Limit High Limit Physics-based optimization is applicable to a 10.32 9.87 10.76 range of other heat transfer problems, e.g., Transient enhancement of heat transfer Three-dimensional heat sinks Very large potential design space Essential design tool to minimize time and expense All Rights Reserved © Alcatel-Lucent 2008 Current Core Network Example Example of a core network 800Gb/s capacity packet network from Paris to Barcelona Power Consumption 800Gb/s Optical transport on protected ring between 8kW Terabit class routers in Paris and Barcelona 26kW Cooling and power supply equipment x2 = 68KW. Assume 30% link utilization this gives power consumption of ~300W per Gb/s Of the core power 25% is transport, 75% switching (33% layer 1 and layer 2, 66% is layer 3 IP). Half of the network power consumption is because of Layer 3 functionality All Rights Reserved © Alcatel-Lucent 2008 Paris Barcelona