Future of Cooling High Density Equipment Steve Madara Vice President and General Manager Liebert Precision Cooling Business Emerson Network Power Agenda z Issues facing the Data Center – Managing the explosive growth – Power consumption getting the attention of Governments z Developing the right cooling strategy for today and into the future z Examples of how the right cooling strategy can lower the Total Cost of Ownership z Myth – High Density computing is more costly 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 2 Managing High Density Servers Rack density trend when fully populated with the newest server technology How most sites are dealing with server density 2000 2002 2006 28 x 2U Servers 42 x 1U Servers 6 Blade Centers 2kW Heat Load 6kWw Heat Load 24kW Heat Load 2009 Rabid Blades 40kW Heat Load Heading Toward 50kW 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 3 Progression to High Density The average server replacement cycle is 3-4 years Rack kW 1 kW 2 kW 5 kW 10 kW 15 kW 20 kW Issues facing the IT Manager z z z z z Getting Air out of the Racks Hot air mixing with the inlet of other racks Diversity of loads in the Data Center Not aware that more Fans create heat Flexibility of “On Demand Cooling” 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 4 DCUG Survey Results – What are the Biggest Issues facing the IT Manager 0% Heat Density (Cooling) Space Constraints/Grow th Pow er Density Adequate Monitoring Capabilities Availability (Uptime) Technology Changes / Change Energy Costs / Equipment Efficiency Other Security (Physical or Virtual) Data Center Consolidations Data Storage Hardw are Reliability Regulatory Compliance Staffing/Training Limitations 10% 20% 30% 40% 50% 22% 19% 18% 8% 7% 7% 5% 4% 3% 2% 2% 1% 1% 0% 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 5 Cooling Presents an Opportunity For Energy Savings Data Center Power Draws Cooling About 37%45% z z z z z z Electricity Transformer/ UPS 10% Air Movement 12% Cooling 25% Lighting, etc. 3% IT Equipment 50% Source: EYP Mission Critical Facilities Inc., New York Sources of Energy Waste Fans / Blowers running on redundant units Lack of air containment (cable openings, room leakage) Unnecessary cooling unit cycling on and off Lack of humidification control between units Mixing of Hot and Cold air lowering the effectiveness of the cooling unit Excess Fan energy that turns into heat 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 6 US EPA Report Data Center Efficiency Public Law 109 -431 109-431 z Energy consumed by servers & data centers has doubled in last 6 years and is expected to double again in next 5 years to > 100 Billion kWH z State-of-the-Art technologies and best management practices could reduce electricity use by up to 55% z Recommendations include: – Standardized performance measurements for data centers and equipment – Incentive programs – Research and development – Industry collaboration and partnerships www.energystar.gov/ia/partners/prod_development/downloads/EPA_Da tacenter_Report_Congress_Final1.pdf 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 7 Understanding The Basics z Heat generated is directly related to the server power (100%) z As server power increases (kW), the airflow (CFM) through z z z z the rack increases proportionally Raised floor tiles are limited in airflow (about 500-1000 CFM) Higher entering air temperatures on a cooling unit will provide more capacity and increased efficiency Higher density servers will have a greater range of temperatures leaving the rack over time (larger swings of server load) Fan horsepower to move the air is significant and all the power turns into heat (100 kw Cooling unit uses typically a 10 HP motor that generates 8.5kW of heat) 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 8 Having The Right High Density Cooling Strategy Delivers. . . z Cooling for High Density racks – Increase servers per rack – Increase number of racks per room z Energy Efficiency – Lower Operating Costs – More power allocated for IT/Server loads Get more out of your existing facility 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 9 Planning for High Density requires A Systems Approach to the Cooling System Traditional Floor-Mount through the first 100-150 w/sqft (or 4-5 kW per rack) and Supplemental Cooling above that level Provides high density sensible cooling at the source Computer Room / Data Center SUPPLEMENTAL COOLING TRADITIONAL Floor-Mount 0 5 Controls Humidity and Filtration + Creates the base airflow distribution 10 15 20 25 30 35 40 kW / Rack 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 10 Critical Requirements Of The Base Cooling Load Equipment z Cooling units with Variable cooling capacity – DX – Compressors such as the Digital Scroll – VFDs on the fans z Controls so that units can work as a team – Eliminate dehumidification/humidification fighting – Balance the load – Optimize the cooling performance z High efficiency condensers z “Green Refrigerant” products 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 11 Moving the Heat Room Servers Chip Heat Sink Rack CRAH Memory Other components Network Devices Power Supplies Chiller Heat Rejection Areas for Improvement z Reduced server fan power z Higher temperatures over cooling coils (room temperature / less mixing of hot and cold air) z Reduced resistances lower fan power 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 12 Cooling Solutions to Meet the Higher Density Requirements z Requires moving the cooling closer to the heat source to more precisely cool the specific load and not waste energy “brute force” cooling the whole room. z Cooling coils may be in multiple locations – External to the Rack – Overhead, Back, Side – In the Rack – Under, Side – Part of the Server z Requires a fluid to be delivered to the cooling coils to transport the heat out of the Data Center – Chilled Water (CW) – Refrigerant (pumped low pressure) 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 13 Energy Efficiency Benefits for Cooling Closer to the Load Traditional Cooling Only Fan Power- 8.5kW per 100 kW of Cooling Average entering air temperature of 80-84°F Cooling Unit Liebert XD & Base Cooling Fan Power- 3.5 kW per 100 kW of Cooling (XD @ 2 kW per 100kW) Average entering air temperature of 96-98°F Blower Resistances z 65% less fan power z Greater cooling coil effectiveness 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 14 Liquid Cooling Configurations – with Building Chilled Water Building Chiller XDWP CW Building Chilled Water Valve Heat Exchanger Tank CDU Pump Pump 2nd Loop XDP Refrigerant 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 15 Open and Closed Architectures z Open and Closed Architecture Systems as defined by ASHRAE – The open architecture systems utilize cooling coils near the heat load either inside or outside the server rack and utilize the room air volume as a thermal storage to ride through short power outages. – The closed architecture fully encloses the rack with the cooling coil inside. Other provisions are required for power loss ride through. 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 16 Secondary Fluid Comparisons Pumped Refrigerant System Chilled Water Based System Advantages z Advantages – No Water in the DC or – Lowest Fluid Cost electrical hazards – No limitation to room size – Micro-channel coil efficiency – – (+50%), lower air side pressure drop => lower operating costs Smaller piping requirements z Cooling Modules can be located anywhere Scalable Capacity (2-3x to CW ) – z Disadvantages – Small room scalability – Higher Fluid Cost Disadvantages – Electrical Hazard – Lower Operating Efficiency – May require fluid treatment to – prevent fouling Limited overhead cooling options 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 17 Flexibility - Extends Your Existing Infrastructure Investment z z z z z z z On demand, plug-and-play flexibility to add additional capacity Cooling at the source of heat with advanced compact heat exchangers Multiple module configurations to meet any Data Center layout Works with any brand of racks Cooling fluid is a gas (no water) Self-regulating capacity 100% Sensible cooling Liebert XDV Liebert XDO Hot Aisle – Cold Aisle Configuration Hot Spots, Zones and Hot Rooms Liebert XDH 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 18 Plug and Play Capacity on Demand 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 19 Liebert XD Energy Efficiency Benefits z Cooling closer to the source – Dramatically less fan power required to move the air – Higher air temperature entering the coil results in increase performance z Coil Technology – Microchannel coils – most efficient coil surface z Sensible cooling – All cooling module operate the coil at 5 degree F above the dew point in the room – Does not unnecessarily dehumidify requiring additional humidification (value of 7% in efficiency) 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 20 The Traditional Way AC AC AC PDU PDU PDU PDU AC PDU (120 racks @ 8kw/rack) z (12) 30 ton CW Air Handlers PDU – 10 operational for the load – 2 stand by PDU PDU AC PDU PDU PDU PDU AC AC PDU PDU PDU PDU AC PDU z Floor space – 4256 sqft PDU z Requires a raised floor of 48” PDU AC PDU PDU AC PDU PDU AC PDU AC 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 21 Liebert XD Solution (120 racks @ 8kw/rack) AC AC z (4) 20 ton CW Air Handlers PDU PDU PDU PDU PDU PDU XD – 3 operational for the load – 1 stand by XD z (6) XDP with (96) XDV XD PDU PDU PDU PDU PDU PDU PDU PDU PDU PDU PDU PDU – 5/80 operational – 1/16 redundant XD z Floor Space – 3640 sqft XD XD PDU PDU PDU AC PDU PDU PDU AC 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 22 Total Room Load Calculations 120 Racks 8 kW per Rack Rack Loads Fans(BHP) Room Latent Excess Latent PDU People Build Env Lights Total (kw) Floor Mount AH Liebert XD & Floor Mount 960.0 101.7 5.1 137.3 28.8 1.5 7.9 5.6 960.0 44.6 5.1 29.8 28.8 1.5 7.9 5.6 1247.9 1083.3 Liebert XD benefits – Less Fan power – 100% sensible Smaller load to size the Chiller 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 23 Summary of Equipment 120 Racks 8 kW per Rack Floor Mount AH Liebert XD Liebert XD & Floor Mount (vs traditional method) Chillers CW Pumps (3) 250 ton (3) 25 HP (3) 200 ton (3) 20 HP Floor Mount Units Liebert XD (12) 30 ton (4) 20 ton (96) XDV Floor Space (sqft) Raised Floor Height (in) 4256 48 3640 24 – 15% less floor Scalable Design 8kw 20kw – Scalable – 20% less Chiller plant space Platform 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 24 Annual Energy Consumption (kW) 120 Racks 8 kW per Rack Floor Mount Liebert XD & AH Floor Mount Prec Air Units XDV XDP Pumps CW Pumps Chiller Tower Fans Condenser Pumps Rehumidification 101.7 0.0 0.0 42.4 195.1 15.3 15.3 51.5 25.4 19.2 6.3 33.9 169.4 13.2 13.2 11.2 Total Kw 421.3 291.9 Operating Costs (@$.08/kw) $ Delta 295,231 $ $ Cooling closer to the source is more efficient. 204,571 (90,660) -31% 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 25 Capital Costs – Total Impact 120 Racks 8 kW per Rack Results Floor Mount Liebert XD & AH Floor Mount Chiller Cooling Units Installation Floor Space Total (E,I,S) $ $ $ $ $ Delta Operating Savings Payback (yr) 187,500 135,360 322,860 645,720 $ $ $ $ $ $ 150,000 258,015 452,265 (123,200) 737,080 91,360 $ (90,660) 1.0 Traditional – 4256 sqft Liebert XD – 3640 sqft (-616 ft²) Example at $200 / ft² Industry range $250-1000 / ft² 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 26 Liebert XD Technology - Fluid Cooling at the Source Drives Down Operating Costs Chiller Capacity Latent Load Fan Load Sensible Load 1.20 1.00 Annual Power Consumption 0.80 0.450 30% Lower 0.60 0.400 0.40 0.20 0.00 Traditional CW CRAC CW Enclosed Rack Refrigerant Modules 20% less capacity of the support equipment – Chiller (s) – Cooling Tower / Condensers – Chiller water circulating pumps kW power to cool 1 kW of sensible heat kW Chiller Capacity per kW of Sensible Heat Load 1.40 0.350 0.300 Fan 0.250 Pump (CDU) Pump (CW) 0.200 Chiller 0.150 0.100 0.050 0.000 – Emergency Generators – Electrical Switch Gear Traditional CW CRAC CW Enclosed Rack Refrigerant Modules 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 27 Liebert XD Full Range of Opportunities XDO20 XDV10 XDC or XDP Base Infrastructure (160 kw) Future pumping units of larger capacity XDH20/32 Standard Cooling Modules 10-35kw +++ Embedded Cooling Embedded & Chip Cooling (microchannel intercoolers) (microchannel intercoolers and Cooligy chip cooling) Tested 35-60kw Developing up to 100kw Tested 50 kw (100% redundant) Capable over 100kw 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 28 Energy Savings driven by reduction in Fan Power (cooling system and server) plus Heat Transfer efficiency 0.65 0.6 ÅTraditional CoolingÆ 2010 2006/2007 2004/2006 2006/2007 2005 2000 0 0.2 Future 0.1 45% Component Cooling (>50 kW) 0.2 0.31 30% Embedded Cooling (35 – 60 kW) Optimal 0.3 0.36 (10 – 20 kW) Data Center Best Practices Enclosed Rack 0.4 0.42 Egenera 0.5 XD Modules (10 – 35 kW) 0.7 1990 kw for cooling per kw of server heat load Additional System Opportunities for Improved Cooling Efficiencies Å Liebert XD Opportunities Æ 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 29 TCO Flexibility Availability TCO Flexibility Availability TCO Flexibility Availability TCO Flexibility Availability TCO Flexibility Availability TCO Flexibility Availability TCO Flexibility Availability TCO Flexibility Availability TCO Flexibility Cooling Process Availability Cooling Process Throughout The Range Of Rack Loads Chip/Component Cooling Embedded Rack - Refrigerant Rack Mounted - Refrigerant Rack Mounted - CW Traditional CRAH >0-5 >5-10 >10-15 >15-20 >20-25 >25-30 >30-35 >35-50 >50 Rack Load (kW) 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 30 Improving The Total Cost Of Ownership with the Liebert XD Cooling Systems How Cooling closer to the source of the heat makes the heat exchangers more efficient (higher entering air temperatures) Lower total Fan HP Sensible cooling eliminates wasted energy dehumidify unnecessarily and then having to re-humidify Less Chiller or DX infrastructure required Results in … ¾ More cooling capacity for energy consumed ¾ Less Power (energy consumed) ¾ Less Power (energy consumed) ¾ Less Power and Capital Equipment Overhead cooling modules require no ¾ Less Floor Space additional floor space consumed Cooling solutions that meets the requirements ¾ Less Floor Space consumed to fill racks of high density servers Infrastructure for modules today and for future ¾ Extends your capital life server / rack designs 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 31 Lower Power For Cooling Provides More Power For IT Equipment Data Center Power Draws 12% 26% Electricity Transformer/ UPS 10% Air Movement 12% Lighting, etc. 3% 59% IT Equipment 50% Cooling 25% For the same building power, you can allocate more power to the IT Equipment (18%) 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 32 Take Aways z Cooling solutions for higher density will need to move closer to the load and will require a reliable fluid delivery means – Cost effective cooling solutions exist that can be employed today that meet future needs – Allow racks to be fully populated z Cooling at the source of the heat load will actually lower your incremental energy consumption – Less power for the cooling system provides more power for the IT equipment Results in more available Floor Space and Growth capability for the IT equipment 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 33 Thank You Contact: Steve.Madara@EmersonNetworkPower.com 2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved. 34