27 April 2009
Tomm Aldridge, Principal Engineer
Joint work prepared by:
Intel Corporate Technology Group
Emerson Network Power
EYP Mission Critical Facilities
Intel may have referenced third-party information throughout this presentation. Intel’s use of third-party copyrighted or trademarked materials are for reference only. Intel does not warrant the adequacy, accuracy or completeness of the third-party information nor does Intel claim that it has obtained a license to the third-party material. Copyright to the third-party referenced materials remain with their respective holders. LIMITED RIGHTS
Energy Systems Research Lab
Intel Corporate Technology Group

2
Summary Vision
DC
300-
400V
DC/DC
AC/DC
60 Hz AC
480V AC/DC
Lighting loads
DC/AC
Ballast
DC/DC
Electronic loads
VR
PSU
Motor loads
DC/AC
ASD
Energy Systems
Research Lab

3
 Inefficiencies in data center power delivery are significant, expensive, and ecologically unsustainable
– 50% of power is lost in conversions, transformations, and distribution
– US data centers consumed 61B kWh ($4.5B) in 2006, doubling by 2011
– Worldwide data centers consumed 123B kWh in 2005, and is expected to increase 40% to 76% by 2010
– By 2011 the energy costs over the life of a server will exceed the purchase price
– Half of all data centers are already power constrained and unable to expand
– EPA estimates 10 new power plants will be required by 2011 just to support US data center growth
– By 2011, powering US data centers will be responsible for ~70M metric tons of atmospheric CO
2 per year
Energy Systems
Research Lab

4
 Higher efficiency
 Lower equipment and installation costs
 Easy integration with alternate power sources
(solar, wind, etc.)
 Applicable across entire data center (lighting, cooling, energy storage), not just compute loads
 Improved reliability
 Smaller footprint for power conversion equipment
Energy Systems
Research Lab

Motivation for improving power distribution
Power distribution - typical US data center today
MV
BLDG
PWR 480V
AC
?
UPS
480V
AC
PDU
208V
AC
L-L PSU
Rack
Server
12V
VR
Fans
Electr.
loads
Electr.
loads
5
Total
255W –
375W
Room cooling
50W – 150W
UPS
18W
PDU
4W
PSU
48W
Server fans
13W
VR
22W
~50% of power lost in power distribution
Energy Systems
Research Lab
100W
Server

480V
Optimizing Data Center Power Delivery
400V
AC/DC
400V
DC
DC/AC
F F
AC/DC
400V
DC
DC/DC
Baseline 480Vac
Direct 400Vdc
 Intel and partners working to make optimal power delivery a reality
6
>20% improvement in power delivery efficiency
Energy Systems
Research Lab

7
Facility 400Vdc
Facility 550V/48Vdc
Facility 48Vdc
Rack-level 400Vdc
Rack-level 48Vdc
Specialized AC UPS
400Vac
High efficiency AC
Baseline AC
50.0
~5%
~20%
55.0
60.0
65.0
Power delivery efficiency [%]
70.0
Power delivery loss reduction:
20%
60%
75.0
Key Conclusions
 400Vdc most efficient
 DC solutions simplify power management
– no harmonics or phase balancing
– easy to parallel sources
 AC solutions require additional conversions which degrade efficiency and reliability
 400Vdc requires only modest changes to existing equipment
 All techniques suffer increased losses at light loads
 48Vdc also promising, but suffers important drawbacks
– Requires ~100x more copper to distribute power
– Suffers greater conversion loss from utility to building entrance
– Other components less efficient at 48 V (e.g. lighting, adjustable speed drives,
HVAC)
Energy Systems
Research Lab

AC and DC power distribution : Benefits of 400Vdc
Analysis of 2009 Data Center Design*
 ~7% Facility energy savings, incl. cooling
– 7.7% at 50% load; 6.9% at 80% load
 33% Space Savings
– No PDUs, simplified switchgear
 200% Reliability improvement
– 2x lower probability of failure in 5 years
– 1000% reliability improvement if direct connect to batteries
 15% Electrical facility capital cost savings
– Electrical is ~40% of total facility cost, i.e. saves 15% of 40%
~ 6% of total
 Using fewer of the earth’s resources
– 15% Component volume reduction in every server power supply (PFC)
– Some additional reduction in battery cabinets without DC/DC ++
8
*Compared to a modern, high efficiency 480-208 Vac design
**++DC/DC assumed in other comparisons.
Energy Systems
Research Lab

9
AC and DC power distribution : Benefits of 400Vdc
 Additional benefits to DC distribution
– No phase balancing => reduces power strip & wiring complexity
– No synchronization required to parallel multiple sources
– No harmonics => no PFC circuits
– Fewer breakers required because of fewer stages
– Simplifies wiring, since only two wires required
– No need for complex interlocks - simpler procedures, less time
– Only resistive voltage drop in wires
 Additional benefits specific to higher voltage DC
– Simplifies wiring, especially at higher power densities
– Lower currents than at 48Vdc, so smaller physical wires
– Use fewer natural resources & less energy to process materials
– ~400Vdc bus in light ballasts and Adjustable Speed Drives (ASDs)
– Simpler/more efficient connection to renewable energy sources
– Photovoltaics, fuel cells, wind with variable frequency drives
Energy Systems
Research Lab

 Why use 400Vdc and not a higher or lower voltage?
– 400Vdc link voltage already exists in power supplies
– Straightforward to extend across data center (UPS, PDU)
– Well within 600V safety limit
– Operates over standard 600V wiring and bussing systems
– Commercial solutions already emerging
10
400Vdc is a natural choice – efficient, safe, and commercially easy to adopt
Energy Systems
Research Lab

11
 Two-story, vertical flow-through air design
 Designed for ~500 watts per square foot
 Centralized air cooling plant
 6,000 square feet of raised metal floor (RMF) per module
 240 cabinets per module (20 network)
Energy Systems
Research Lab

Modules C - E:
Future Full
Build Out
Module C (Next phase)
(220 racks x 25 kW/rack)
Module E Module D Module C
5.5MW IT
Load
Loading
Dock
Facility
Electrical
Room
Utility Spine
Chilled Water
Storage Tanks
Low-Temp
Chiller
Plant
High-Temp
Chiller
Plant
High-Temp
Chiller
Plant
Expansion
Module B
3.3MW IT
Load
Module A
3.3MW IT
Load
Modules A - B:
Existing
(220 racks x
15 kW/rack)
400Vdc proposal for Module C
- Intel
- Emerson Network Power
- EYP Mission Critical Facilities (HP)
12
Energy Systems
Research Lab

Comparisons : Efficiency
 Compare proposed 480V/208Vac & 400Vdc designs for Intel facility **
13
– For 5.5MW facility
– Iso-redundant 480Vac
– Distributed-redundant
400Vdc
– Assumed 2.15% total cable loss for both
– Non-redundant and
– redundant PSU cases modeled
Cooling estimates included in total energy savings
80
75
70
65
750
625
500
375
60
55
50
250
High eff 480VAC
Facility 400VDC
Facility 400Vdc (Red PSU)
High Eff 480Vac (Red PSU)
Total savings [kW]
125
0
0 10 20 30 40 50 60 70 80 90 100
Load [%] *Facility savings includes cooling power
400Vdc is the highest efficiency choice.
~7.5% Energy savings compared to 480/208Vac distribution
** Results will vary for different facilities and for different AC architectures
Energy Systems
Research Lab

AC versus DC distribution
MV
480V
AC
AC/DC
Chrg
UPS
DC/AC
PDU
208V
AC
L-L
AC/DC
400V
DC
DC/DC
PSU
Rack
12V
VR
Server
 Specialized UPSs avoid double conversion
• Include Delta conversion and Eco-mode double conversion
 End user concerns about line disturbance susceptibility remain
 480V/277Vac directly to PSU (server) has been proposed
 Remove PDU transformer
 Expect power supply to be more efficient than at 208V/230Vac
 Efficiency data not yet available
 Need new power supply since bus voltage > 430Vdc required
14
AC Claims it can be only 4-6% less efficient than DC
… But at what cost?
Energy Systems
Research Lab

AC and DC power distribution
MV
480V
AC
UPS
AC/DC
Chrg
Bypass
DC/AC
208V
AC AC/DC
380-
410V
DC
DC/DC
PSU
12V
VR
Rack
Server
MV
Battery cabinet(s)
480V
AC
Rect
AC/DC
PDU
400V
DC
400V
DC
DC/DC
12V
VR
DC/DC PSU
Server
Battery cabinet(s)
PDU
Rack
 Proposed distribution bus of 400Vdc directly connected to batteries
 Optionally: Batteries connected through bi-directional DC/DC
– Battery voltage can be optimized since not directly connected to bus
15
Energy Systems
Research Lab

16
Direct connection of batteries to 400Vdc bus
480V
AC
Rect
AC/DC
400V
DC
(float)
400V
DC
DC/DC
12V
VR
Chrg PSU
Server
Battery cabinet(s)
Rack
 Batteries connected directly to the distribution bus
– Increased reliability
– Space and cost savings
 This implies a wide input range DC/DC
 UPS mostly used only till generators on-line, 30sec - 2min
– Typically size banks for 5 minutes  won’t reach discharge voltage
– Prototypes rated at 350Vdc for continuous operation
– Standard DC/DC designed to operate 20msec at 300V, bench test showed capable of running several hours at 320Vdc
Recommended implementation Energy Systems
Research Lab

17
400Vdc concept : Energy savings
 Rack power reduced with higher efficiency PSU
– 400Vdc rack power ~ 0.98 x 208Vac rack power
 400Vdc power supplies are prototypes from Delta Electronics
– Will be able to optimize further, but not significantly so
100
95
90
85
80
75
0 0.2
Typical operating range
0.4
Load [pu]
0.6
2.5
2
1.5
1
208V ac eff
400V dc eff
400V dc est
Eff delta
Est eff delta
0.8
0.5
1
0
100
95
90
85
80
75
0 0.2
Typical operating range
0.4
Load [pu]
0.6
1
0.98
0.96
0.94
208V ac eff
400V dc eff
400V dc est
Pin ratio
Est Pin ratio
0.8
0.92
1
0.9
Energy Systems
Research Lab

Results & Comparisons
AC design requires 50% more space for 5.5MW than 3.3 MW IT load
Module E Module D
Utility Spine extension
Module C
5.5MW IT
Load
Loading
Dock
Utility Spine
Facility
Electrical
Room
Chilled Water
Storage Tanks
High-Temp
Chiller
Plant
Module B
3.3MW IT
Load
Module A
3.3MW IT
Load
Low-Temp
Chiller
Plant
High-Temp
Chiller
Plant
Expansion
400 Vdc design can fit in the same space as 3.3MW facility
18
33% Space Savings: Fits within original Module C footprint
Research Lab

Results & Comparisons: Reliability
 Relex calculations by EYP Mission Critical Facilities
Option Availability
AC Tier IV configuration
DC configuration **
DC Improvement
0.999996
0.999998
Unavailability Probability of failure in 5 years
3.9 e -06
2.4 e -06
62.5%
13.63%
6.72%
200%
2X lower probability of failure compared to equivalent Tier IV AC facility
19
** Reliability numbers for Rectifier from simulation of power train only, will be lower with all components included.
Assumes DC/DC converter between Battery & Bus – will be higher reliability in systems without converter.
Energy Systems
Research Lab

Description Ratio Remarks
Labor
Materials
AC Data
Center
[pu]
0.193
0.215
DC Data
Center
[pu]
0.156
0.227
Owner furnished items
Electrical Cost of
Work (COW)
0.592
1.0
0.467
0.85
0.8
Due to smaller wiring sizes
1.06
Wire, pipes, Supports, Panel boards,
Busways etc
0.79
Equipment, UPS, static switches, transformers, etc.
Total cost to implement all the electrical work including equipment
UPS / Rectifier
480/208V
Transformers
Server Power
Supply
AC
1.0
1.0
1.0
DC
0
Remarks
0.83
Inclusive of batteries, DC/DC converter, input & output switchgear
Distribution transformers are eliminated in DC Data Center
1.0
Assumed same cost although 6% reduction in BOM cost estimated
20
Electrical cost for dc Data Center estimated 15% lower than for ac
Cost comparison for a 3.3 MW load rather than the 5.5 MW load due to available data.
Cost estimate for maturity, i.e. high volume.
Energy Systems
Research Lab

 “In typical high-power-density ac/dc power supply 60-65% of volume is taken by
EMI filter, PFC, and bulk capacitors”

Estimate ~15% for PFC stage
3 kW / 48 V
25 W/in 3
1U
BULK CAPS
DC/DC OUTPUT
STAGE
PFC
(no longer needed)
21
EMI FILTER
15% Fewer Components by Volume – Conserving the Earth’s Resources
M. Jovanovic, APEC 2006 keynote
Energy Systems
Research Lab

22
 Need new/modified equipment
– Much equipment already exists for other industries
 May need new standards
– International Standards already exist
– Unfamiliarity in the Server Industry can leave standards open to interpretation, so have to engage local authorities
 Market acceptance
– Need end users to demand it
– Put it in your Request for Proposals
Energy Systems
Research Lab

Equipment : PSU
 For servers, only power supply enabling required
 Remove PFC stage, re-use existing DC/DC stage
 Enabling circuit proposed only for input current > 5A
90 – 264Vac
50/60 Hz AC fuse
L
BOOST
+
EMI
Filter
(AC) s
BOOST
C
O
GND
400Vdc
GND
DC/DC
Chassis
-
Modified supply to accept 400Vdc input
DC fuse
EMI
Filter
(DC)
+
DC/DC
C
O
Chassis
EN
-
23
Energy Systems
Research Lab
12V
12V

24
Equipment : PSU
 Current PSU prototypes from Delta use Anderson Power
Products connector from their PowerPak series
– Power contacts rated for up to
30A @ 600V (Intel’s version is rated 10 A)
GND
(make first)
+ 400Vdc
– connectors have UL, CSA and TUV approvals.
– receptacle fits standard chassis opening for IEC320 C22 inlet
400Vdc return
Enable
(make last, break first)
 For the next step
– Receptacle compatible with the
IEC320 C14 chassis opening
– Molded rather than “modular” plug for reduced costs
– Proposed 15A continuous rating
– Expect 5 – 8A disconnect rating
– Prototype under development
– Will include 48Vdc keying option
Energy Systems
Research Lab

25
Equipment : CDU
 No phase balancing
 IEC 309 connector
– Standard for international AC power strips
– Rated for hot disconnect
– Recommend mechanical interlock
– IEC specified for >250VDC
– Not currently UL listed in the US
Energy Systems
Research Lab

Comparisons : Efficiency
 Expect lower conduction losses : 5 vs. 10 breakers
480Vac
3W + Grd
280Vac
4W + Grd
AC UPS
4000A
1600A
1600A
600A
12.47kV
12.47kV
10 breaker in series
480Vac
3W + Grd
2500A
1000A
DC UPS
Load
Bank
3000A
400Vdc
2W
Floor
Box
Busduct
26
5 breaker in series
** It may be possible to eliminate the 2,500A breaker at rectifier input
Energy Systems
Research Lab
100A

27
Maintenance
 Personnel access equivalent to AC system
 Power supply/server replacement
– Appliance couplers will have UL, CSA and TUV approvals
 Rack connection and disconnect
– DC rated IEC 309 or Anderson connectors
– Rated for hot disconnect, but propose mechanical interlock
 Rectifier servicing same as AC UPS
– Addressed by ability to isolate each rectifier
Energy Systems
Research Lab

28
Summary Vision
DC
300-
400V
Vs. Current
 28% Energy Savings
DC/DC DC/AC
Ballast
Lighting loads
AC/DC
60 Hz AC
480V AC/DC
Vs. Best AC
 7% Energy Savings
 15% Less Capital Cost
 15% fewer PSU components
 33% Datacenter Space Savings
 200% Reliability Improvement
DC/DC
Electronic loads
VR
PSU
Motor loads
DC/AC
ASD
Energy Systems
Research Lab

29
Energy Systems
Research Lab