Fortune Oregon Data Center Increases
Reliability with a High Resistance Grounding
System
Cory David Smith, Project Manager, ECOM Engineering Inc., and David Lawrence Smith, Principal,
ECOM Engineering Inc.
Abstract— A comparison of a High Resistance Grounding
System and a Solidly Grounding system in a Data Center
application. This document provides the pros and cons of using a
high resistance ground system in a data center. It also looks at
the common elements of a Data Center’s power distribution
system and explains the different setup between a High
Resistance Grounded system and a Solidly Grounded System.
I. NOMENCLATURE
HRG: High Resistance Ground, PDU: Power Distribution
Unit, UPS: Uninterruptible Power Supply, NRK: Neutral
Reference Kit, Transient Voltage Surge Suppressor
(TVSS)/Surge Protection Device (SPD), IT: Information
Technology, NEC: National Electric Code 2008.
II. INTRODUCTION
T
HIS document looks at the advantages and disadvantages
of using a High Resistance Grounding system in a data
center. The data center is an 12 Megawatt facility in Oregon
built and operated by Fortune Data Centers. In this document
a solidly grounded system and a high resistance grounding
system fault current will be compared for the same system, the
verified phase to ground fault for the high resistance ground
will be measured and the charging current of the system will
be verified for the grounding system. With the mitigation of
damage during a single phase to ground fault and continuity of
service during a single phase to ground fault, high resistance
grounding is an ideal application for protecting critical loads
in a data center.
III. HIGH RESISTANCE GROUNDING IN A DATA CENTER
The use of High Resistance Grounding or HRG at Fortune
Oregon Data Center for critical loads allows for a more robust
distribution system. The distribution system becomes more
robust with the use of HRG by being able to continue service
without interruption through a single phase to ground fault,
location of the ground fault and then isolation and clearing of
Fortune Oregon Data Center. Dave Smith, ECOM Engineering, Owners
Representative, Jesse Smith, Nova Partners, Construction Manager, Travis
Schumacher, DPR/FORTIS Mission Critical, MEP Coordinator. Fortune
Oregon Data Center Engineer of Record Steve Emert PE, Rosedin Electric.
Eaton PowerWare, Rosedin Electric.
the ground fault with minimal damage to equipment and no
interruption of critical power distributions systems.
A HRG system has several advantages to implementation in
a data center. The application of HRG eliminates the need for
complex and expensive zone interlock ground fault breaker
schemes.
With the HRG system having a low phase to ground fault
current, a restrained transient overvoltage and ability to
continuously serve the critical load during a phase to ground
fault condition, makes the HRG system an attractive solution
to the data center community. The HRG system insures high
availability and concurrent maintainability as demanded by IT
managers served by data centers.
By using a HRG system single phase to ground fault
available energy is greatly reduced. The concern for groundfault protection is based on four factors:
1. The majority of electric faults involve ground. Even
faults that are initially phase to phase spread quickly to
any adjacent metallic housing, conduit, or tray that
provides a return path to the system grounding point.
Ungrounded systems are also subject to ground faults
and require careful attention to ground detection and
fault protection.
2. The ground-fault protective sensitivity can be
relatively independent of continuous-load current
values and therefore, have lower pickup settings than
phase protective devices.
3. Because ground-fault currents are not transferred
through system power transformers that are connected
delta-wye or delta-delta, the ground-fault protection for
each system voltage level is independent of the
protection at other voltage levels. This configuration
permits much faster relaying than can be afforded by
phase-protective devices that require coordination
using pickup values and time delays that extend from
the load to the source generators and often result in
considerable time delay at some points in the system.
4. Arcing ground faults that are not promptly detected
and cleared can be destructive. [1]
A. Types of Grounding in a Data Center
Per the NEC only two types of grounding can be used in a
low voltage system in the United States. A solidly grounded
system and for three phase loads High Resistance Grounding
can be implemented [2]. Both systems offer advantages and
disadvantages based on application.
B. Advantages and Disadvantages of Solidly
Grounded Systems in Data Centers
A solidly grounded system offers several advantages:
1. During a phase to ground fault the system does not
suffer from overvoltages.
2. The solidly grounded system has low step potentials
between ground and the electrical system in normal
operation.
3. Single phase to ground loads can be applied to the
electrical system without the need for isolation
transformers between the critical equipment and
single phase loads for lighting and equipment.
4. Paralleling UPS units in a Solidly Grounded system
is not as complicated as in a HRG system.
Disadvantages of a solidly grounded system in a data center
include:
1. High ground fault current and a voltage dip on the
phase that has the ground fault.
2. Interruption of service during a phase to ground fault.
3. High ground fault current is available during arcing
faults.
4. Phase to ground faults cause mechanical stresses in
circuits and equipment caring the fault current.
5. Damage to electrical equipment caused by high phase
to ground fault currents.
6. Complicated protection schemes to provide ground
fault protection. A lack of selectivity is inherent in
ground fault protection schemes. Per NEC, Section
230.95(A) the maximum ground fault sensor trip
setting is restricted to 1200 amps.
a. Thus data center critical power distribution
systems larger than 1200 amps must rely on
zone interlock schemes to coordinate ground
fault protection. These schemes rely on
breaker to breaker communication which is
inherently a single point of failure for the
critical power distribution system during a
single phase to ground fault.
b. Ground fault selectivity can also be achieved
through a limited amount of time delay
between breakers however, this allows a
larger magnitude of fault current during a
single phase to ground fault.
c. The ground fault time delay available for
coordination limits the number of breakers
that can realistically be coordinated in
series.
C. Advantages and Disadvantages of High Resistance
Grounding in Data Centers
Advantages to high resistance grounding in a data center:
1. Reduces burning and melting effects in faulted
electric equipment, such as switchgear, transformers,
cables and rotating machines.
2. Reduces mechanical stresses in circuits and apparatus
carrying fault currents.
3. Reduces electric-shock hazards to personnel caused
by stray ground-fault currents in the ground return
path.
4.
Reduces the arc blast or flash hazard to personnel
who may have accidentally caused or who happen to
be in close proximity to the ground fault.
5. Reduces the momentary line-voltage dip occasioned
by the occurrence and clearing of a ground fault.
6. To secure control of transient overvoltages while at
the same time avoiding the shutdown of a faulty
circuit on the occurrence of the first ground fault
(high-resistance grounding).[1]
7. The load is maintained during a single phase to
ground fault.
8. Allows time to respond, isolate and repair the single
phase to ground fault event while maintaining the
critical load.
9. HRG’s reduces the available phase to ground arcing
fault because the phase to ground fault current is
limited to a maximum of 10 Amps by the HRG
resistor. However there is still the danger of a three
phase arcing fault which must be used to calculate the
protection required to protect the technician working
on the equipment. The limited phase to ground
available current reduces the chance of a phase to
ground arcing fault creating a three phase arcing
fault.
10. During a Ground Fault Condition the ground fault
can be left until an ideal time to clear the fault as long
as the insulation of the cable is rated at 173% [3].
For a 480V system the conductors insulation is rated
at 600V thus a 173% voltage of the phase to ground
voltage brings the two un-faulted phase voltages in
respect to ground to 480V. This does not exceed the
insulation of the conductors. See Figures 1.A and 1.B
for the normal operating characteristics and fault
operating characteristics of the voltage with a HRG.
Disadvantages of using a HRG system must be considered
before implementing the system in the design of any building
but especially a non-industrial application. Some of the
following disadvantages must be addressed:
1. Transient overvoltages up to 250% of nominal
system voltage can occur in the system.
2. Single phase loads cannot be connected to the HRG
system. Life Safety loads must be separated by a
transformer with a solidly grounded neutral.
3. Power Electronic equipment such as the UPS and
TVSS must be selected so that they are able to
operate with a HRG typically connected in a Delta
configuration.
4. Switchgear control transformers cannot connected be
phase to neutral and must be connected phase to
phase.
5. Paralleling UPS modules with an HRG increases the
complexity of the system.
6. During a ground fault event qualified personnel must
go into energized equipment to meter feeders to find
the location of the ground fault. With the Personal
Protection Equipment (PPE) for Arc Flash based on
the three phase arc flash energy. This condition can
be mitigated by the installation of permanent amp
meters on feeder circuits.
D. Data on the HRG System at Fortune Data Center
Table 1: Unit Substation ‘USG-1-4” Ground Fault Data
Electrical
Ground Fault Ground
Ground
Element
“Solid
Fault
Fault
Name
Ground”
“Resistance
“Resistance
Ground”
Ground”
Maximum
Measured
Allowed
Current
Unit
44,508.94
10 Amps
3.5 Amps
Substation
Amps
‘USG-1-4’
UPS-14-1
39,379.49
10 Amps
3.5 Amps
Amps
UPS-DP-14-1 39,493.32
10 Amps
3.5 Amps
Amps
Input of
37,699.58
10 Amps
3.5 Amps
PDU-14-11
Amps
Fig. 1.A: Voltage Characteristics of an HRG System Under
Normal Operating Conditions.
Fig. 1.B: Voltage Characteristics of an HRG System Under
A Single Phase to Ground Operating Conditions.
By utilizing a HRG system the phase to ground fault current
of the system decreases by a factor of ten thousand or four
orders of magnitude. The dramatic reduction of phase to
ground fault current mitigates damage to the system during a
phase to ground fault event. Based on IEEE Research Bus
Duct ground faults are 2.3 times more likely to occur
compared to phase to phase faults. Cable ground faults are 73
times more likely to occur compared to phase to phase faults
and Cable Joints ground faults are 7.8 times more likely to
occur compared to phase to phase faults [5]. Using this
research as a guide the ability the HRG system has the ability
to limit the amount of energy delivered by the most common
type of fault.
E. Challenges for the HRG System
Charging current of the system must be less than the total
ground fault current. For most 480V HRG systems the
charging current is under 2A which can be measured or
calculated [3].
While the overvoltage can be a problem to the insulation of
the cable in a medium voltage system, in a low voltage 480V
system where the phase to neutral voltage is 277V a 173%
overvoltage brings the two un-faulted phases to a voltage of
480 Volts. The cable insulation is rated at 600 Volts which
mitigates the danger of insulation break down avoiding a
double phase to ground fault. Ground Faults should still be
cleared as soon as possible as a second fault will create a
phase to phase condition tripping upstream breakers.
Intermittent ground faults may also cause transient
overvoltages of 250% that damage the insulation of the cables
and can overtime lead to a double phase to ground fault
condition.
PPE is based on the three phase arc flash condition in
electrical panels. In an HRG system if any panel has been
found in the Arc Flash study to be a non approach panel while
energized. Design coordination measures such as permanently
installed amp meters on feeders and or breaker trip units with
a maintenance setting that reduce arc flash energy should be
installed so the location of the fault can be found while
keeping the operators safe.
F. UPS Grounding
There are three choices for UPS grounding: four-wire
source and four-wire load solidly grounded, three-wire source
and three wire load solidly grounded and three wire source
and three wire load – high resistance grounded. See Figures 1,
2 and 3.
Fig. 1: Four Wire Input/ Four Wire Output Solid Ground of a
UPS [6].
In this configuration during a phase to ground fault the UPS
will go to static bypass during the fault to be able to deliver
enough current to clear the fault.
Fig. 2: Three Wire Input/Three Wire Output Solid Ground of
a UPS [6]
In this configuration during a phase to ground fault the UPS
will go to static bypass during the fault to be able to deliver
enough current to clear the fault.
Using a high resistance ground for the UPS that limits the
phase to ground current to a maximum of 10 amps allows the
UPS manufacture to disable the function on the UPS that
when the UPS detects a ground fault to go to static bypass and
instead have an alarm only. This is allowable because the
UPS is able to supply the current to the system in the phase to
ground fault condition. Thus the UPS stays online protecting
the critical load.
G. HRG System and Equations
Type of High Resistance Grounding, there are two types of
high resistance grounding.
The first type of high resistance grounding is used with a
relay or ground fault breaker that allows a certain amount of
current through the resistor for up to 10 seconds and then trips
the protection and continuity of service is broken.
1. The advantage to this is that the system is cheaper to
install and gives the fault time to clear itself. This
allows for more coordination with the ground fault
protective scheme and makes coordinating the system
easier.
2. The disadvantage of this system is that the resistor is
only rated for a limited duration and the expensive
breakers and relaying is still required and your
system still loses continuity of service. Resulting in
loss of critical load.
The other type of High Resistance Grounding has a
continuous duty rated resistor and can have a ground fault
on the system for any amount of time.
1. With this configuration all advantages of High
Resistance Grounding apply.
For the HRG system to have stable operation some of the
following conditions must be met. The ground fault current
must be at least three times the charging current of the
capacitance of the electrical system. Because of system
reliability and the capacity to continue serving critical load a
fully rated continuous duty resistor is the logical choice for a
Data Center application.
R NGR
R NGR
IG
PNGR
VLL
√ IG
XC
Ohms
Ohms
3ICO Amperes
IG R NRG Watts
Total Ground Current
|IG |
I R IC
(1)
(2)
(3)
(4)
(5)
Capacitive Reactance
XC
C
ohms/phase
(6)
Zero Sequence Capacitance
Fig. 3: Three Wire Input/Three Wire Output High Resistance
Ground of a UPS [6].
C
micro Farads uX /phase
Charging Current
(7)
√
C E
3IC
Amperes
Where: f = Frequency in hertz
C0 = Capacitance to Ground in µF
E = Line to Line System Voltage
Cable Capacitance Three-conductor Cable
.
µF
C
D
LOG
F
(8)
(9)
Where: C0 = Capacitance to Ground in mF per 1000 feet.
= specific inductance of insulation
D1 = d + 3c +b for three-conductor cable.
d = diameter of conductor
c = thickness of insulation of conductor.
b = thickness of belt insulation.
Charging Current Estimates
Transformers
0.01 0.001µF
C
Charging Current
.
LE
Amperes
3IC
√
Where:
E = Line to Line System Voltage
L = Line length in ft./1000
Motors
HP
0.05
Amperes
3IC
RPM
Measuring System Charging Current
VLL
R MAX
Ohms
√ · IC
(11)
(12)
(13)
Where:
3IC0 = the estimated charging current
VLL = the system line to line voltage [7]
(10)
Fig. 4: One Line Diagram for location of Phase to Ground Fault Measurement and Test at Fortune Oregon Data Center during
commissioning
Fig. 5: Sample One-line Diagram for location of Ground Fault Location Circuitry
Figure 4 shows the location for where the phase to ground
faults where tested on site at Fortune Oregon Data Center
during commissioning. In Figure 5 the location of phase to
ground fault location circuitry is shown. In Figures 6 and 7
the circuit for measuring the fault current is shown.
Fig. 7: UPS Output Switchboard Ground Fault Sensing
Circuit.
IV. EQUIPMENT COORDINATION FOR SUCCESSFUL HIGH
RESISTANCE GROUND OPERATION
Fig. 6: Unit Substation Switchboard Ground Fault Sensing
Circuit.
To insure a successful commissioning and operation of an
HRG system on a highly reliable UPS critical power
distribution system. The following items must be coordinated
through the design, specification, submittal review and
equipment start up.
Starting with Electrical Switchboards components of the
switchboards must be configured in to work with a HRG.
7
TVSS/SPD or Surge Protective Device must be a delta
configuration. If a 4 wire solidly grounded SPD was installed
the neutral wire would defeat the HRG of the system and
provide the path of least resistance for the phase to ground
fault. The surge arrester ratings must be specified so that
neither the maximum continuous operating voltage nor the
one-second temporary overvoltage capability is exceeded
under system ground-fault conditions [1]. Control
transformers in the electrical equipment must be phase to
phase not phase to neutral as this will also defeat the HRG
system. All control transformers and SPD’s should be rated
for line to line voltage, not line to ground.
During testing the measured fault current of the system was
3.5 amps. Using the equation IG 3ICO Amperes (3)
to find the charging current of the grounding system to be less
than or equal to 1.167 amps. See Figure 4, 5, 6 and 7 for
Measurement of Charging Current diagram.
The HRG itself must be configured for use in the system so
there are no nuisance’s alarms. In this project a PulserPlus
Pro-Low Voltage High Resistance Grounding by Post Glover
was used. Variables that needed adjustment for proper
function included: adjustable pulse rate, adjustable time
delays for fundamental and third harmonic settings to prevent
nuisance’s alarms.
System monitoring is achieved through communications via
RS232, Modbus or Ethernet protocols, this provides 7x24
monitoring via the BMS system.
The HRG reduces the system ground fault current to a
maximum of 10 Amps and for the installed system a measured
3.5 Amps during commissioning. Note the HRG system
should be a continuous duty rated resistor. This is desired so
that during a ground fault there is no interruption in service
while the ground fault location is found and repaired.
The Eaton 9395 UPS software must be enabled to allow the
UPS to operate during a Ground Fault Condition without
transferring to bypasss. On the Eaton 9395 a Neutral
Reference Kit (NRK) must be installed. Battery monitoring
systems should not use ground as a floating reference; they
need an isolated ground reference for control circuits.
For the Power Distribution Unit four items must be
addressed. First the ground fault function on the primary side
of the PDU must be turned off and have Annunciation only.
Second Phase loss/ rotation shutdown must be turned off. The
Over/Under voltage shutdown must be turned off. These three
functions can be turned off during setup by the manufacture.
Lastly on PDU the Auto-restart function must be enabled or
the 480V main breaker will shunt trip on Undervoltage
/Overvoltage phase loss/rotation and ground fault while used
in conjuncture with a HRG during a single phase to ground
fault condition.
V. CONCLUSION
Using an HRG in a critical facility such as a data center
makes sense for a number of reasons.
At Fortune Oregon Data Center the single phase to ground
fault current is limited to 3.5 amps at the Main Switchboard
compared to the 44,500 amps of the same system with a solid
ground. The reduction of the fault current by 4 orders of
magnitude reduces the single phase to ground arc flash
potential energy. The mechanical stress of a single phase to
ground fault on the electrical system is reduced so that a single
phase to ground fault is no longer a catastrophic failure of the
electrical system requiring an immediate outage of service and
potentially extensive repair.
Finally the downstream customer does not notice the single
phase to ground fault. The critical load is continuously fed
thru the UPS during a single phase to ground fault. Allowing
maintenance personal time to find, isolate and repair the fault
while maintaining critical load. Using an HRG system in
Fortune Oregon Data Center, increase the electrical system
reliability for Fortune’s customers through the implementation
of a HRG system.
VI. ACKNOWLEDGMENT
The authors gratefully acknowledge the contributions of
Brian Kane PE, Ed Spears and Steve Emert PE for their
information and review of the paper.
VII. REFERENCES
[1]
[2]
[3]
[4]
[5]
[6]
[7]
IEEE Recommended Practice for Grounding of Industrial and
Commercial Power Systems, IEEE Standard 142-1991, December 1991.
NFPA 70, National Electric Code 2008.
IEEE Recommended Practice for Protection and Coordination of
Industrial and Commercial Power Systems, IEEE Standard 242-2001,
June 2001.
IEEE Recommended Practice for Electrical Power Distribution for
Industrial Plants, IEEE Standard 141-1993, December 1993.
IEEE Recommended Practice for Design of Reliable Industrial and
Commercial Power Systems, IEEE Standard 493-2007, February 2007.
Eaton Technical Brief, UPS Grounding: An Objective Overview,
September 7, 2011.
Post Glover Application Guide, Ground Fault Protection on
Ungrounded and High Resistance Grounded Systems, Copyright 2007
VIII. BIOGRAPHY
Cory David Smith is a Project Manager at ECOM
Engineering and is based in Sacramento, California. Cory has
spent 8 years in various positions at ECOM Engineering
specializing in design of Data Centers and Hospitals. Cory is
a graduate of California State University Sacramento with a
BSEE and a member of the IEEE.
David Lawrence Smith is a Principal at ECOM Engineering
and is based in Sacramento, California. Dave has 30 years of
electrical design, project management, construction, and
commissioning experience specializing in Data Centers,
Hospitals and Critical Facilities. Dave is a member of the
IEEE.