INTELLIGENT ELECTRONIC DEVICE DATA EXTRACTION AT HYDRO ONE
Sheng Hsu
Senior Protection & Control Engineer
Hydro One
sheng.hsu@hydroone.com
Adam Gauci
Field Applications Engineer
Cooper Power Systems
adam.gauci@cybectec.com
INTRODUCTION
Hydro One is the transmission and
distribution utility for the province of Ontario,
Canada. It is one of the largest government
owned utilities in North America with 28,600 km of
transmission lines and 122,800 km of distribution
lines connecting 350 transmission and 1,035
distribution stations.
Within the next few years a large
percentage of Hydro One’s skilled workforce will
be retiring. This has lead to a rethinking of how
soft resources are allocated as they begin the
process of acquiring and training new talent. In
Ontario’s northern regions the travel distances
between substations can be tremendous. It can
sometimes take more than a day’s drive through
treacherous winter conditions to retrieve one
protection event file. Add to this the amount of
time to locate a line fault in heavily forested areas,
and the amount of time and man power required
for a fault can be quite extensive.
Over the last two decades Hydro One has
been replacing electromechanical relays with
Intelligent Electronic Device (IED) protections.
These microprocessor devices contain advanced
communication features which were, until
recently, never fully utilized at Hydro One.
Hydro One primarily uses General Electric
(GE) Multilin Universal Relays for their ‘A’
protection schemes and Schweitzer Engineering
Laboratories (SEL) relays for their ‘B’ protection
schemes (but are not necessarily limited to these).
These relays are part of their engineering
standard and are replacing end of life
electromechanical and microprocessor based
protection devices.
Improving the ability of their staff to
access and perform quick analysis of event data
has allowed a faster response time, decreased
the time of the decision making process and has
made finding the location and origin of a fault
much simpler.
Secure networks were then
designed at the substation level. All IEDs were
connected through an intelligent substation
gateway. In some newer installations SCADA
data is collected directly from the protection IEDs
via DNP3. This eliminates the need for redundant
cabling to a substation Remote Terminal Unit
(RTU).
NON-OPERATIONAL DATA
Each of the IEDs that Hydro One uses in
their protection schemes contains different types
of data. In this section it will be explained what
the different data types are and how they are
collected from the two different types of relays.
GE Multilin Universal Relay Data Types
The GE Multilin relay has three types of
events. These types of events are called: A fault
report (FAULT), an event record (EVENT) and an
oscillography file (OSC).
Fault Report: This report can be triggered by any
operand within the IED. Generally the virtual
output used for tripping the circuit breaker is
selected as the trigger.
Information on the
protection state immediately before and after the
fault report is triggered is recorded into the report.
Event Record: Any operation within the relay
can be selected to be reported as an event.
When an event is triggered the event identifier,
cause and timestamp of the event are recorded
into the record.
Oscillography File: This file is triggered by a
logic operand. It contains waveforms of analog
data at a configurable sampling rate.
SEL Relay Data Types
The SEL relay has two types of events.
The two types of events are called: The standard
event report (EVE) and the sequential event
report (SER). A GE EVENT file is similar to a SEL
SER file, and a GE OSC file is similar to a SEL
EVE file.
Presented at the Eskom Southern Africa Power System Protection Conference, November 12-14, 2008
Standard Event Report: This is triggered when
an output or a trip is executed. This report gives
information on time, currents, voltages, frequency,
etc. The data can be shown by sending an EVE
command to the relay.
output to a file. For the sequential reports, the
gateway periodically issues the SER command
and records any changes to a file. Once a new
file is created it is then sent to the location that is
configured within the gateway.
Sequential Event Report: Relay elements that
trigger a SER event are selected by the user. The
data contained in a SER event is a timestamp,
relay element or condition and element state.
Event Data Locations and User Access
Event Data Collection Protocols
The event files are collected and pushed
to a configurable location by a Cooper Power
Systems Cybectec SMP 16 Substation Gateway.
The gateway uses two protocols with two
completely different methods for collecting the
events.
GE Multilin UR Events Master Protocol: The
GE relays contains a MODBUS register for each
type of event, when a new event is available the
register is incremented. The gateway checks the
register every minute to see if there are any new
event files and if a file is available, it will send it to
the location configured in the gateway.
SEL Events Master Protocol: For standard
reports the gateway periodically issues a HIST
command. It then identifies if any new events
have occurred. If a new event is identified the
gateway will run the EVE command and copy the
Substation Human Machine Interface (HMI):
Some substations contain an HMI for local control
of the station. Hydro One uses Cooper Power
Systems Cybectec Visual Substation software
package running on a Windows XP Professional
substation hardened computer. This software
package maintains a local database of event files
and provides tools for viewing and analyzing
event data.
These tools compliment the
protection commissioning process by allowing
protection engineers immediate access to data
generated by the events.
The substation gateway can be
configured to copy the event data files that are
extracted from the relays to the HMI computer.
The file is then copied to three places. The first
place the file is copied to is a folder where Visual
Substation processes it into the local database. A
second backup copy of the file is stored on the
hard drive of the HMI computer. A third copy of
the event file is copied up to an enterprise level
server via Hydro One’s Wide Area Network
(WAN).
Presented at the Eskom Southern Africa Power System Protection Conference, November 12-14, 2008
Enterprise Interface Server: The event data
files are sent to the server either by the substation
HMI computer or directly from the substation
gateway. To manage event files at the enterprise
level, Hydro One uses the Cooper Cybectec
Event Manager Server software. Event files that
are received by the Event Manager are processed
and stored in a Microsoft SQL server database.
Certain events can be configured to automatically
send an E-Mail or a SMS message. A web-based
interface allows users to access the event data
from anywhere in the company over Hydro One’s
corporate network. The Event Manager allows for
the implementation of a data security policy. A
user database can be created or the package can
be tied into the corporate Active Directory
structure. User rights are assigned determining
what data the user can access, down to a per IED
level. From the web interface users can query the
event file database and see what event files have
been collected from each individual IED. Text
based event files are viewable directly in the web
browser. Oscillography files are converted to the
COMTRADE format and can be downloaded and
viewed in any standard COMTRADE viewer. All
event files can be downloaded in their original
format and viewed in their native vendor
applications. The Event Manager is also able to
directly retrieve events on a periodic interval. The
events are checked by the Event Manager once
every 24 hours to ensure that no events have
been missed during the push-up process. A user
control is available in the web interface that forces
a check of the relay by the Event Manager on
demand.
OPERATIONAL DATA
Traditionally all control and operational
data collected in a substation was done through a
central RTU device connected to the SCADA
system. These devices have a finite capacity in
terms of inputs and outputs. Once this limit is
reached expansion of the RTU can be very costly.
The solution for Hydro One was to use
the communications ability of each relay via DNP3
to read telemetry, protection related alarms and
some basic control. Gathering telemetry from the
protection relays eliminated the need for extra
current
transformers
(CT)
and
potential
transformers (PT) to be wired to the RTU.
Instead of requiring direct wiring of inputs/outputs
for protection alarms, communications cabling is
replacing enormous amounts of copper wire.
Some basic control such as protection scheme
selection and re-closure control are also
performed directly from the relays. Important
Trip/Close controls are still performed from RTUs,
this allows for an extra backup to trip a breaker in
case the protections are not functioning correctly.
By using a substation gateway as a data
concentrator, the substation appears as a single
device to the SCADA master. By having the
SCADA master poll one device instead of many it
greatly simplifies the configuration at the control
center.
NETWORK ARCHITECTURE
Substation Local Area Network (LAN)
The IED Data Extraction Project at Hydro
One was tasked with the creation of substation
LANs at selected stations and making
connections to protection relays with the intention
of retrieving event data. The preference for the
physical connection medium was fibre optic
Ethernet for its inherent resistance to
electromagnetic noise.
Due to the harsh
substation environment and the requirement for
many different connection types, hardened
switches and routers were selected.
If possible, each substation was fitted with
fibre optic cable between buildings. Existing
outdoor cable pans were used and different
building entrance/exits were required.
The
switches were connected with 1000 GB fibre optic
links in a redundant ring structure. If it was not
possible to run fibre optic cable due to site
conditions an 802.11b network was utilized.
A substation gateway was placed in the
control house building and connected via fibre
optic patch cable to the switch. Once the gateway
had been configured it was immediately able to
retrieve events from any IEDs on the network.
Some older relays such as the SEL 200 and 300
series relays do not have network connections.
Serial cables were directly connected between
these types of relays and the gateway allowing it
to retrieve events via RS-232.
When operational data is involved at the
substation level, redundant network architecture is
required. Each switch and router is doubled up in
all buildings. Switches are simply added to the
same ring structure, but are usually placed in
physically different locations and powered from a
separate source. Two substation gateways are
used in a redundant configuration. One gateway
is always active and the other is in a hot-standby
state.
When a failure condition occurs the
standby gateway becomes active without any
interruption to the operational data that is being
sent to SCADA. In this type of situation the
gateway would be connected via LAN or RS232/485/422 to protection relays and RTUs.
Presented at the Eskom Southern Africa Power System Protection Conference, November 12-14, 2008
Some of the protocols in use at Hydro One are
DNP3, MODBUS, MDAC and Tejas.
Substation Wide Area Network (WAN)
Some substations are able to connect to
Hydro One’s synchronous optical networking
(SONET) network, which have direct point-to-point
access to the datacenter at T1 speeds. In
stations that do not have access to the SONET
network, the Bell Canada Frame Relay network is
used at speeds of 56k, 128k or T1. In northern
areas of the province where WAN communication
mediums are limited, satellite communications are
used.
Issues Encountered
Cyber Security: After the events of September
2001 and the North American northeast blackout
of August 2003, the North American Electricity
Reliability Corporation (NERC) was tasked with
creating the Critical Infrastructure Protection (CIP)
standard. Part of this standard includes cyber
security. To remain a part of the interconnected
North American grid, all utilities must strive to
meet these standards.
Each IED that is
accessible through a routable protocol must be
enclosed within an electronic security perimeter.
Any communications paths that cross the
boundaries of these perimeters must be
documented, managed and secured.
The
substation gateway gives the ability of having all
data pass through a single device, which makes
documenting communication outside of the
electronic security perimeter much easier. Hydro
One is still in the process of developing and
implementing a comprehensive solution to meet
all of the NERC CIP standards.
The implementation of this network within
substations has also created a requirement for a
physical security perimeter as per the NERC CIP.
This requirement has given Hydro One another
item to evaluate as they judge the benefits versus
the costs of implementing such solutions.
Network Bandwidth:
With traditional DNP3
SCADA traffic a 56k network link was sufficient.
Now with file transfers, online diagnostic and
configuration tools a 56k link can rapidly reach its
capacity. A single substation fault can result in
Presented at the Eskom Southern Africa Power System Protection Conference, November 12-14, 2008
thousands of events files from multiple IEDs,
which has resulted in an overloading of the link.
In the long term, substations will be integrated into
the SONET network which should resolve most
bandwidth issues. In the short term, the solutions
are to increase the bandwidth of the link if
technically feasible, or to add a second link and
separate operational and non-operational data.
Network Boundaries:
Determining the
boundaries between networks for the enterprise
solution is a challenging task. The enterpriselevel Event Manager must have connection
between two varying networks with different levels
and policies for security, and with completely
different purposes. Currently at Hydro One the
Event Manager is positioned in a DMZ between
the two networks. The best location for the server
is currently being studied in an architecture review
for the NERC CIP Cyber Security project.
Understanding the behaviour of relays: Fault,
Events, Oscillography are all outputs given by the
relays. Before the automated data extraction, staff
would drive to the station and retrieve the
information they were told to extract from the
relays. They were interested in the actual fault
and would feed the fault information manually into
a program and an estimated fault location
calculated. With automation, it was realized that
many of the relays were not fully configured or
were not configured to provide the correct data,
allowing for a lot of data to be sent out through the
network or erroneous data to be sent out. For
example, certain relays will create event files if
there is a 2% fluctuation of the analog values.
This will allow for tremendous amounts of
information that are not relevant unless one is
looking for power quality reports. The actual fault
location can be extracted from the relays within a
minute of occurrence with this technology.
However if the impedance of the line was not
properly calculated and configured into the relay,
the information coming out to dispatch a crew to
investigate will be incorrect, causing further delays
to restore the power. These are just a couple of
issues from a list of facts that came to our
realization during and after the commissioning of
this project.
Implementing IED Data Extraction gave
Hydro One a better understanding of how they
were calibrating their relays. In many instances
when faults occurred in long high voltage lines,
adjacent relays also detected the fault and
captured oscillography traces. Relay events for
the neighbouring lines show that they almost
tripped as well. This allowed Hydro One to better
refine their relay settings and protection design so
that faults in one line do not inadvertently cause
faults on other neighbouring lines. This is not
possible with the older method of retrieving data
where only one relay had information gathered.
CONCLUSION
Hydro One has found efficiencies by
extending network architecture to its substations.
Enabling extended functionalities of protection
relays with cost effective management tools has
allowed for non-operational data to be used for
quickly making decisions when a fault event
occurs. Responsible staff spends less time in
transit to diagnose a problem and is instead sent
directly to solve the problem. This is particularly
important in remote northern regions where
inclement weather conditions can impact travel
times and a lot of stations would require the staff
to travel the whole day just to retrieve one event.
Engineering and operations staffs now have easy
access to current event data for analysis, helping
them improve grid management and predict future
trends. Quicker response to restore the power can
be seen with this service to the operators.
Significant savings have been recognized by the
reduction of direct I/O wiring with copper cables.
New operational data systems are easily
expandable and do not carry the same capacity
limitations as previous RTU systems.
Other benefits from the new architecture
at Hydro One include remote troubleshooting and
configuration of protection and control devices.
Troubleshooting tools include a network statistics
tool and a real-time protocol analyzer which
allows technical staff in the control room the ability
to try to diagnose and correct problems in the field
before field staff need to be contacted.
Configuration of protection, control and telecom
devices can be performed remotely by head office
engineering staff.
Overall Hydro One has benefited from this
initiative by understanding their system better
through increased substation monitoring; this has
achieved the goal of reducing the need for having
personnel on site in substations.
REFERENCES
Matt Efremov, “IED Data Extraction Data Types”
Hydro One.
Jean-Louis Paquet, Anthony Wright, “Strategies
for Automating the Retrieval and the Management
of Non-Operational Substation Data” Western
Power Delivery Automation Conference, April
2004.
Presented at the Eskom Southern Africa Power System Protection Conference, November 12-14, 2008