LONG ISLAND CITY NETWORK
JULY 17 – 25, 2006
INCIDENT INVESTIGATION COMMITTEE
Submitted by:
Robert W. Donohue – Chairman
Wade Malcolm
Edward Neal
Ron Williams
Submitted on:
February 12, 2007
1
Use
This report documents work by the authors who were contracted with Con
Edison. The authors’ opinions, findings, conclusions, and recommendations
are provided solely for the consideration and use of the contracting party.
Any statements, allegations, and/or recommendations in this report should
not be construed as a Con Edison position, policy, or decision, unless so
designated by other documentation.
The report was based on information available to the authors at the time of
publication, and therefore, is subject to change. The use of trade names in
this report does not constitute an official endorsement or approval of the use
of such commercial products.
Acknowledgement
Con Edison provides service to its customers as a result of innumerable, wellperformed tasks. However, focusing on those tasks was not the mission of the
Committee.
The mission of the Committee was to investigate the causes of the Long Island
City network event and recommend actions to limit the possible reoccurrence
of the type of event that was experienced in July 2006. The Committee
necessarily focused on identifying areas for performance improvement.
Consequently, this report offers the authors’ view of where improvement
opportunities exist and where changes may be beneficial. This singular focus
on the Long Island City event and the areas in which change may be beneficial
may mislead the reader to form a too narrow view of Con Edison’s reliability.
The Con Edison network system was reliable during the summer of 2006 with
the exception of the Long Island City network event.
The Committee found that the Con Edison personnel with whom they met
were competent professionals dedicated to safely providing reliable service to
customers and being responsive to resolving problems that may occur during
the provision of that service.
The authors are grateful for the openness and participation of the Con Edison
executives, managers, and engineers who participated in this investigation.
2
Table of Contents
Introduction .................................................................................................................... 5
Executive Summary........................................................................................................ 9
Overview ...................................................................................................... 9
Context ......................................................................................................... 9
Uniqueness of this Event............................................................................. 11
Sequence of the Event................................................................................. 12
Causes ........................................................................................................ 16
Network Shutdown Decision .........................................................................................18
Situation ..................................................................................................... 18
General Criteria for Consider a Network Shut Down................................... 18
Conclusions ................................................................................................ 22
Recommendations....................................................................................... 23
Ratings and Load Cycles ...............................................................................................25
Network Design Overview.......................................................................... 25
Ratings Overview ....................................................................................... 27
Feeder and Transformer Ratings ................................................................. 28
Load Cycles................................................................................................ 31
Conclusions ................................................................................................ 32
Recommendations....................................................................................... 33
Summer Preparation ......................................................................................................34
Overview .................................................................................................... 34
Network Programs...................................................................................... 35
Operations Analysis.................................................................................... 38
Substation Preparation ................................................................................ 39
Conclusion.................................................................................................. 40
Recommendations....................................................................................... 40
Event Progression ..........................................................................................................42
Pre-Event Preparation ................................................................................. 42
Geographic Areas ....................................................................................... 44
Pre-Event Transformer Failures .................................................................. 46
Sunday, July 16th ........................................................................................ 49
July 17th Secondary Cable Insulation Fire ................................................... 49
Fire Damage Caused 1Q17 to De-energize.................................................. 49
Fire Damage Caused 1Q16 to De-energize.................................................. 50
A Cable Failure and a Faulty Circuit Breaker caused 1Q21, 1Q07, 1Q15, and
1Q81 to De-energize................................................................................... 51
Transformers and Secondary Mains Progression ......................................... 53
Why Transformers Failed ........................................................................... 81
Transformer Failures from a Forensics Perspective ..................................... 82
Self-Reinforcing Progression ...................................................................... 82
Conclusions ................................................................................................ 83
3
Recommendations....................................................................................... 86
Trouble Analyses...........................................................................................................88
Overview .................................................................................................... 88
Conclusions ................................................................................................ 96
Recommendations....................................................................................... 97
PVL, WOLF, and RMS .................................................................................................99
Overview .................................................................................................... 99
Poly Voltage Load Flow (PVL) ................................................................ 100
WOLF ...................................................................................................... 103
Remote Monitoring System (RMS)........................................................... 107
Conclusions .............................................................................................. 112
Recommendations..................................................................................... 114
North Queens Substation .............................................................................................117
Situation ................................................................................................... 117
The Improper Operation of the Circuit Breaker 1Q21 ............................... 118
Relay Performance.................................................................................... 123
Relay Targets............................................................................................ 126
Breaker Operations ................................................................................... 126
Conclusions .............................................................................................. 126
Recommendations..................................................................................... 128
Feeder Processing ........................................................................................................130
General ..................................................................................................... 130
Definitions................................................................................................ 130
Events....................................................................................................... 130
System Operations Rapid Restoration Procedure....................................... 141
Cut-in Open-autos..................................................................................... 143
Conclusions .............................................................................................. 146
Recommendations..................................................................................... 150
Recommendations .......................................................................................................152
Recommendations Pertinent to Direct and Root Causes ............................ 152
Recommendations Relating to Broader Issues........................................... 154
Attachment A - Committee Members...........................................................................162
Attachment B - Pre-Event Banks-Off and Open Mains ................................................164
Attachment C - Limited Use Computer Applications ...................................................166
Attachment D - Engineering Applications Defined ......................................................170
Attachment E - Feeder Processing Listing of Definitions .............................................173
Attachment F - Failed Transformers from the Long Island City network event.............176
4
Introduction
In mid July 2006, a prolonged period of high heat and humidity led to
progressively higher electrical demand from Con Edison customers and
concurrent increases in thermal stresses on the electric distribution system. On
July 17th the Con Edison Long Island City network electric distribution system
began to experience operating problems and distribution equipment failures.
The number and severity of the problems and failures increased over the next
several days. Con Edison estimated that ultimately approximately 25,000
customers served by the Long Island City network experienced interrupted
electric service during the event.
Lost or degraded electric service is a serious event for customers and for the
community where they are located. Residential customers may be adversely
affected by the loss of use of electricity powered appliances and devices that
have become necessities. Elevators, water pumps, lights, refrigerators, air
conditioners, fans, special medical equipment, entertainment equipment,
computers and home office equipment, as well as certain types of telephones do
not work. Commercial and industrial customers may be adversely affected by
loss of business and their employees may suffer loss of income. Normal
services from traffic signals to streetlights may be affected and adversely disrupt
the ebb and flow of commerce and private activity throughout the community.
Transportation facilities such as airports, subway, and train service may also be
adversely affected.
Con Edison knows the critical needs that its services meet and that those needs
are highest during periods of high heat in the summer months. Further, it
understands that during those periods of critical need and high heat, the
underground electric distribution system may be vulnerable to high loading and
thermal stresses. During those periods, there are heightened risks of overloads,
equipment failures and customer outages. The Company has designed and built
its underground system to reduce the risks and the Con Edison underground
network is well known for its high level of reliability. However, during the
period July 17 to July 25, portions of the Long Island City network failed and
service to thousands of customers was interrupted.
Con Edison’s CEO, while in the midst of restoration, took immediate action to
determine the causes of the event and how the Company could prevent future
occurrences of similar events. Along with other internal initiatives, he began
assembling an Incident Investigation Committee of industry consultants on
Sunday July 23rd while the event was still in progress. The Committee was fully
5
assembled and began its work during the following week. Biographical
summaries of the members of the Committee are located in Attachment A.
The Committee construed its mission broadly and pursued whatever lines of
investigation it deemed material to its mission. It conducted its investigation
with no constraints placed upon it by Con Edison and had unrestricted access to
company records and employees.
The Committee’s investigation process was intended to identify causes and
make recommendations supported by information provided in the report. The
investigation followed distinct phases of work as follows:
• On the basis of the Committee’s knowledge of utility practices in general,
and Con Edison practices and procedures in particular, it reviewed the
early information available regarding the event then began a process of
reviewing data and documents.
These reviews were undertaken
immediately after the event to ensure that critical information was not
lost.
• After an initial review of that data, interviews were conducted of
engineers, operators, managers, and executives who were either directly
involved in the event, had oversight responsibilities for departments or
functions involved in the event, or were responsible for event preconditions.
• The findings were analyzed to identify areas where the Committee
believed it should focus its investigation to best achieve its mission. The
Committee evaluated the Long Island City network event from the
perspective of transformer and secondary mains failures, feeder
processing, and the North Queens substation. The Committee identified
direct and root causes of the event, and provided recommendations
pertinent to those causes. From there, the Committee reviewed broader
issues which may reveal opportunities for Con Edison to take actions to
improve overall performance and reliability, and to reduce the possibility
of network outages occurring in the future. Those issues included
engineering and planning, trouble analysis, summer preparation, network
shutdown, engineering applications, and the remote monitoring system.
• The Committee has included an Executive Summary that provides the
most salient points of the report. The subsequent sections are organized
6
to present the specific areas on which the Committee focused its
investigation in the following order:
o
o
o
o
o
o
o
o
Network Shutdown Decision
Ratings and Load Cycles
Summer Preparation
Event Progression
Trouble Analysis
PVL, WOLF, and RMS
North Queens Substation
Feeder Processing
Based upon these analyses and findings, the Committee has proposed a number
of recommendations which it believes will benefit Con Edison’s customers and
improve system operations and performance.
During the investigation process, the Committee periodically reviewed its
progress, including preliminary findings and conclusions, with various
individuals at Con Edison. The purpose of those periodic reviews was twofold:
first, the periodic reviews were intended to promote discussions that may clarify
a topic or identify information that may have been missed during the
Committee’s data and document reviews or interviews; and second, to keep Con
Edison informed as to the progress being made by the Committee toward its
completion target.
The Committee received information from Con Edison employees who were
interviewed or who provided calculations, maps, documents and other data upon
which the Committee relied to reach its findings and conclusions. Con Edison
conducted extensive analyses as part of their internal review of the Long Island
City network event. The Committee relied upon the outcome of internal
technical analyses and did not believe that duplication of those studies would be
productive. These analyses are mentioned in the report. Although the
Committee is confident about the accuracy of the information provided, this was
a fast developing, complex event and additional information may be found in
the future.
7
The writing of this report has attempted to meet the dual objectives of being
understood by persons who are familiar with the industry but may not fully
understand all of the technical subjects presented herein and also of meeting the
information needs of Con Edison executives who have deep knowledge and
understanding of those technical subjects.
The Committee believes its investigation is responsive to Con Edison’s request
for an incident investigation and this report fulfills its mission.
8
Executive Summary
Overview
Con Edison provides electric services to approximately 3.2 million customers
located within a territory of 604 square miles with an estimated population of
9.1 million people. The Company distributes its electric service over 30
thousand miles of overhead and 92 thousand miles of underground electric
circuits through 262 thousand manholes and service boxes. It supplies service
across 57 underground secondary distribution networks. One of which is its
Long Island City network which distributes service to nearly 115 thousand
customers located in an area measuring approximately 9 square miles in the
New York City borough of Queens.
The Long Island City network receives electric power from the North Queens
substation then distributes the power through twenty-two 27,000-volt (27kV)
primary feeders installed throughout the network. Each feeder is identified with
an alphanumeric name, such as 1Q01. In total, per Con Edison’s 2006
Distribution Manual, the feeders supply power to 1,194 transformers distributed
throughout and connected to a grid of underground and overhead secondary
mains to which the individual customer services are connected.
The Long Island City network serves an area with a population of
approximately 350,000 people. Its population is similar to the population
residing within the city limits of Buffalo, Cincinnati, Pittsburgh, Miami, Tampa,
Minneapolis, or St. Louis. Because of its size, and strategic location, the
consequences of shutting down the Long Island City network require careful
consideration, particularly true during a heat wave when customer need is at its
highest, and at a time during which the network may have sustained damage
that may complicate and delay re-energizing it.
Context
Assuring service reliability and quickly responding to power outages is
critically important to Con Edison. The Con Edison engineers have designed a
highly reliable system and maintenance and operating personnel ensure
equipment performs as designed. Con Edison prepares for outages by preassigning emergency management roles to its personnel and by conducting
annual emergency drills to test the capabilities of personnel, processes, and the
technologies upon which they rely to be effective. Special emergency
management organizations are established and guided by detailed procedures to
ensure communications and decision-making are of the quality needed to
9
quickly respond to emergencies including power outages. During emergencies,
personnel are mobilized from within Con Edison and from external sources such
as contractors and other utilities to minimize resource shortages that would
delay a response. After emergencies, post-event critiques are conducted to
identify areas for improvement and action plans are developed and implemented
to drive further improvement in system design, management processes, and
support technologies.
Con Edison’s preparation, mobilization, and
improvement practices conform to standard utility practice. Those practices
ensure that utilities become increasingly proficient at responding to
emergencies. Especially, they ensure increasing proficiency at responding to
emergencies similar to those they have experienced in the past and therefore
within the scope of emergencies anticipated by their system designs,
management processes, and support technologies.
Con Edison’s distribution network system is designed with sufficiently
redundant equipment to provide high levels of reliability. This is referred to as
N-2 contingency design criteria. It is uncommon in the utility industry. It is
costly but highly reliable. It is also complex. The large number of components
and interrelationships associated with the design redundancy adds to the
complexity and the Con Edison system, particularly the Long Island City
network, is large. Con Edison manages this complexity through the use of
technology that monitors, simulates, and predicts system behaviors well enough
in advance to allow operators to anticipate problems and prevent them. The
technology is designed to support the complexity of the system that would be
expected during normal operations through N-2 contingency operations at peak
design loads and, in some case, slightly beyond N-2. The Long Island City
network event caused the network to be operated well above N-2. Even so, Con
Edison managed to prevent a complete shutdown of the network and limit the
extent of the power outages. The Committee believes that most of the
damaging aspects of this event occurred well beyond the N-2.
N-10 contingencies were experienced during this event. Yet, Con Edison’s
Long Island City network continued to provide service to most of its customers.
Operating at such a high level contingency is bound to cause damage and
damage occurred. It is not reasonable in cases such as the Long Island City
network event for Con Edison to shut down a network whenever it exceeds the
N-2 contingency design criteria. Networks have been successfully operated at
third and fourth contingencies and higher without them experiencing significant
damage or resulting in the need to shut down those networks.
10
So, a quandary is created. It is not reasonable for Con Edison to shut down
networks simply because they exceed the N-2 design criteria. It is not
reasonable for Con Edison customers or shareholders to fund unlimited
redundancy of equipment needed to increase the N-2 criteria in order to prevent
shutting down networks when conditions exceed N-2. Con Edison is funded for
and operates an N-2 system that is expected to operate well above N-2 and not
interrupt service to customers.
Con Edison’s distribution network design is one of the most reliable in the
world. Its customers have become accustomed to the high reliability and expect
it. Yet resources are finite and there is a limit in terms of technology,
investment dollars and human resources that prohibit any system from being
designed as completely free of failure.
In this context, the Committee has critically analyzed the Long Island City
network event and has recommended ways Con Edison can learn from this
occurrence in a way that should improve its daily and emergency operations.
Uniqueness of this Event
Con Edison’s strategy to respond to the Long Island City network event focused
on encouraging customers to reduce load and restoring primary feeders as
quickly as possible in order to ensure that feeder outages in the network did not
escalate. This approach has been effective for many years and, in the last seven
years, expedited feeder restoration has become an emphasized strategy. Feeder
restoration time has been significantly reduced and has proven to be an effective
method in addressing heat related events.
Several circumstances made this event unique. The sequence of feeders that deenergized was very close to being the worst possible sequence Con Edison
could face. While the distribution system has been designed to withstand
multiple feeder outages, in this particular case, the sequence of feeder outages
progressed very fast. The network went from a zero contingency status to a
fifth contingency in about six hours. As such, the time to restore feeders before
problems multiplied was significantly reduced.
The number of feeders de-energized and transformer banks off early in the
event was unremarkable. However, the unusual combination of feeders that deenergized created a scenario in which the secondary grid became the medium
for instability to spread.
11
There were several reasons why the uniqueness of the event went unrecognized:
• Primary feeder loadings were within their operating limits
• The customer call volume was low and one would deduce that
minimal customer complaints would mean minimal secondary damage
• Looking at these parameters, and since no similar event had transpired
in Con Edison history, there was little relevant experience that existed
to draw upon to recognize the situation
Several key factors added to the complexity of the event and differentiated it
from past experiences. The event was compounded by the sheer size of the
Long Island City network and number of primary feeders. The length of the
feeders and the number of attached components, along with the territory the
feeders cover, made it difficult to recognize any patterns of operation and
quickly diagnose them.
The speed at which pockets of high secondary load occurred and grew was
unprecedented based on Con Edison’s experience. The high demand placed
additional strain on the remaining energized transformers and the associated
secondary cable still supplying the areas. The secondary cables ultimately
failed and became a vehicle for primary feeder damage. The event was the first
that had evolved in this way in Con Edison corporate memory.
The fact that call volume and primary feeder loadings were low reinforced that
the strategy of restoring as much of a feeder as quickly as possible in an orderly
sequence was the correct approach. In fact, the approach did address many of
the aspects that needed to be addressed. However, the nature of the event meant
there were few indicators that might lead one to understand that some other
underlying issues were going unnoticed until the problems were recognized well
into the event.
Sequence of the Event
Prior to the start of the Long Island City network event, as a result of high
demand and transformers that were not in service, secondary mains were
overloaded in the area of 30th Avenue and 44th Street in Long Island City.
The Long Island City network event was initiated on July 17th, when the
insulation on these secondary cables began to burn in the vicinity of 30th
Avenue and 44th Street. The exact cause of the fire remains uncertain, as the
level of loading should not have been enough to cause the cable insulation to
burn. The burning insulation caused damage to the cables on feeders 1Q17 and
12
1Q16 in the conduit run and adjoining manhole. When the feeders were
damaged sufficiently to cause arcing to ground, feeders 1Q17 then 1Q16
automatically de-energized at approximately 15:50 and 16:22 p.m. respectively.
The loss of 1Q17 and 1Q16 caused additional transformers in the network to deenergize and further deteriorated conditions in the vicinity of 30th Avenue and
44th Street. This event caused a second contingency in the Long Island City
network. It is notable that the Con Edison system is designed to withstand any
second contingency on a peak load design day such as was being experienced
on July 17, 2006.
The secondary grid in the vicinity of 30th Avenue and 44th Street was further
weakened when secondary cables, in nearby structures had to be cut to isolate
the burning secondary cables. At approximately the same time, secondary fires
were reported one block south of this location. These fires also required
secondary cables to be cut to extinguish the source of energy supplying the short
circuit current and contain the fire and secondary cable failures. This cutting of
additional secondary cables further reduced the support of the 30th Avenue and
44th Street load pocket from the south.
At 18:47 on July 17th, while restoration efforts were in progress on feeder 1Q17
and 1Q16, feeder 1Q21 experienced a failure of a cable termination connecting
to a transformer. The cause of this failure could not be determined as the failure
specimen was not recovered. Feeder 1Q21 was supplied from Bus Section 3S at
North Queens substation. The circuit breaker on 1Q21 failed to operate as
designed. To remove the primary fault, the bus tie breakers opened and deenergized the entire Bus Section 3S and de-energized four additional primary
feeders 1Q07, 1Q15, 1Q81, and 1Q21. This was a proper operation upon the
failure of the 1Q21 breaker to operate. The cause of the 1Q21 circuit breaker
failure was a sliding contact problem with the control circuit connectors on the
circuit breaker. This was compounded by a wiring design issue in the circuit
that would have alerted substation operators to a potential problem.
Feeder 1Q15 was rapidly restored within minutes. Feeder 1Q81 is a nonnetwork feeder that supplies load to and from the NRG Gas Turbines; it was not
part of this investigation as it was not a feeder in the Long Island City network.
The circuit breaker for feeder 1Q81 also had the same sliding contact problems
as the 1Q21 circuit breaker.
At 18:54 on July 17th, Con Edison implemented an 8% reduction of the
operating voltage in the Long Island City network in accordance with its prior
directives. This voltage reduction was effective in reducing load in the network.
13
This action was successful in reducing the overall network demand and likely
avoided some damage to the network.
At 19:09 on July 17th Con Edison attempted to “cut-in” and re-energize 1Q07
but it immediately “opened-auto” and de-energized. 1Q07 feeder was in service
previously but had de-energized when the Bus Section 3S tripped. The cut-in
open-auto was caused by the magnitude of the inrush currents exceeding the
instantaneous phase over-current relay operation points. The cut-in open-auto
caused delays to the restoration of 1Q07. During multiple contingency events
such as this Long Island City network event, delaying the restoration of a feeder
increases the time other network components are at a higher load and increases
the risk of further component failures. During the Long Island City network
event, the magnitude of the inrush currents which exceeded the instantaneous
phase over-current relay operation points delayed the return of four feeders
including 1Q07 at various points in the feeder restoration process.
Later that evening, when 1Q17 was restored, Con Edison crews attempted to
restore feeder 1Q16 but were unsuccessful and later discovered a faulty cable
joint. The cause of that fault was thermal degradation over the long term.
These types of faults are known to occur on thermally sensitive joints.
On July 18th at approximately 08:30, about sixteen hours after the start of the
Long Island City network event, other secondary cable insulation burned and
the damage communicated to primary feeder 1Q02 and it de-energized.
Overloading as a result of multiple feeder outages was the probable cause of this
secondary mains fire.
At 09:33 on July 18th, Con Edison crews returned 1Q16 to service. To expedite
the return to service of 1Q17 the previous evening and 1Q16, Con Edison crews
had live-end capped five transformers in the vicinity of 30th Avenue and 44th
Street. The combination of pre-event transformers not in service, and the five
transformers live-end capped, created higher loads on the remaining equipment
supplying the local load pocket in the vicinity of 30th Avenue and 44th Street.
Additionally, when 1Q16 was returned to service, crews had also live-end
capped two transformers located in another area of Long Island City. The lack
of those two transformers caused secondary mains to become highly loaded in
that area as well.
The manner in which 1Q17 and 1Q16 were restored to service was beneficial to
the overall Long Island City network as it restored approximately 136
transformers and reduced the contingency in the network. However, live-end
14
capping the five transformers provided little benefit to the local load pocket in
the vicinity of 30th Avenue and 44th Street. Even with 1Q17 and 1Q16 back in
service the local condition was stressed because:
• Live-end capping the transformers required load to be supplied from
transformers located further away, thereby increasing the load on the
transformers and secondary cable mains in the area
• Due to the necessity of disconnecting the secondary cable to
extinguish the manhole fires, the number of secondary cables and
pathways were reduced. This reduced the number of pathways that
electricity could flow to serve customer load and increased the load
each pathway was required to carry
Just before noon on July 18th, the transformer in VS 7995 failed as a result of all
three phases inside the transformer arcing to ground. This de-energized feeder
1Q17 for the second time, approximately 12 hours after it had been restored to
service.
On July 18th, as heat and load increased throughout the day, starting at about
three in the afternoon until just after midnight, six feeders were de-energized as
a result of transformer failures.
There were five feeders out of service at mid day on July 18th (1Q21, 1Q20,
1Q02, 1Q01, and 1Q17) which placed stress on the remaining secondary mains
and transformers in pockets of the Long Island City network. As load increased
throughout the day and approached its peak in the evening, along with cut-in
open-autos and other delays in restoring feeders, the conditions were set for a
self-reinforcing progression described as follows:
• The progression began when transformers in a given area had become
de-energized
• Customer loads still existed in the area
• Secondary mains and transformers surrounding the area picked-up the
load of the de-energized transformers
• Secondary mains and transformers that picked-up this load became
highly loaded
• Secondary mains cable insulation and transformer oil temperatures
increased and became subject to overheating and failing
• When secondary mains and transformers failed, load shifted to other
mains and transformers that became more highly loaded and became
subject to failure which reinforced the progression
15
The self-reinforcing progression continued until it was arrested by a
combination of remedial actions by Con Edison crews, secondary limiters
opening, and secondary mains damaged extensively enough to isolate
transformers from the secondary grid.
At mid-day on Thursday morning July 20th, Con Edison mobilized its Corporate
Emergency Response Center (CERC) to coordinate and support the repair of the
network secondary system. During that day, Con Edison came to realize that
more customers could be out of service than was reported by its information
systems. On the evening of July 20th, Con Edison conducted a survey of
damaged areas in the network that indicated approximately 25,000 customers
may have been affected.
Although the full CERC was mobilized on Thursday July 20th, executives and
senior managers had been engaged in providing high-level oversight from the
start of the event on Monday night July 17th.
Con Edison reported that on Friday July 21st, approximately 25,000 customers
were out of service. By Sunday at midnight, approximately 80% were restored
to service. By Tuesday July 25th, service to all had been restored.
Causes
Direct Causes
The Committee considers a “direct cause” to be all causal factors that resulted in
a primary feeder being de-energized. There were five types of direct causes as
follows:
Secondary Cable Burnout
Secondary burnouts resulted in six primary feeders opening automatically
causing the de-energizing of approximately 300 transformers. This resulted in
load being shifted to other transformers and secondary mains. Some of the
transformers and mains failed due to exceeding design limits that caused breakup of the secondary grid in certain locations. This interrupted or degraded
service to customers.
Circuit Breaker Failure
There was one circuit breaker that failed to operate properly and resulted in deenergizing three network primary feeders. This resulted in 157 network
transformers de-energizing which shifted load to other transformers and
16
secondary mains, some of which subsequently failed due to exceeding design
limits. This caused break-up of the secondary grid in certain locations which
interrupted or degraded service to customers (there was a second circuit breaker
failure that was on 1Q81 that did not involve the Long Island City network).
Relay Settings
There were four instances where the magnitude of the inrush currents exceeded
the instantaneous phase over-current relay operation points resulting in
extending the duration in which feeders and their associated transformers were
de-energized. This caused other transformers and secondary mains to carry load
that had been shifted to them resulting in higher thermal stresses.
Cable and Joint Failures
There were two instances where primary cable failed, one instance where a
cable failed as a result of a non-secondary related manhole fire, one instance
where a cable failed as a result of a defective duct and fourteen instances where
joints failed causing or extending feeder outages (through cut-in open-auto
operations).
Transformers Failures
There were ten transformers that failed due to overheating, one that failed due to
a defective primary bushing, and two that failed due to corrosion and leaked
dielectric fluid. Transformer failures caused twelve open-auto operations on
eight different feeders at various times in the Long Island City network event.
Root Causes
The Committee considers the root cause(s) to be causal factors that initiated the
event or significantly contributed to it becoming a large event. The root causes
are:
• The Long Island City network event was initiated when a secondary
cable burned which resulted in two feeders (1Q17 and 1Q16) deenergizing.
• A primary conductor cable failed which caused the third feeder
(1Q21) to de-energize.
• The 1Q21 Circuit Breaker failed to open which resulted in the deenergizing of network feeders1Q21, 1Q07, 1Q15, and non-network
feeder 1Q81.
• The magnitude of inrush current was higher than the relay setting
which resulted in delayed return to service of feeder 1Q07.
17
Network Shutdown Decision
Situation
The Long Island City network serves an area with a population of
approximately 350,000 people. Its population is similar to the population
residing within the city limits of Buffalo, Cincinnati, Pittsburgh, Miami, Tampa,
Minneapolis and St. Louis. The consequences of shutting down the Long Island
City network are significant, particularly during a heat wave when customer
need is at its highest, and at a time during which the network has sustained
damage that may complicate and delay re-energizing it.
To justify shutting down the network during the Long Island City network
event, Con Edison would have had to conclude that shutting it down would have
resulted in less negative consequences to the Long Island City community and
the distribution system than not shutting it down. Neither case is easily
quantifiable, especially in the midst of the event. The key factor to consider in
both cases is public safety.
Clearly, safety is highest when the network is operated normally and its
components are within their design limits. When a network becomes involved
in an event such as the Long Island City network event, there are safety
concerns associated with overloaded equipment that may result in fires,
explosions, and stray voltage. If to avoid those potential safety issues, the only
other alternative is to shutdown the network, then consideration must be given
to the safety concerns associated with community-wide power outages that
affect the wide-range of public services, infrastructure, and other electricitydependent needs of people.
General Criteria for Consider a Network Shut Down
Con Edison Engineering Specification EO-4095 “Distribution System Operation
Under Contingency Conditions,” Section 10.0 “Underground Distributed
Networks,” provides basic steps to be taken during “Substation Contingencies”
(Section 10.1) and “Distributed Network Contingencies” (Section 10.2).
The Committee reviewed Section 10.2 and offers the following discussion
regarding Incident Command’s compliance with key provisions of that Section.
Section 10.2 provides guidance regarding what parameters to monitor during
contingencies such as:
18
• Review of load flow using the WOLF program including review of
exception reports concerning feeders, transformers and the secondary
grid to detect current overload conditions, and projected overloads
based on expected maximum load.
• Continuously monitor feeder load, transformer load, and the status of
the secondary grid including reports of manholes smoking or on fire.
Then, as necessary, the specification provides guidance to take actions to reduce
load, cool transformers, and cut-clear affected secondary mains.
Section 10.2.6 indicates that:
“If the actions have an impact on eliminating the overloads on the primary
feeders, there are no reports of cascading manholes fire and there are no
network transformers loaded beyond the allowable limits when flooded,
continue to monitor the network as it is progressively returned to normal
conditions.”
Section 10.2.7 indicates that:
“The shutdown of the distributed network should be considered if the above
actions are not successful in correcting the emergency condition.”
Section 10.2.7 does not say, but the Committee assumes, that the term
“emergency condition” means that the shutdown of a distributed network should
be considered when these or a combination of these exist:
1) Overloads on primary feeders can not be eliminated, or
2) There are reports of cascading manholes on fire, or
3) Network transformers are loaded beyond the allowable limits when
cooled
The risk of an incident that subjects the public to an unsafe condition is
heightened when feeders are overloaded, manhole fires occur, and/or
transformers are loaded beyond allowable limits. Therefore, the Committee
understands this section of EO-4095 to mean that a shutdown should be
considered when, in the judgment of the decision-makers, the occurrence of
those, or a combination of those, three conditions has become inevitable. At the
point of inevitability, shutting down the network should be considered as one
option to correct the emergency condition. With that understanding: The
19
Committee believes that the criterion in EO-4095 requiring consideration of
shutting down the network is appropriate.
Adherence to the General Criteria during the Long Island City
event
As explained above, the Committee understands EO-4095 to require that: The
shutdown of a distributed network should be considered if:
1) Overloads on primary feeders can not be eliminated, or
2) There are reports of cascading manholes on fire, or
3) Network transformers are loaded beyond the allowable limits when
cooled
With regard to the Long Island City network event:
• Primary feeder overloads were not frequent and the four that did occur
were quickly eliminated when detected
• The Committee was advised that manhole incidents were being
monitored in the Control Room. However, the Committee saw no
evidence of a process for systematic analysis of those three emergency
conditions. The Committee understood that the frequency and number
of manhole events were being considered.
• Network transformers were highly loaded and some transformers
failed during this event prior to reaching their allowable limits.
Incident Command was responding to the high loading and had the
engineering group prioritize transformers and issue orders for them to
be cooled. In addition, general and specific load reduction appeals
were made.
• How this information was evaluated in considering whether or not to
shutdown the network was not clear but Incident Command indicated
that these were the key indicators being considered when discussing a
possible shutdown of the network.
During the event, Incident Command considered shutting down the Long Island
City network. The Committee was advised that the motives for that
consideration were linked to one or more of the three “emergency conditions’
described above.
20
Further, it is the view of the Committee that determining strict compliance with
this provision of EO-4095 is very subjective because the requirement to
“consider” is not testable without a requirement to document the information
leading to the consideration and without documenting the rigor of analysis
during the consideration. Based upon interviews conducted, and the subjective
nature of the provision of EO-4095, the Committee concludes that Incident
Command complied with this provision of EO-4095 because they were aware of
and did considered feeder loads, manhole events, and transformer loads in their
management of the event.
Specific Criteria for Considering a Network Shut Down
Con Edison’s EO-4095 specifies the following:
“Conditions Requiring Immediate Network Shutdown - To the extent possible,
the Regional General Manager of Electric Operations should determine in
advance which network conditions would require network shutdown and convey
this to the Chief District Operator. For example, with several feeders already
out of service, determine the condition that an additional tripping of one, two or
three feeders would dictate network shutdown. If the feeders trip, the network
should be shut down immediately without evaluation and decision making, as
directed by the Chief District Operator or his designee.”
The “Regional General Manager of Electric Operations” is the position closest
to the situation and therefore has the best understanding of the three criteria
listed in EO-4095 and described above and it is therefore appropriate that the
position have the delegated responsibility to determine which specific
conditions would require shutdown of the network. Delegated responsibility
authorizes one person to act on behalf of another. It is delegated not transferred.
Others who have the responsibility include the management chain above the
Regional General Manager of Electric Operations which in this case, includes
the Vice President of the Brooklyn/Queens Region and the Senior Vice
President of Electric Operations. All of those people were appropriately
involved in considering whether or not to shutdown the Long Island City
network.
Late Tuesday night July 18th, Long Island City network went from a 7th
contingency to a 10th contingency in about 30 minutes. At that point the
Regional General Manager of Electric Operations in consultation with his Vice
President and the Senior Vice President, considered shutting down the network.
Although there were some feeder overload conditions, he expected two feeders
to be restored to service soon. The network load had been reduced and was
21
expected to continue to do so. Ambient temperatures were declining. Some
transformers were overloaded but it appeared they could be cooled. Customer
call volumes remained low with only 840 trouble tickets from customer calls on
Tuesday and 19 trouble tickets from manhole events. Incident Command
decided that the network would be shut down if it reached an 11th contingency.
It did not. Therefore:
• The Committee believes that Incident Command complied with
requirements provided in specification EO-4095 with regard to
determining in advance “which network conditions would require
network shutdown.” And, they appeared to have been prepared to
shut-down the Long Island City network had those conditions been
met.
Conclusions
• Within the context of a Long Island City network type of an event,
deciding to shutdown a network carries significant, unpredictable, and
negative consequences to the community.
Weighing those
consequences against the known consequences of an outage event inprogress, may rarely result in a decision to shut down the network.
Shutting down the network may be a viable option when the inprogress event has resulted in a public safety incident in which case
the decision will have been made too late.
• The fact that an event such as this one, could lead to a decision to
shut-down the network, is problematic for Con Edison. It suggests
that at some point, having been unsuccessful in arresting the event
with the tactics they were employing, Incident Command thought that
possibly, the only remaining action available to stop the event, was to
shutdown the network, a very difficult decision to make.
• In contrast, the decision is more easily reached if it were driven by a
problem at the North Queens Substation. Major equipment damage at
the substation would mean that the Long Island City network would
be shut down for weeks or even longer before a plan could be
developed and implemented to restore the network. In the case driven
by a problem at the substation, it would be a choice between a short
network shutdown to protect major substation equipment and an
extended network shutdown if the equipment were damaged. That is
different than in the case of a Long Island City network type event in
which it becomes a decision between two choices. One, continue to
22
operate the network and incur damages or two, shutdown the network
and incur unknown consequences. The magnitude of the damages and
the consequences are different depending upon when during the event
a decision is made to shutdown the network.
• The Long Island City network is one of the largest of Con Edison’s
secondary networks. It has 22 feeders and 1,194 transformers and
network protectors. The design objective is to maintain a “uniform
and well-diversified intermesh of feeders and transformers.”
However, during multiple contingency operations, with changing
loads and shifting secondary load paths, maintaining a well-diversified
intermesh of feeders and transformers while ensuring that feeders,
transformers, and secondary mains remain within design limits, is a
very complex task.
Con Edison’s people and systems have
successfully achieved this task. But, the task was more complex in the
Long Island City network than would be a similarly configured but
smaller network.
Recommendations
Network Shutdown
• Consider ways to reduce the significance and negative consequences
to the community of shutting down the network by making the
network smaller or effectively smaller. This should include, but not
be limited to, considering a new substation to split the network and the
consideration of primary and/or secondary grid sectionalizing
capability.
• Consider changing the design criteria, now based solely on station
capacity, to include local demographics and the community impacts of
a network shutdown.
• Consider splitting the Long Island City network to reduce average
feeder loading, length and shut down impact to the community.
• Investigate and as appropriate develop detailed plans to manually
sectionalize the secondary into predetermined sections.
• Develop a training program to ensure high levels of analytical skills
and strategic decision-making for Incident Commanders and others
involved in the process of evaluating multiple contingency situations
and deciding the most effective tactical responses.
• Organize the Incident Command team so that the unique responsibility
for the analysis and remediation of each of the three considerations
23
provided in EO-4095 (and listed below) are clearly assigned in the
structure of the response team.
o Overloads on primary feeders can not be eliminated, or
o There are reports of cascading manholes on fire, or
o Network transformers are loaded beyond the allowable limits
when cooled.
Network Size
• Utilize “Jeopardy” analysis as part of the criteria in evaluation of the
need for reinforcement and prioritization of relief projects.
• Investigate what role the Jeopardy application should play in the
evaluation of a possible network shutdown decision.
24
Ratings and Load Cycles
Network Design Overview
The vast majority of Con Edison customers served by the Long Island City
network are provided power directly from a grid of cables called secondary
mains supplied by transformers dispersed throughout the grid. Sections of
secondary mains are installed along nearly every street in Long Island City and
each section is connected to other sections at nearly every street corner.
Figure 1: Depiction of a Con Edison grid of secondary mains, transformers, and customers
(The source of this depiction is Con Edison)
Compared to typical electric distribution designs, the secondary grid is an
extremely robust design with very high reliability as measured by traditional
metrics such as SAIDI and SAIFI. Its strength is also demonstrated by the
amount of damage the grid can withstand before customers begin to experience
service interruptions. Generally, it would require multiple primary feeder and
secondary mains outages to de-energize a city block of customers in the Long
Island City network.
The grid of secondary mains is supported and supplied by secondary
transformers energized by independent and separate primary feeder cables that
are not interconnected except at the area substation bus. The feeders are
connected at the substation such that the loss of one bus section would not
create a local second contingency. These feeder cables are connected to the
secondary grid, generally through 25 – 50 underground network transformers.
The objective is to install transformers at locations in the secondary grid where
25
the transformers will: “Form a uniform and well-diversified intermesh of
feeders and transformers.”
Figure 2: Depiction of a Con Edison network showing the substation, primary feeders,
transformers, secondary mains, and customers (The source of this depiction is Con
Edison)
As much as possible, the primary feeders are designed to be kept separated so
that damage to one will not affect any others. Two feeders may be banded
together sharing the same ducting in some areas, but no more than two. Bands
of feeders are kept off the same street if possible. If two bands cross, special
criteria apply to prevent problems from one band affecting the other.
Maintaining a well-diversified intermesh along with applying special rules
regarding feeder banding and separation adds additional robustness to the
design of Con Edison networks.
The design of the number and size of secondary mains, and primary feeders, as
well as the number, size, and location of transformers, is such that the loss of
any two primary feeders would not result in interruption or degradation of
service during peak design summer conditions. “Size” in this context refers to
the physical properties of the component that ensures it can sustain the desired
loading.
26
Ratings Overview
Ratings are extremely important to determine the need for network
reinforcement and to determine the ability of the network to operate under
multiple contingency conditions such as during the Long Island City network
event. Con Edison has developed a series of engineering specifications and
engineering applications which are used to determine proper ratings.
The ratings are temperature adjusted and use Con Edison design Temperature
Variables (TV) to make certain that all facilities are rated using peak load data
in accordance with the Company’s design criteria.
The rating process is accomplished using methods that are outlined in the
engineering specifications. The preferred method utilizes the Poly Voltage
Load Flow (PVL) program which was developed by Con Edison to calculate
feeder ratings. The PVL program has a feeder rating feature that determines the
load and rating for the limiting circuit component. It does so for normal
operation and up to second contingency operating conditions for each feeder in
a given network. The relationships between normal and emergency loads in the
network are determined automatically by PVL, but only if the network is
correctly modeled.
The PVL model determines how loads are distributed to other transformers and
their supply feeders. The model then determines if the feeder becomes
overloaded as a result of picking-up load from de-energized feeders and if it
does, the model identifies where and what system components require
reinforcement. The load flow program used at Con Edison is a powerful tool. It
has been periodically updated to improve its effectiveness as an important
engineering application used to ensure the reliability of the Con Edison
networks.
The quality of results from the PVL model is directly related to the quality of
the network connectivity model. The connectivity model is defined as the
database depicting the interconnection of all of the system components. The
connectivity model and associated loadings must be kept accurate at all times to
reflect the changing connectivity and additions that Con Edison makes to its
distribution system. If not done, the quality of results from the PVL model (as
well as other application that rely on the PVL model) will be degraded.
Reinforcement plans to eliminate overloads are developed as needed by the
regional engineering department. If it can be achieved economically, the
engineers will provide a reasonable amount of margin to meet additional
27
capacity to cover several years load growth. Once the reinforcement plans are
developed, the proposed modifications are incorporated into a test model of the
network to evaluate their effectiveness in eliminating the overloads.
Reinforcement may include modifications such as: replacing existing equipment
with larger capacity equipment, building new feeders, or transferring load from
one feeder to another by disconnecting a feeder branch and connecting that
branch to a different feeder.
New feeder ratings for both normal and emergency operating conditions may
then be obtained from the PVL test model or may be calculated in accordance
with specification EO-2048 based upon the new configuration, projected KVA
loading of network transformers and high-tension and radial loads.
The reinforcement work is then incorporated into the regions’ relief programs
and scheduled to be completed prior to the next load peak that produced the
overloaded condition. Usually this is the next summer. However, in some
cases due to the amount of reinforcement required, projects for load relief may
take years to complete. These projects are started well in advance of need to be
certain the relief is completed when needed.
Feeder and Transformer Ratings
Long Island City network is the largest of Con Edison’s secondary networks
(based on the number of secondary transformers). The estimated peak demand
for the summer of 2006 was 395MW. The network has twenty-two 27kV
feeders. The feeders have an average length of 12.94 miles and supply an
average 54 network transformers. The design of the network is similar to Con
Edison’s other networks which provide an “N-2” second contingency design.
This is the most important aspect of the network design. During second
contingency operation the resultant utilization voltages to consumers fed from
the affected network should be within the limits stipulated in specification EO2065 “Low Tension AC Service Voltage Limits.” The loadings on primary
feeders, transformers and secondary mains should operate within the prescribed
design limits.
Long Island City network feeders in the Electrical Distribution System 2006
manual are all shown with forecasted normal and emergency summer loads
below their normal and emergency ratings. However, the information indicates
that 10 of 22 feeders or 45% of the Long Island City Network feeders will be
operating at or above 90% of their normal rating on a peak day. In addition 18
of 22 or 82% of the Long Island City network feeders will be operating at 90%
or above, of their emergency rating on a peak day during their worse
28
contingencies. This suggests that the feeder cables – although within their
ratings – will be at the high end of the allowable temperature range during
emergency operation. There may be such a narrow margin that the risk of
overload is heightened during contingencies above N-2. This suggests that
relief might be appropriate when looking at the loading of adjacent feeder bands
as well as for individual feeders.
Likewise normal and emergency ratings for transformers in the Long Island
City network are being used which permit loadings that result in narrow
capacity margins when operating above design conditions. Therefore, if there
were pre-existing local conditions such as open secondary mains or transformer
banks off the system, a first or second network contingency could result in local
facilities being highly loaded. In the case of the transformers being at or near
their operating ratings during periods of high ambient temperature with
concurrent high customers demand, and contingencies beyond design, i.e. fourth
or higher contingencies, it can be expected that these events may cause
overloading. If not remediated, this may lead to cascading failures and
infrastructure damage and customer interruption.
Con Edison Specification EO-2002 entitled, Loading Limits for Network
Transformers and Associated Protectors, provides specific design load limits for
network equipment installations. These loading limits are intended to be used
on installations that meet the Con Edison design operating criteria for a standard
natural ventilated vault. There are four basic components associated with
network vault installations and all must be considered in determining the design
loading limit of the complete installation. These components are the:
•
•
•
•
Network Transformer
Network Protector
Network Protector Fuses
Secondary Tie Cables with their associated limiters that connect the
installation into the street secondary grid where other cable
components must also be appropriately rated
Specification EO-2002 provides the designers with the loading limits for both
the transformer and network protector. Their respective limits are generally not
the same for any given operating condition. It is up to the design technician or
engineer to apply the proper rating which is the lower limit of either the
transformer or network protector. Factors which affect the ability of
transformers to carry varying levels of load include:
29
•
•
•
•
•
Design top oil and hot spot limits which are not to be exceeded
Vault condition and design criteria
The shape of the load curve
Modes of operations
Operating ambient temperature
The top oil temperature is a function of the operating ambient temperature, vault
rise, rated top oil rise, and hot spot rise for the peak and cycle for a given load.
The specification clearly cautions the designers that the maximum allowable
temperature limits must never be exceeded for any design condition.
Note: Definition – “Transformer Heating” – A condition that leads to the
degradation of a transformer. Exposure to elevated temperatures, over an
extended time, degrades the transformer winding insulating materials resulting
in the formation of gas bubbles that facilitate the breakdown characteristics of
the transformer dielectric fluid. The winding hot spot temperature is usually the
principal factor limiting the loading of transformers.
These temperature limits are shown in the table below:
Maximum Allowable Hot Spot Temperature Limits
Transformer
Refer to EO-
Refer to EO-
Type
2002 Appendix
2002 Table
B
Liquid Filled
Dry
Modes of Operation
Normal
1st Contingency
2nd Contingency
(degrees C)
(degrees C)
(degrees C)
1, 2
95
115
135
B
3-7
105
125
150
C
1
200
240
285
C
2
180
220
240
Table 1: Maximum Allowable Hot Spot Temperature Limits (From Con Edison
specification)
The table above gives designers other criteria to consider such as vintage and
load cycles before assigning the appropriate loading limit and rating to the
transformer and/or network protector. Most important is the stipulation that top
oil temperature limit for any mode of operation is 1250C and that is not to be
exceeded.
“The design criteria to be applied to a network area include normal and
contingency load for feeders, transformers, and secondary mains. The
responsibility for determining that transformer, secondary main and feeder
30
loads are within the design criteria rests with the Regional Engineering
Department.”
Load Cycles
The introduction of computerized tools such as PVL, described in the PVL,
WOLF, and RMS Section of this report, has led designers to near total reliance
on these tools for analysis and operation. However, the computer generated
analyses rely almost exclusively upon various tables as well as data from the
remote monitoring system (RMS) being utilized in the algorithms and
calculations performed by PVL. These results are then reported to the designers
on an exception basis. This means that the results of the analysis are typically
not printed out for review by the designers unless the PVL study indicates there
is an overload condition. Unless specifically requested, the designer will only
see overloaded results reported. However, reports can be configured to provide
various output options. The results are provided in tabular form and the output
stated in percentage of rating. The designers rely on these tools, accept the
results, and infrequently test the results with other information to verify them.
However, the results are not always correct. The assignment of load cycles is
an example.
The assignment of load cycles which determine the loading limits used in PVL
is done by accessing RMS data directly from the historical RMS database. With
this data for the projected load and load cycle, another database is utilized to
determine what equipment is shown as installed in the field (i.e. class of
transformer or network protector so that a rating can be assigned to the
installation).
The cycle assigned may be either an 8-hour, 12-hour,
combination, or continuous. It is very important that the load cycle be
accurately applied as it determines what percentage loading this installation will
be allowed to be loaded to in the computer simulation of the network. It is also
important in setting the load cycle for secondary cables that will carry this load.
The load cycle determines the length of time that equipment will carry the load
and the period it has to cool. Upon examination of transformers in the
following underground vaults TM 5750, VS 8786, and VS 7981, a clearer
picture of the problem develops.
These underground vaults are located in the vicinity of 62nd Street and 39th
Avenue. Physically and electrically the transformers in those facilities are very
close. In fact, two of the three transformers associated with these facilities are
electrically connected in the same manhole, M 2890 (refer to Figure 15 in the
Event Progression section of this report). The third transformer connects to the
subject manhole via one section of secondary mains. For all intent and
31
purposes, these transformers may supply the same customer load and should
exhibit the same load profile.
The PVL model shows ratings for TM 5750 of 138%, 162%, and 170%, ratings
for, VS 8786 of 125%, 145%, and 170%, and ratings for VS 7981 ratings of
95%, 110%, and 120%. As can be seen, the three units under review were
assigned three different load cycles. As these three transformers supply the
same immediate load pocket, they need to be assigned the same load cycle.
Likewise, the rating that is assigned to the secondary mains and street ties from
each transformer that provides the interconnection between the two should
match the same load cycles.
The street tie cables are also an issue. The cables are four sets of four-500MCM
and have a rating of 567kVA under normal operating conditions. As stated, the
normal rating for two of the three transformers is 125% and 135% on normal for
626kVA or 675kVA respectively. For both cases, the street ties would be
overloaded and do not match the ratings of the transformer. The street ties
would exceed their normal rating once the transformer exceeded 113% of its
normal rating.
Two conclusions come from this example: First, the ratings that are applied
may not be representative of the infrastructure in place and second, the tools
that are used to analyze the infrastructure typically only report overloads and
these mismatched ratings would not be observed.
Conclusions
• The current feeder ratings are such that the feeder cables – although
within their ratings – will be at the high end of the allowable
temperature range during emergency operation. While this provides
assurance that cables will operate within design conditions on a peak
day in a second contingency, it narrows the design margin such that
during multiple contingencies in which the system must operate
beyond its N-2 design. The risk of overloading is heightened
especially if adjacent feeder bands are also highly loaded.
• Normal and emergency ratings for transformers in the Long Island
City network are being used which permit loading during normal,
first, and second contingency situations that result in oil temperatures
nearing the maximum level of their operating value. Therefore, in
multiple contingency events, the transformer begins at a higher
temperature and the risk is heightened of the transformer reaching its
32
oil temperature limit sooner than Con Edison can de-load or cool the
transformer.
• For several cases reviewed in the Long Island City network, the load
cycle was not uniformly applied to transformers and street ties. It is
important in setting the load cycle for equipment and cables that will
carry this load.
Recommendations:
• Conduct a study to determine the expected time to failure in a multicontingency event. The study should determine the time required to
enable remedial actions to prevent /control overloads and failures. If
insufficient time is available, consider and develop appropriate
engineering solutions.
• Evaluate if feeder ratings, estimated loads on associated feeder bands,
and individual feeders should be considered in planning network
reinforcement.
• Investigate and develop criteria for application of “band relief” which
may suggest relief be implemented earlier than only relieving
individual overloads.
• Utilize available breaker positions at the North Queens substation to
establish new feeders to reduce the average normal and emergency
feeder loading and improve diversity on the Long Island City network.
• Review the use of ratings in PVL and establish criteria for when the
use of these ratings should be applied, specifically during operating
and planning conditions.
• Review the transformer ratings that are presently assigned in PVL and
ensure they are appropriate for the actual load cycle for the area
during peak conditions.
• Review the transformer ratings being assigned for normal, first, and
second contingency operation and ensure that the secondary street ties
and mains cables are within their ratings during the corresponding
operation.
• Evaluate load curves currently applied to transformers and confirm the
curves are appropriate. If not, revise them so that the proper curve is
used and the PVL model is updated. Specific attention should be
given to nearby transformers.
33
Summer Preparation
Overview
In the summer of 2005, all Con Edison operating Regions began their annual
process to prepare the Con Edison electric system to meet the reliability and
capacity needs expected during the summer of 2006.
Computer models of the Con Edison system were used by its engineers to
forecast the highest peaks in load that may occur during 2006. The models
identified portions of each network (including the Long Island City network)
that required some type of engineering solution to ensure customer reliability
during 2006. Solutions included:
• Reinforcement – such as replacing a transformer with a higher
capacity transformer or adding a new transformer
• Load Relief – such as transferring load from one feeder that is
expected to be highly loaded to a different feeder that is expected to
be less loaded and has spare capacity
• Reliability Replacement – such as replacing a certain type of cable
(such as Paper Insulated Lead Covered – PILC) or joint (such as
thermally sensitive joints). Certain types of joints associated with the
connection of PILC cable to solid dielectric cable are known to have a
higher failure rate during high temperature and load conditions. They
are replaced with a type that is known to be more reliable
• Repair – such as removing a weak section of cable or joint that may
have been detected by exposing it to high voltage (usually about twotimes normal voltage for fifteen minutes, known as high potential or
“hi-pot” testing)
In 2005, the Central Distribution Engineering Department analyzed the past
performance and other information about the networks and predicted future
network behaviors under normal and contingency operations. Distribution
Engineering made recommendations to the Brooklyn/Queens Region. Their
recommendations outlined which feeders and transformers to relieve or
reinforce as well as which feeders to hi-pot test. The Brooklyn/Queens
Engineering Department is responsible for assuring that the Long Island City
network transformers, secondary mains, and feeder loads operate within design
criteria under forecast summer peak load conditions.
34
In 2005, the Brooklyn/Queens Engineering Department considered the
recommendations for the Long Island City network and other work
commitments such as the PILC replacement program, and the manhole
inspection program. The engineers implemented specific projects to ensure that
the network and its feeders and transformers were prepared to meet the
performance requirements expected in the summer of 2006. The engineers
reported that Con Edison technical specifications were followed on how to:
1) Plan network reinforcements
2) Utilize the Poly Voltage Load-flow (PVL) model to establish a
uniform method for the identification of network primary feeder and
network transformer overloads and development of reinforcement
strategies
3) Avoid exceeding loading limits for network transformers and
associated equipment
4) Archive critical information to be used for other functions such as
future planning and analyses during emergency situations
Network Programs
High Potential Testing (hi-pot)
The primary feeder hi-pot program was developed to detect the weak
components of the system by subjecting a 27kV feeder to a high voltage (60kV
DC) condition for a 15 minute period to attain an acceptable level of leakage
current. The test is used as part of the Con Edison’s summer preparation and in
the restoration process following a feeder “Open-Auto.” A less severe hi-pot
test to solid dielectric cable is one performed with a lower AC test voltage
(50kV for 15 minutes) but the DC voltage hi-pot is predominantly used. The
AC test is still under evaluation by Con Edison.
The summer preparation feeder list is developed from a feeder failure index
created by Con Edison’s Central Distribution Engineering. Long Island City
network had 12 feeders recommended for testing prior to June 1, 2006. Hi-pot
tests were conducted on 8 of the 12 feeders with only 5 feeders successfully
completed. It is not unusual that a number of failures occur before a feeder is
completed. Statistics indicate that feeders that complete a hi-pot or have been
hi-pot tested without completion of a successful test have less “open-auto”
operations during the summer period. However, the Committee found no
correlation between hi-pot testing completion and feeder failures during the
Long Island City network event. This is most likely due to various factors;
feeder overloads were minimal during the event, and the network load was
35
lowered through a variety of methods reducing the strain on feeders. The use of
the hi-pot during feeder restoration was limited during the early stages of the
event and was used more consistently beginning on Wednesday morning after
eleven cut-in open-auto operations occurred. After which a modified (lower
voltage/less duration) hi-pot was utilized and the cut-in open-auto operations
were significantly reduced.
Manhole and Service Box Inspections
The manhole and service box inspection program which is part of the Reliability
Program requires an inspection of each manhole and service box every 5 years.
It has become the driver of secondary reliability projects. Although the
underground inspection is not considered a part of summer preparation, the
secondary mains cables and transformers identified as needing repair and
replacement are part of system preparation. Upon inspection, when a condition
is found that needs remediation, it is assigned one of two repair priorities, either
Tier I or Tier II. Tier I repairs are more serious and are to be repaired when
found. Tier II repairs are of a lower priority and are to be scheduled over a
longer duration as they do not have an immediate impact on secondary
reliability. Brooklyn/Queen’s is responsible for almost 60% (30,879 out of
53,243) of the targeted annual underground structure inspections at Con Edison.
As of November 30, 2006, Brooklyn/Queens has completed 49,802 total
inspections that represent 22,430 unique structures for the period from
December, 2005 through November, 2006. Con Edison has also inspected over
350,000 electrical facilities since the program began in 2005, including
underground and overhead facilities, which met the overall inspection
milestones. Since the beginning of the inspection program in 2005, the
company has replaced 3,062 cable sections in Brooklyn/Queens. This
represents a 61% increase in the number of replaced sections compared to the
1,905 sections replaced over the previous 4 years (between 2001 and 2004) in
Brooklyn/Queens. With respect to the Long Island City network event, the
Committee found that Brooklyn/Queens had inspected M 11711 which was
involved in the initial event and found no defect that required immediate
actions.
Remote Monitoring System
Con Edison utilizes the most extensive remote monitoring system in the
industry on each network transformer to monitor the transformer and secondary
network load flow. The availability of the system is critical especially during
high load summer conditions. The system is required to be at 95% availability
by Con Edison specification. Beginning in 2005, Con Edison qualified a
contractor to augment its current staff in order to improve the availability rate.
36
A more detailed description of the RMS system and performance can be found
in the PVL, WOLF, and RMS section of this report which also includes a
description of the problems associated with RMS being at a lower level of
availability.
Paper Insulated Lead Covered (PILC) Cable and Stop Joint
Replacement Program
In 1999, Con Edison embarked on a multi-year PILC cable and stop joint
replacement program to replace 6,700 Raychem three-way/one-way joints
manufactured before 1994 and replace 2,500 Elastimold two-way/one-way
joints since.
As of May 1, 2006, 3,801 pre-1994 Raychem had been replaced of which 289
were in the Long Island City network. 1,014 Elastimold joints had been
replaced of which 86 were in the Long Island City network. The program to
replace Elastimold joints is expected to be completed in 2008.
Since 1999, the percentage of PILC cable system-wide has been reduced to 27%
of the total cable from 75% of the total in 1984. Over 21,000 cable sections of
PILC have been replaced. The program is on schedule to be complete in 2024.
The PILC replacement program eliminated not only the paper cable, but it
eliminated lead wiped joints. These required more time to make which can
delay restoration of feeders during outages. Con Edison has been aggressive in
its removal of PILC for the Long Island City network which now has one of the
lowest percentages of paper cables in service. Lead wiped joints are replaced
with mechanical splices that can be completed and modified in significantly less
time. This allows faster feeder restoration times which are important to network
reliability.
Network Investments in Preparation for Summer 2006
Various regional engineering departments revised Con Edison’s computer
models of its electric system to accurately reflect changes that have been made
since the prior year. This insures that these models represent the latest
condition. The models are relied upon by a number of engineering and
operating functions to manage the electric system, this is especially critical
during contingency situations. These studies are the foundation upon which the
summer preparation program is built and capital budgets are based.
Con Edison invested $401 million Company-wide in its efforts to ensure
reliability and capacity through reinforcement, relief, replacement, and repair
37
activities during the period January 1, 2006 and June 30, 2006. This included
replacing or adding transformers as well as reinforcement for new business,
burn-outs, meter installations, and replacing primary and secondary cable
sections. In the Long Island City network a total of $14 million was invested as
compared to $119 million invested in all of the Brooklyn/Queens region. All
work that was scheduled for the Long Island City network was completed prior
to the summer peak.
Operations Analysis
As part of its ongoing improvement efforts, the Brooklyn/Queens Operations
Services Section conducted a review of Brooklyn/Queens Engineering in June
of 2005 to determine whether preparation for 2006 reinforcement projects of
primary and secondary systems and the preparation of the 2005 distribution
manual were in compliance with Con Edison specifications CSP 5-3-19, EP2072 and EO-2048.
The review conducted by Brooklyn/Queens Operations Services determined
that:
•
•
•
•
Queens spring 2005 preparation for 2006 secondary reinforcement
projects was based on analyses that were in compliance with CSP 5-3-19
and EO-2072
Queens’ Primary Distribution Manual’s feeder ratings for 2005 were
based on manual calculations in lieu of using PVL as required by Con
Edison specification EO-2072 and EO-2048. The Queens Engineering
Group considered their manual calculations to be more stringent than
those developed with PVL. The calculations did not conform to the
corporate guidelines presented in specifications EO-2072 and EO-2048,
which required compensation for the temperature variable and archiving
the results
Preliminary Queens’ primary reinforcement studies for work to be done
in 2006 including those for the Long Island City network were based on
analyses that were not in full compliance with CSP 5-3-19, EP-2073 and
EO-2048 at the time of the operations analysis
Ten randomly selected engineering layouts for transformers installed in
Brooklyn and Queens were reviewed and some did not conform to
established corporate practices concerning the level of detail to be
provided on the engineering sketch
38
The recommendations from the review were, in part:
•
•
Brooklyn/Queens Engineering update all Brooklyn/Queens PVL model
databases
Brooklyn/Queens Engineering re-evaluates all proposed 2006 primary
reinforcement projects for Queens based on the updated PVL models
and seek concurrence from Distribution Engineering on any departure
from procedures when selecting reinforcement projects
Progress has been made, particularly in the primary feeder model, but some of
the issues identified in the assessment, in particular the PVL model and the
sizing of secondary mains and street ties, require ongoing attention.
Substation Preparation
The substations summer preparation was performed in conjunction with the
regional engineering input for improvement in the networks reliability that can
be driven by substation projects. Projects range from establishing a substation
to installing a new feeder position. Some types of substation projects need
longer lead time to insure availability when relief is required. This is why close
coordination and planning is required by engineering groups. The Summer
Preparation Program is defined in the Central Operations budget under Load
Relief for substations and Load Relief for Transmission. Most projects are
multiple year projects with an average of 46 active substation projects over the
past six years and an average of 15 transmission relief projects over the same
period. Company wide there was seven reliability and two relief projects on the
transmission system that were scheduled and completed as part of the 2006
summer program. The nine substations that were projected to be overloaded in
2006 were being relieved prior to June 1. In the Substation Reliability area
eight preparedness projects were completed. All transformer pumps and fans
were cleaned as scheduled. All substations were Infrared inspected and repairs
made.
The North Queens substation supplies the Long Island City network. It was
projected to be within its capacity for the summer of 2006 (483MW). There
were projects planned at the substation that improved reliability by eliminating
over-duty breakers by replacing stationary breakers with modern higher-rated
rack-out breakers. These breakers replacements are being coordinated with new
ground and test breakers. This is a key improvement that helps resolve the
limitation of the station test bus by permitting multiple feeders to be processed
concurrently. This was a limitation during the Long Island City network event.
39
Refer to the Feeder Processing section of this report for additional information
regarding the test bus and its impact on the Long Island City network event.
Substation Planning
There is a proposed project that calls for the development of a new substation in
the southern area of Long Island City. Its purpose is to relieve the North
Queens substation in the future when it is projected to reach its capacity. The
substation will be established with 3– 93 MVA transformers. It will be supplied
by tapping three existing 138KV transmission feeder lines in the area.
The new substation when installed will relieve the projected future overload of
the North Queens substation, reduce the length of the feeders and reduce the
number of transformers on each feeder. This will be done by transferring
existing feeders from the North Queens substation and connecting them to
feeder supply positions at the new substation. This will enable an improvement
in feeder diversity in the network and simplify the analysis during multiple
contingency operations. There are also three spare breaker positions in North
Queens Substation that are available for establishing new feeders in the shorter
term. Use of these positions would lower the average load per feeder, reduce
the failure exposure of the feeders, and could also improve the feeder diversity
of the network.
Conclusion
• Some of the issues identified in the 2005 Brooklyn/Queens Operations
Services Section analysis of their Engineering Department’s
compliance with CSP 5-3-19, EO-2072, and EO-2048 still remain
unresolved. For example, there are still issues with how elements of
the system are represented in the PVL model. These issues generate
calculation errors and can adversely impact the accuracy of feeder
loading projections used for summer preparations. In addition, a
review of the planned and conducted transformer upgrades revealed
instances where the street ties ratings do not match the transformer
ratings and the shortcomings are consistent with the results of random
sampling conducted in 2005.
Recommendations
• Use the operations analysis of the Brooklyn/Queens Engineering
Department’s compliance with CSP 5-3-19, EO-2072, and EO-2048
as a guide and consider similar analyses of engineering departments in
other operating regions. Re-analyze Brooklyn Queens Engineering to
ensure all recommendations have been implemented.
40
• Share the operations analysis among all regional engineering groups.
Consider following future engineering analyses with field audits or
analyses of the completed work.
41
Event Progression
Note: Throughout this report, all references to the temperatures at which
transformers failed are RT3 calculated temperatures and the percentages
provided for pick-up utilize the PVL model maintained by Con Edison and are
approximate. Detailed load flow calculations were not conducted for each and
every case since exact loads and secondary connectivity were not known.
However, this data and real time data, where available from RMS, and the
transformer forensic analyses, confirms for the Committee the reasonableness
of these values.
Note: RT3 is an application used to assess the temperature effects on an asset,
such as a transformer. While portions of Distribution Engineering depend on
RT3 as an analytical tool to assist in forensic analysis, other portions of
Distribution Engineering did not use RT3 based ratings in their initial post Long
Island City event analysis.
Note: The Committee defines several key terms used throughout this report as
follows:
•
•
•
Highly Loaded – A portion of a conductor or an apparatus operating at
an electrical loading approaching, but not exceeding, its determined
equipment rating in the appropriate Con Edison specification.
Overloaded - A portion of a conductor or an apparatus operating at an
electrical loading exceeding its determined equipment rating in the
appropriate Con Edison specification.
Local Load Pocket – An area including the transformers and associated
secondary mains and services, within the boundaries of a larger
secondary network, which jointly supply a set of connected loads. This
subset of the network can typically range from one to four city blocks in
size.
Pre-Event Preparation
Since Con Edison anticipated a system peak in the coming week greater than
10,500 MW, it declared an Extreme Weather Criteria on July 13. That
declaration triggered personnel to prepare the system to perform at its maximum
capacity. This declaration required that all regions stop scheduled maintenance
work and return all available system components to service. Further, it required
that staffing levels be increased in critical areas and preparations made to
mobilize special emergency management groups to monitor the system and
42
provide quick response to any problems which may arise. Key groups included
System Operations, Substation Operations, Distribution Engineering Command
Post, Brooklyn/Queens Region Emergency Management Center, Customer
Service and all the comparable Regional departments and during large-scale
events, the Corporate Emergency Response Center (CERC).
Key Organizations
The Distribution Engineering Command Post was mobilized on July 16th to
provide (among other duties) engineering support to all of the regional
engineering departments including the Brooklyn/Queens Region which handles
the Long Island City network.
At the Call Center, staffing levels were available to meet needs that arose from
heat-related service interruptions. Call Center practices provide for assigning
all staff to handle only the service-related calls as the need arises.
The Call Center utilizes an automatic Voice Response Unit (VRU) system to
provide information to callers and route calls efficiently. Call Center personnel
were preparing to use messages that had been used previously to inform
customers who might experience service problems in geographic areas. The
message also requested that customers who might be calling for routine inquires
such as billing questions to please call back another day so that attention could
be given to outage related calls. These are standard messages used throughout
the industry. Also, anticipating that some customers might notice a reduction in
voltage due to an 8% voltage reduction on the Long Island City network, the
call center had prepared a special message informing callers that voltage had
been reduced in their area and asking callers to please cut back usage on nonessential appliances.
The Brooklyn/Queens Emergency Management Center was mobilized on July
17th to centralize critical information and thereby improve management
decision-making for Con Edison’s response to trouble throughout the
Brooklyn/Queens region including the Long Island City network. Critical
information to be evaluated to determine the extent of an event included among
others, operating parameters such as load, physical parameters such as
equipment temperatures, customer reports of trouble such as loss of power, and
fire department reports of emergencies such as manhole fires.
The
Brooklyn/Queens Emergency Management Center was the headquarters of
Incident Command and managed the Long Island City network event for the
first days until the Corporate Emergency Response Center (CERC) was opened
on Thursday July 20th.
43
Readiness
The plan to manage the event was straightforward and consistent with Con
Edison’s past practices. For years Con Edison successfully managed through
these types of extreme weather conditions. As the heat of the day increased,
demand for electricity would increase and there would be an increase in the
amount of electrical load flowing through the network. The increased load flow
would produce thermal stresses. Brooklyn/Queens managers would be prepared
to respond to feeder failures and to reduce load through a number of strategies.
This would be implemented if load began to exceed emergency limits thereby
preventing damage and consequent power outages to customers. Con Edison
would know when those limits were threatened or if damage had occurred by
analyzing critical information such as load, temperature, customer outage
reports and reports of manhole fires or smokers. That information would be
provided through a series of information systems.
Based on past experience with network contingency operations, Con Edison
anticipated that if problems were to occur, there would be low volumes of
customer outages and manhole events at least in the early stages. Therefore, in
the early stages when volume would be low, it would be more expedient to:
•
•
Reroute customer outage information from the central trouble analysis
group directly to field supervision in the Brooklyn/Queens control room
to enable quick dispatch of first responders and repair crews
Send reports of emergencies such as manhole fires directly to the
operations center emergency desk for action which is the Con Edison
practice during normal conditions
These actions were intended to provide the highest level of emergency
responsiveness as long as the volume of trouble tickets was low. If and when
the volumes increased, Incident Command was prepared to change the process
and send the outage and emergency information to a central trouble analysis
group to organize the work and to provide engineering analysis and assessments
needed to ensure a technically optimal response to the event.
Geographic Areas
In this report, the Committee refers to certain geographic areas of Long Island
City network. The areas are shown on Figure 3.
To better understand the progression of the event, the Committee has segmented
the Long Island City network into three areas. These three areas are defined as:
44
Area I is where the first feeder fails and the Committee has determined a pocket
of high load existed at the start of the event. Area I is bounded to the north by
Grand Central Parkway Extension and Tri-Borough Plaza. It is bounded to the
south by Northern Boulevard. Area I is bounded to the east by the BrooklynQueens Expressway and to the west by 29th Street.
Area II is the next separate pocket of high load to develop early in the event. It
is bounded to the north by Northern Boulevard. It is bounded to the east by
Brooklyn-Queens Expressway. It is bounded to the west by 39th Street. The
southern boundary of the area, moving from east to west, follows Queens
Boulevard from the underpass of the Brooklyn-Queens Expressway to 48th
Street and the south on 51st Street until the boundary moves west again along
the Queens Midtown Expressway until it meets 39th Street.
Area III is a third separate area of high loads that experiences limited supply
during various contingencies. It is bounded to the south by the Grand Central
Parkway Extension and Tri-Borough Plaza. It extends to the outer fringes of the
network in the other three directions. This is represented 20th Avenue and
Berrian Boulevard in the north, 82nd Street to the east and 21st Street to the west.
Many of these area boundaries, in particular the boundaries to the east, represent
physical boundaries of the Long Island City network. The boundaries between
the areas follow physical boundaries that exist, such as expressways or other
structures. The boundaries limit the number of connections and cable runs
between the defined areas. Examples of this are Astoria Boulevard and
Northern Boulevard. The secondary cables that crossed these boundaries
became highly loaded as the those cables were the only links between highly
interconnected areas with shortages of supply, and as the event progressed and
supply was further reduced, the Committee determined that many of the these
secondary cables overloaded and started to form a natural separation between
the defined areas. This assessment is supported by the plotting of the no light
jobs, manhole events and wire burning jobs from the event.
45
Figure 3: Three areas of interest, Area I / Area II / Area III
Pre-Event Transformer Failures
In the Long Island City network, 25 of the network’s 1194 transformers were
out of service prior to the start of the event (refer to Attachment B). This
number is not considered unusual by Con Edison or by the Committee.
Generally this number of transformers out of service from the 1194 transformers
supplying the Long Island City network would not present a problem if the out
of service transformers are fairly well dispersed throughout the network and a
thorough analysis of these transformers confirms that other transformers and
mains can pick-up the load without becoming overloaded.
46
VS 5447
Transformer Location
Date and Time of Failure
th
Area
VS 5447
June 29
I
Connected Primary Feeder
Street Location
M&S Plate
1Q19
25th Avenue and 42nd Street
68AF
The failure of the transformer in VS 5447 on June 29th increased load on nearby
transformers and secondary mains. Refer to the portion of Con Edison’s M&S
plate shown on Figure 4.
When VS 5447 failed, 12% of its load was picked-up by the nearby transformer
in V 7813 and 27% was picked-up by the nearby transformer in TM 1007 in the
immediate vicinity. This contributed to the progression that caused the
transformer in V 7813 to fail due to overloading on the first day of the Long
Island City network event – 21:56 on July 17th. That failure is described later in
this document.
V 9426
Transformer Location
Date and Time of Failure
Area
V 9426
July 11th
I
Connected Primary Feeder
Street Location
1Q16
th
M&S Plate
rd
30 Avenue and 43 Street
67AE
This transformer failed on July 11th. It is estimated that the load on the
transformer in V 9426 was picked-up by V 7813 (19%) and by V 7914 (39%)
with others seeing a slight increase in loading. The loss of this transformer
caused highly loaded secondary mains in the vicinity. The area of highly loaded
secondary mains created by the June 29th loss of VS 5447 was expanded when
V 9426 failed. Refer to the portion of Con Edison’s M&S plate shown on
Figure 4.
47
Figure 4: Secondary Mains / VS 5274 (1Q23) / V 9426 (1Q17) / VS 5447 (1Q19) / TM 6008
(1Q21) / V 7813 (1Q20)
48
Sunday, July 16th
On this day, the high ambient temperature continued and was predicted to
increase in the coming week. Transformer TM 1007 located at 25th Avenue and
Steinway Street was already in a contingency condition and according to RMS
data, was loaded to 150% of its normal rating during the peak on July 16th. This
suggests that the secondary mains supporting the load from this transformer
were already highly loaded and being thermally stressed.
July 17th Secondary Cable Insulation Fire
Prior to the start of the Long Island City network event, as a result of
transformers off the system, secondary mains were above rating in the area of
44th Street and 30th Avenue in Long Island City. Refer to the portion of Con
Edison’s M&S plate shown on Figure 4.
The Long Island City network event was initiated when insulation on a
secondary cable which was overloaded, began to burn in the location between
service boxes 1345 and 30112 located at 44th Street and 30th Avenue. The exact
cause of the fire remains uncertain as the level of loading should not have been
enough to cause the cable insulation to burn. The burning of this secondary
cable insulation communicated to the primary system causing external damage
to the cables on feeders 1Q17 and 1Q16 in the conduit run and adjoining
manhole M 11711.
The secondary grid was further weakened when secondary cables in adjoining
structures had to be cut to isolate the burning secondary cables. At about the
same time a manhole fire and overhead wires burning were reported one block
south of this location. These fires also required secondary cables to be cut to
extinguish the source of energy supplying the short circuit current and contain
the fire and cable failures. This cutting of additional secondary mains further
reduced the support for the 44th Street and 30th Avenue load pocket from the
south.
Fire Damage Caused 1Q17 to De-energize
At 15:50 on July 17th, fire from the burning secondary mains and cables at 30th
Avenue and 44th Street (Area I) damaged primary feeder 1Q17 sufficiently to
result in the loss of insulating materials, and created a fault by arcing to ground.
Feeder 1Q17 de-energized automatically at North Queens Substation as a result
of this event, causing the boundary of highly loaded secondary mains to expand.
The loss of 1Q17 caused additional transformers in the network to de-energize
49
and further deteriorate conditions in the vicinity of 30th Avenue and 44th Street
(Area I). Refer to Figure 4.
Later, that evening, when 1Q17 was restored, two additional transformers were
disconnected from 1Q17 (through live-end capping). Crews were still clearing
secondary mains as a result of the burned secondary mains cables that initiated
the event. The crews were in the process of installing secondary shunts to
restore secondary main support and restore customer outages in and around the
immediate area. The live-end capping was done to expedite the return of 1Q17
to service, since 1Q16 and secondary cables were damaged in the same duct run
as 1Q17. Feeder 1Q16 failed 32 minutes after 1Q17 and is discussed in the
16:22 July 17, 2006 event below. Concurrent repairs to both feeders and the
secondary cable were not possible. Extensive repairs including the construction
of new underground conduits were needed to completely restore the secondary
cables that had been damaged and remove the temporary shunts that had been
installed to restore service to customers in the area of the burnouts.
The manner in which 1Q17 was restored to service was beneficial to the
network as it relieved a high contingency in the network. However, the live-end
capping of the two transformers to restore feeder 1Q17 provided little benefit to
the local network load pocket in the vicinity of 30th Avenue and 44th Street (in
Area I). Even with 1Q17 back in service the load pocket loading was higher
due to the following:
•
•
By live-end capping the transformers, first on 1Q17 then on 1Q16, load
would have to be supplied from transformers located further away,
thereby increasing the load on the secondary cable mains in the area;
and,
Due to the necessity of disconnecting the secondary cable, to extinguish
the manhole fires, the number of secondary cables and pathways were
reduced. This created customer outages and reduced the number of
pathways that electricity could flow to serve customers and increased the
load each remaining pathway was required to carry.
Fire Damage Caused 1Q16 to De-energize
The failure of primary feeder 1Q16 within 32 minutes of the 1Q17 failure in the
same location and was a result of damage by the same fire that caused 1Q17 to
fail. While the 1Q16 failure created a broader and more highly loaded pocket in
Area I, it also created a second contingency in the Long Island City network. It
is notable that the Con Edison system is designed to withstand any second
contingency including this one.
50
In accordance with prior instructions to system operations, Incident Command
asked for an 8% reduction of the operating voltage in the Long Island City
network at 18:54. This voltage reduction was effective in reducing load in the
network. The Committee believes this action avoided damage to the network
and had relatively little impact on the customers supplied by the network.
Later, at 23:33 on July 17th an attempt to restore feeder 1Q16 failed as the
circuit breaker at North Queens Substation opened automatically. This failed
attempt (referred to as a cut-in open-auto) was one of thirteen to occur during
the Long Island City network event that involved a number of different feeders.
The mechanisms for these failed attempts to restore feeders are described in
detail later in the report in the section titled Feeder Processing during the Long
Island City Network Event.
When 1Q16 was returned to service at 09:44 on July 18th, Con Edison crews
had live-end capped three additional transformers in Area I. The combination
of prior transformers out of service, the two transformers that were live-end
capped from 1Q17 and these 3 additional transformers exacerbated the
problems described above. When 1Q16 is returned to service, crews had also
live-end capped two transformers in Area II. The lack of those two
transformers started a second case of high secondary mains loading but now in
Area II.
As a result of the transformers that had been out of service before July 17th and
the transformers live-end capped to expedite restoration of feeders 1Q17 and
1Q16, a total of 7 transformers supplying secondary mains in these locations
had been de-energized. This resulted in highly loaded local pockets in Areas I
and II being supplied by transformers that were physically distant from those
pockets, increasing the load and therefore stresses on the transformers and the
secondary mains.
A Cable Failure and a Faulty Circuit Breaker caused 1Q21,
1Q07, 1Q15, and 1Q81 to De-energize
On July 17th at 18:47, while restoration efforts were still in progress on feeder
1Q17 and 1Q16, feeder 1Q21 experienced a failure of a cable connecting the
transformer in TM 0804 to the feeder. This feeder was supplied from Bus
Section 3S at North Queens substation. The circuit breaker on 1Q21 did not
operate as designed. Subsequently, in order to remove the primary fault, the bus
breaker opened and de-energized the entire Bus Section 3S of the North Queens
substation. This was a proper operation upon the failure of the 1Q21 breaker to
51
operate. This de-energized four additional primary feeders 1Q07, 1Q15, 1Q81,
and 1Q21.
Feeder 1Q15 was rapidly restored within minutes. Feeder 1Q81, a non-network
feeder which supplies load to the NRG Gas Turbines, was not part of this
investigation as it is not an active feeder in the Long Island City network. A
description of the operation of the circuit breaker connecting 1Q21 and Bus
Section 3S can be found in the North Queens substation section of this report.
When feeders 1Q07 and 1Q21 go out of service, combined with feeders 1Q16
and 1Q17 being out of service for an extended period, new problem areas with
limited local transformer supply were formed. As a result of these four feeders
being out, several small areas began to develop local problems of concentrated
load with critical transformers out of service and higher power demands being
picked-up by nearby transformers.
When feeder 1Q21 was lost, the area of high load that the Committee has
defined as Area I which by now had lost two sources of supply (1Q17 and
1Q16) also lost a third source of supply, feeder 1Q21. Feeder 1Q15 was also
involved in the area but was restored quickly and therefore had little impact of
the Long Island City network event. Loss of the three feeders placed additional
loading on nearby transformers including the transformer in V 7813 which, as
mentioned previously (refer the previous discussion of June 29, 2006), had
already picked up additional load as a result of the June 29th failures of VS 5447
and VS 9426 on July 11th. Further, with each feeder failure described
previously (1Q17 and 1Q16) the loading on V 7813 had increased. Within
hours, at 21:42 on July 17th, the transformer in V 7813 failed due to overload
(refer to Figure 4). This will be discussed in more detail later.
At 19:09 Con Edison attempted to cut-in and re-energize 1Q07 but it
immediately opened-auto and de-energized. This feeder, which was in service,
had previously been de-energized when the Bus Section 3S tripped. This new
operation regarding the immediate open-auto is referred to as “cut-in open-auto”
at Con Edison and is discussed in the Feeder Processing section of this report.
It does not appear that an operating failure caused it to de-energize. The
Committee believes that an in-rush current condition, with a magnitude greater
than the relay setting was responsible for the cut-in open-auto and subsequent
delays to the restoration of 1Q07.
52
Transformers and Secondary Mains Progression
V 7813 – The First Transformer Failed Due to Overload and
De-energized 1Q20
Structure Location
Date and Time of Failure
th
Area
V 7813
July 17 at 21:42
I
Connected Primary Feeder
Street Location
M&S Plate
1Q20
28th Avenue and 42nd Street
68AE
The transformer in V 7813 was the first transformer to fail during the Long
Island City network event. Its failure was diagnosed to be due to overload after
having picked up load from a number of nearby transformers that were out of
service. Its top oil temperature exceeded the design limits reaching 1400C
versus a design of 1250C. The transformer winding hot spot reached 1600C
versus a design of 1350C.
As mentioned earlier, the transformers in VS 5447 and VS 9426 failed on June
29th and July 11th respectively. Both were electrically close to the transformer
in V 7813. Refer to the portion of Con Edison’s M&S plate shown on Figure 4.
The transformer in VS 5447, a 500 kVA transformer in the vicinity of 25th
Avenue and 42nd Street (Area I) was supplied by feeder 1Q19. The transformer
in V 9426, a 1000 kVA transformer located north of 30th Avenue on the west
side of 43rd Street (Area I) was supplied by feeder 1Q16.
With the two transformers off the system, portions of their load were picked-up
through the secondary mains by V 7813. This additional load, combined with
the increased demand due to the hot weather, meant that V 7813 entered the
Long Island City event operating within design but at a higher than normal
design load and elevated temperature.
Compounding the situation was the fact that another nearby transformer, TM
6008, served by 1Q21 and located at 28th Avenue and 44th Street (Area I),
appeared highly loaded before July 17th based on a Poly Voltage Load flow
(PVL) analysis conducted by this Committee. Actual field data from the
Remote Monitoring System (RMS) was not available for TM 6008 since its
transmitter was not functioning to confirm the calculated values. Based upon
the PVL study, TM 6008 was in a highly loaded condition when the
transformers in structures V 5447 and V 9426 failed on June 29th and July 11th
respectively. The estimated load for this operating condition was projected to
be 121% versus a rating of 125%.
53
The North Queens Substation Bus Section 3S tripped out approximately three
hours after feeder 1Q16 failed. This de-energized feeders 1Q15, 1Q07 and
1Q21 and further increased V 7813’s loading. It exceeded 150% of nameplate
for a period of approximately three hours which drove its hot spot and top oil
temperature above its design limits. Feeder 1Q21 supplied the transformers
closest to V 7813. V 7813 was already supporting the area load due to the
nearby feeder band that was de-energized (1Q16 and 1Q17). The additional
load V 7813 picked-up when 1Q21 and 1Q15 de-energized placed the
transformer in a critical state and it failed and caused 1Q20 to de-energize
approximately three hours after the Bus Section 3S tripped.
Operating at this load during this multi-contingency period exceeded the ratings
assigned to the transformer in V 7813 in Con Edison’s PVL model for operating
at a second contingency. The PVL ratings for V 7813 were 102% of nameplate
loading for normal operation, 124% of nameplate for first contingency and
143% of nameplate for second contingencies. At the time of failure the Long
Island City network and the transformer were operating in a 5th contingency,
well above design. Its failure caused a 6th contingency and created a condition
of having the outage of two adjacent feeder bands. This resulted in four
adjacent feeders out of service. These were 1Q16, 1Q17, 1Q20, and 1Q21.
This condition has a higher probability of cascading failures and the network
sustaining damage to the primary feeders, network transformers, and secondary
cables.
Two adjacent bands in Area I and in Area II were out of service, compounded
by a de-energized transformer in structure V 5447 on 1Q19. This created a high
load local condition west of 30th Street (1Q21, 1Q20) and heavily loaded the
transformer in structure TM 5810 (fed by 1Q06 at 21st Avenue and 27th Street)
when 1Q20 de-energized, causing the B-phase fuse to operate based upon RMS
data and increased the loading on the secondary mains extending into Area III.
A post-event simulation conducted by Con Edison Distribution Engineering
staff predicted possible instances of secondary overloads in M&S plate 68AE.
Multiple sections of secondary cable were also damaged and replaced as part of
the restoration effort.
54
VS 477 – The Second Transformer Failed and De-energized
1Q01
Structure Location
Date and Time of Failure
th
VS 477
July 17 at 21:49
Connected Primary Feeder
Street Location
1Q01
st
Area
II
M&S Plate
th
at 41 Avenue and 67 Street
60AH
The transformer in VS 477 was located on the fringe of the network in Area II.
Refer to the portion of Con Edison’s M&S plate shown on Figure 5. Feeders
1Q16 and 1Q17 had provided major support for this area. With those feeders
de-energized, the transformer in VS 477 began to pick up significant load as the
evening peak approached. The top oil and hot spot oil temperature of the
transformer exceeded design, and the transformer over-heated and sustained
failure to all three primary coils.
Based on a forensic analysis of the transformer, corrosion was observed but the
transformer passed a pressure test indicating the tank integrity was intact. The
tank was bulged and the oil had reached a maximum temperature of 160 degrees
C. Its average loading was at approximately 175% of nameplate prior to failure.
For planning and engineering purposes, VS 477 is rated in the PVL model to
operate at 102% of nameplate for normal circumstances, 124% of nameplate
loading for situations when a single feeder is de-energized and 143% of
nameplate when two or more feeders are de-energized. The transformer failed
when the network was in a fifth contingency. The operating temperatures also
exceeded the design limits for the transformer. According to the RT3
temperatures, the winding hot spot reached 164 degrees C (designed to 135
degrees C) and the top oil reached 141 degrees C (designed to 125 degrees C).
55
Figure 5: VS 477 (1Q01)
When feeder 1Q01 de-energized, it resulted in a 6th contingency with 6 of 22
feeders being out of service simultaneously. This created several instances of
two adjacent groups of 2 feeders being out of service in Area III. This grouping
of feeders being out of service also created critical feeder issues north of the
Triborough bridge plaza in the network. The northern area (Area III) was very
dependent on the support of secondary load transfer. As such, it drew heavily
on the secondary mains as it tried to pull supply from more distant transformers
and feeders to satisfy local customer loads.
With the failure of 1Q01, and the configuration of feeder banks in Area III of
the network, 6 feeders out of 8 adjacent feeders were de-energized. This
condition was worsened because the transformers in VS 8640 (supplied by
1Q15) and VS 8283 (supplied by 1Q02) had become de-energized prior to the
start of the Long Island City event. Feeder 1Q02 became the only source of
56
supply in the vicinity of VS 477 and after 1Q02 failed at 08:24 on July 18th, the
area around VS 477 lost five direct sources of supply.
As an example of the severity of the lack of supply to the area, RMS data of
transformers supplying Area III revealed that the transformer in VS 7365 was
loaded at 229% of nameplate, and VS 373 was loaded at 221% of nameplate.
The nearby transformer in structure TM 6398 was not reporting via the RMS
system. However, that transformer and the structure tied into manhole M 820
experienced a secondary mains related fire that ultimately caused feeder 1Q02
to open automatically on July 18th at 08:24. Feeder 1Q01, co-located in
manhole M 820, was also damaged by the fire and experienced a joint failure on
July 19th at 11:39.
A post-event simulation conducted by Con Edison Distribution Engineering
staff predicted possible instances of secondary overloads in M&S plate 60AH
and several sections of secondary cable that were damaged were replaced as
part of the restoration effort. To expedite feeder restoration, Con Edison liveend capped the feeder supplying the damaged transformer in vault VS 477.
During this time, Con Edison crews restored 1Q17 after live-end capping two
transformers on that feeder to expedite the feeder’s restoration. The Committee
believes it is highly likely that the prompt restoration of 1Q17 prevented the
transformers north of Roosevelt Avenue on 69th street (Area II) from failing.
Later in the Long Island City network event, upon the loss of 1Q01, with 1Q16
out, VS 7981 became highly loaded. It failed on July 19th at 21:34 due to
overheating.
In the vicinity of Review Avenue, another problem was encountered on the
fringe of the network. Three feeders were de-energized and formed a growing
area of high load. The network fringe in this area, based on RMS data, was
typically a lightly loaded area. It was not experiencing overloads with only
feeder 1Q02 in service. The bushing failure on the transformer in TM 6531
(located on Review Avenue) was found on the 20th of July (refer to Figure 6).
RMS availability in the general area was problematic and could have
contributed to situational uncertainty.
57
Figure 6: TM 6531 (1Q16)
VS 7995 – Transformer Failed Due to Corrosion and Deenergized 1Q17
Structure Location
Date and Time of Failure
Area
VS 7995
July 18th at 11:54 (This is the time the feeder de-
II
energized. The transformer caused a CIOA at 20:53)
Connected Primary Feeder
1Q17
Street Location
M&S Plate
rd
43 Street and Skillman Avenue
62AC
Prior to the CIOA at 20:53, eight of 22 feeders were out in the Long Island City
network and the vicinity of 43rd Street and Skillman Avenue in Area II had been
adversely affected. The transformer in VS 7995 failed as a result of all three
phases inside the transformer arcing to ground. Refer to the portion of Con
Edison’s M&S plate shown in Figure 7.
58
Figure 7: VS 7995 (1Q17)
At the time of the failure, the transformer was not reporting RMS data. RMS
data was not available for several months prior to the failure of the transformer.
However, a transformer forensic analysis revealed that the transformer had
visible signs of heavy corrosion at the top and pitting corrosion on one side,
toward the bottom of the tank. Dielectric fluid was “sweating” through the tank
wall which had become porous as a result of corrosion. The loss of oil exposed
the high voltage ground switch and tap changer, resulting in all three phases
arcing to ground. This caused the internal failure of the transformer and the
resultant automatic opening of feeder 1Q17.
The “sweating” of the fluid is characteristic of a high load condition creating
pressure in the transformer that forced the fluid through the corroded portion of
the tank.
59
Based upon the mode of failure of this transformer, the Committee reviewed the
inspection and maintenance history of this unit. The GE transformer was
manufactured in 1961 and maintenance and inspection frequency is governed by
Con Edison Specification EO-10110 Rev. 12, dated October, 2000. Titled,
“Inspection and Maintenance of Network Type Distribution Equipment and
Bulletin Number 55 dated April 13th, 2005. Its inspection cycle is governed by
two characteristics: 1) Over 25 years of age; and, 2) Visible corrosion. The
inspection cycle for non-reconditioned (original manufacturers age) for
transformers equipped with RMS is once every five years. This inspection
cycle is to be modified as follows if heavy corrosion is found:
“a. If heavy corrosion, leaking oil or other conditions which may result
in imminent failure are noticed, arrangements should be made
immediately to de-energize the bank and replace it
b. If transformer does not hold pressure, every effort shall be made to
determine the origin of the leak. If unable to determine the leak with
nitrogen, helium shall be used
c. If the pressure leak is in combination with porosity of any tank surface
or heavy corrosion, the unit is to be removed from the system
Note 2: Transformers exhibiting heavy corrosion are to be pressure
tested every 3 months. These units are to be categorized as candidates
for replacement”
Inspection reports from November 20, 1996, October 17, 2002, and October 29,
2002, all indicate the reason for the inspection is routine over 30 years. Based
on the requirements in EO 10110 and Bulletin 55, and the fact that the
transformer in VS 7995 passed a pressure test on October 29th, 2002, the next
routine inspection for this transformer is October 29th, 2007.
On the October 17, 2002 inspection report, the first sign of corrosion is
reported. Fluid level as found and as left, does not indicate any leaks from the
pressure test and the unit is left with one pound pressure. The corrosion is
reported as “heavy” on the top of the transformer.
The next inspection is done on October 27, 2002 at which time “defective
magnetic gauge oil at maximum 7-1-93” is noted and the corrosion
classification is noted as at the top of the transformer: moderate. The record
60
indicates the unit was pressure tested and pressure as found and left was 2
pounds. These results are notable because of the impact on the follow-up
inspections. It is assumed that the inspection performed on October 29, 2002
was actually a follow-up inspection due to the heavy corrosion on the top of the
transformer reported on the October 12, 2002 inspection and followed the intent
of EO-10110 which calls for a pressure test every 3 months for units exhibiting
heavy corrosion. The October 29, 2002 inspection changed the classification to
moderate which would mean the unit did not require special inspection and
would revert to its routine 5 year inspection cycle as per Bulletin 55 dated April
13, 2005.
When the unit failed and was inspected at the Astoria shop, the inspection report
indicates, “Main cover heavily rusted and corroded, extreme amount of
corrosion and rust on main tank and bottom (primary side), oil sweating thru
tank wall.” During the Long Island City contingencies, with the high loads and
high oil temperature, the pressure inside the tank would have accelerated the
loss of oil which caused the internal failure of the transformer and tripping out
the feeder at the North Queens Substation.
The engineering ratings assigned for use in PVL and other engineering purposes
for the transformer allowed operation to 95% of nameplate load under normal
circumstances, 125% of nameplate load under first contingency and 140% of
load under second contingency operation. Since the RMS data was not
available for this transformer prior to its failure, real time data could not
determine if these loadings were exceeded.
The loss of 1Q16 and 1Q17 concurrently greatly worsened the local situation in
several parts of the network as those feeders are companions in the same band.
Con Edison post event simulation predicted possible secondary cable overloads
along Skillman Avenue and at the intersection of 43rd Street and Skillman
specifically. Secondary cable was replaced at the intersection of 46th Street and
Skillman Avenue as part of the restoration effort.
61
VS 9819 -- The Third Transformer Failed and De-energized
1Q18
Structure Location
Date and Time of Failure
th
Area
VS 9819
July 18 at 15:14
I
Connected Primary Feeder
Street Location
M&S Plate
1Q18
Berkly Road and the Brooklyn Queens
67AG
Expressway
Feeder 1Q17 which had been restored on the 17th of July at 23:10 opened
automatically again on the 18th at 11:54. This failure put additional loading on
the transformers connected to feeders 1Q18 and 1Q19. This contingency placed
the network in a 6th and was compounded by the outages of feeders 1Q20 and
1Q21. This again created pockets of limited transformer supply in several areas
of the Long Island City network, and on M&S plate 67AG and the surrounding
area in particular. Refer to the portion of Con Edison’s M&S plate shown in
Figure 8.
Figure 8: VS 9819 (1Q18)
62
The 500 kVA transformer in VS 9819 was subjected to very high loads. It was
rated at 124%, 146%, and 165% of its nameplate respectively for normal, first
and second contingency operation.
The transformer failed due to the overheating of its internal coils, attributed to
over-heating when its winding hot spot temperature exceeded the design of
1500C and the top oil temperature reached the design value of 1250C. The
rating was 165% of nameplate. It appears that the transformer did not reach
165% loading prior to it failing and in fact the hot spot oil temperature exceeded
the normal limit well before the unit carried 124% of its nameplate.
Based on information in the transformer forensic analysis, one of the network
protector fuses opened and two started to melt prior to the transformer failing.
Before it failed, its average loading was at over 155% of its nameplate rating for
6 hours according to RMS data.
The assigned ratings for this transformer as indicated above were to operate at
up to 124% of nameplate load in normal situations, 146% of nameplate load for
single de-energized feeder scenarios and at 165% of nameplate for situations
involving the de-energizing of two feeders. According to RT3 temperature
calculations, the winding hotspot reached 1500C and the top oil reached 1280C.
This transformer failed before reaching the limit established in the assigned
ratings. The Committee understands that the transformer and the Long Island
City network were operating well above its design of second contingency.
However, it believes that the load ratings used for this transformer should be
reviewed. This is discussed in more detail in the Ratings and Load Cycle
Section of this report.
The de-energizing of feeder 1Q18 increased the number of simultaneous feeder
outages on the network to six. The Con Edison Distribution Engineering
simulation conducted after the event indicated secondary overloads in the
vicinity of 50th Street and 28 Avenue (Area II). The secondary mains
overloaded in MH 21110 during the event. During the restoration process,
secondary cable was replaced in various spots in the surrounding area.
Following the de-energizing of 1Q18 at 15:16, an area of high load extended
from the Brooklyn Queens Expressway as an eastern boundary, extending south
to 30th Avenue in Area I. The situation is compounded in the area when
secondary mains were cut in the restoration process in line manhole M 235785.
The feeder restoration work was performed under a Section 9.
63
TM 838 – Transformer Failed Due to High Heat and Deenergized 1Q12
Structure Location
Date and Time of Failure
th
Area
TM 838
July 18 at 20:32
I
Connected Primary Feeder
Street Location
M&S Plate
1Q12
30th Street and Broadway
67AB
At this time, the network was operating in a 7th contingency and the RMS
reporting rates for transformers in the area was low. Although the loading at the
time of failure is not recorded, the forensic analysis report concluded that it
failed due to overheating after picking up additional load from the near-by
transformers that were out of service at the time. Refer to the portion of Con
Edison’s M&S plate shown in Figure 9.
Figure 9: TM 838 (1Q12)
The transformer was installed in 1985. The transformer forensic analysis
indicates that the transformer tank was bulged, ruptured and rusted. The ESNA
elbows and bushings were damaged. The temperature gauge could not be read
due to fire damage. Since it was not reporting in RMS, temperature data was
not available from the RT3. The transformer ratings assigned by PVL for
engineering analysis allowed the unit to operate at 125% of nameplate load for
64
normal operation, 145% of nameplate load for first contingency and at 170% of
nameplate for second contingency.
The de-energizing of feeder 1Q12, due to the transformer failure, increased the
number of simultaneous feeder outages on the network to eight. A Distribution
Engineering post-event simulation projected a secondary mains overload in the
vicinity of Broadway and 30th Street in Area II and mains were replaced as part
of the event restoration effort.
Feeder 1Q13 had de-energized approximately 28 minutes prior to the tripping of
feeder 1Q12. Feeders 1Q12 and 1Q13 comprise a feeder band in the network
and, as such, the transformers on one feeder would pick up a substantial amount
of load from the transformers on the adjacent feeder when it de-energizes. RMS
data revealed that the nearby transformers supplied by 1Q13, prior to 1Q13
opening automatically at 20:04, were very heavily loaded. The transformers
along Broadway from 31st Street to Crescent are very highly loaded during the
feeder events in this general time frame.
The de-energizing of 1Q13 at 20:04 created a localized area missing three of its
primary sources of supply in the vicinity of Northern Boulevard and 62nd Street
along with the other feeders that are out at the time. The de-energizing of 1Q13
is the cause of the failure of the transformer in TM 838 on 1Q12. It should be
noted that feeder 1Q13 was restored to service without finding a fault in an hour
and 16 minutes. However, TM 838 had been carrying high loads for some time
prior to 1Q13 tripping. With the additional load being picked-up by this
transformer, it failed only 28 minutes later. When feeder 1Q12 de-energizes, it
creates an area missing two sources of supply along Vernon Boulevard and is in
jeopardy as two pairs of adjacent bands of feeders are out of service (1Q12,
1Q13, 1Q20, 1Q21).
65
VS 339 – Transformer Failed and De-energized 1Q15
Structure Location
Date and Time of Failure
th
Area
VS 339
July 18 at 20:33
II
Connected Primary Feeder
Street Location
M&S Plate
1Q15
40th Street, north of 47th Avenue
60AB
With more than one third of the feeders out of service in the network, several
areas in the network had very limited primary supply. There were heavily
loaded transformers that compensated for de-energized nearby transformers.
One such transformer was in vault VS 339. Refer to the portion of Con
Edison’s M&S plate shown in Figure 10.
Figure 10: VS 339 (1Q15) / VS 7995 (1Q17) / VS 624 (1Q19)
66
According to the forensic analysis, the transformer failed as a result of internal
pressure building up in the tank from over heating. This pressure caused the
area near two spot welds mid-height on the radiator panel to fail, allowed a
substantial loss of dielectric fluid, resulting in over heating of all the windings
and lack of oil caused all three phases to arc to ground and fail. With feeder
1Q15 de-energized due to the failure of the transformer, the network was
operated with 9 of its 22 feeders out of service.
According to the transformer forensic analysis report developed by Con Edison,
at the time of the failure the transformer temperature was above its hot spot
limit of 1350C. The nature of the transformer failure suggests that the PVL
ratings need to be re-examined. VS 339 ratings were 104% of nameplate load
during normal operation, 127% of nameplate for first contingency and 146% of
nameplate for second contingency operation.
After the restoration of feeder 1Q16 earlier in the day, the secondary mains
along 48th Avenue began to separate. The secondary cables either failed or
limiters installed in the secondary cables operated, breaking up the
interconnected grid of secondary cables. This placed even more burden on VS
339, as it supported a larger area without nearby transformer support being
interconnected via the secondary mains to share the load. The fuses blew on the
transformers in service surrounding VS 339 as individual phases overloaded on
the transformers. This caused secondary problems on 48th Avenue, and south of
vault VS 339 on 47th Avenue in Area II.
Con Edison’s PVL based simulation of the event after the fact predicted
possible substantial secondary cable overloads at 48th Avenue and 44th Street,
supporting the Committee’s assessment. The simulation also predicted a
possible secondary overload in the vicinity of VS 339 as well as along 40th
Street. Secondary cable was replaced in the restoration process consistent with
the simulation.
67
VS 479 – Transformer Failed Due to Overheating and Deenergized 1Q16
Structure Location
Date and Time of Failure
th
Area
VS 479
July 18 at 20:38
I
Connected Primary Feeder
Street Location
M&S Plate
1Q16
th
30 Avenue and Hobart Street
65AF
Just prior to the transformer failure in VS 479, the Long Island City network
was in an operating condition well above design. Nine of its twenty-two feeders
were out of service, meaning about 480 of 1194 transformers were deenergized, including transformers that had been live-end capped and
transformers off prior to July 17, 2006. Refer to the portion of Con Edison’s
M&S plate shown on Figure 11.
The Committee believes that the secondary mains network began to fail. This
was due to the sustained high loads the secondary mains had to carry as a result
of de-energized transformers. Limiters may have blown due to secondary faults
and high currents, shifting load to other secondary mains. Likewise, the
remaining transformers in service would necessarily operate at very high loads.
The paths of supply between energized transformers and areas without
energized transformers would become very highly loaded. It appears that the
secondary mains which cross natural boundaries experienced high loads due to
the limited number of paths between areas, and these would either separate or
fail due to the excessive loading. This created isolated local load pockets. The
secondary mains would disconnect by either physically burning apart or by the
installed limiters operating. In many cases, the limiters would only operate after
cable damage occurred as the time constant for the limiter is not intended to
protect against overloads.
As the different load pockets lost virtually all transformer support the energized
areas would attempt to keep them operational. The reduction in load caused by
the isolation would aid in stabilizing the other transformers and feeders.
Examples of this separation of the secondary network are seen between Areas I
and III along Grand Central Parkway Extension and Tri-Borough Plaza and
between Area I and Area II along Skillman Avenue, Barnett Avenue and
Northern Boulevard (refer to Figure 3). The transformer in vault VS 479 was
on the edge of an approximate 20 block area of little or no primary supply left in
Area II when it failed.
68
According to the forensic analysis, the transformer failed as a result of internal
pressure building up in the tank from over heating. This pressure caused the
area near a spot weld mid-height on the radiator panel to fail, allowing a
substantial loss of dielectric fluid, resulting in high heating of all the windings
and lack of dielectric fluid caused all three phases to arc to ground and fail. The
RMS system installed on this unit was not operating. Real time data was not
available to confirm the operating temperature before the failure.
This is the second transformer to fail due to the same mechanism which
indicates that the transformers were operating under high loads and were not
cooled or de-loaded sufficiently to prevent failure. For planning purposes, the
transformer operating limits assigned in PVL were at 109% of nameplate for
normal operation, 133% of nameplate for first contingency and 153% of
nameplate for second contingency.
This failure extended the load pocket along 30th Avenue area the Committee has
defined as Area I. This was a new heavily loaded area, with banks off prior to
the event (see Figure 11). Additionally, banks were dropped from both 1Q16
and 1Q17 during the early sequence of the Long Island City network event. In
Long Island City network there were multiple instances when both feeders in
two adjacent bands are de-energized. Three closely related bands were deenergized in this area. All of the feeders were out in the immediate 30th Avenue
area. Four consecutive bands were de-energized in the area north of the
Triborough plaza. A secondary fire also damaged the feeder in manhole M
8405.
Con Edison’s Distribution Engineering’s post event analysis predicted
secondary overloads in the areas surrounding the transformer in VS 479, within
two blocks in three directions away from the vault. During the restoration
process, cable was replaced near the transformer and to the south.
69
Figure 11: V 7813 (1Q20) / TM 6682 (1Q19) / VS 479 (1Q16) / VS 9819 (1Q18)
TM 6531 – Transformer Failed Due to Bushing Failure and Deenergized 1Q16
Structure Location
Date and Time of Failure
th
Area
th
TM 6531
July 18 at 20:38 (FOT 08:11 July 20 )
Fringe near Area II
Connected Primary Feeder
Street Location
M&S Plate
1Q16
Review Avenue south of 37th Street
57Y
On July 18th at 20:37, 1Q16 de-energized after having been restored for
approximately 11 hours. During the next two days it took to restore the
feeder to service, four separate problems were found as follows: The
transformer in VS 479 had failed, there was a cable failure in MH 8405; a
high voltage bushing on the transformer in TM 6531 was found failed
following a hi-pot test; and a network protector was found in the closed
position, during an ammeter clear test.
The transformer in TM 6531, a 1993 vintage transformer failed due to a
primary bushing failure. The transformer failed on test (FOT) at 08:11 on
July 20th. This transformer is located in an isolated spur on the fringe of the
network. Refer to the portion of Con Edison’s M&S plate shown on Figure
70
12. The spur is supplied by 1Q01 and 1Q02, as well as the supply to TM
6531 (1Q16). The ESNA high voltage C-phase bushing was found damaged.
With the primary bushing removed, the transformer passed continuity,
resistance, and turns-ratio tests indicating that the internals of the transformer
were not damaged. The damage was confined to the bushing.
The transformer failed on test (FOT) when the feeder was “hi-potted” prior to
it being returned to service. The failure was attributed to a damaged “C”
phase ESNA bushing based on a Con Edison conducted transformer forensic
analysis. The cause of the bushing failure is based on the analysis performed
by the National Electric Energy Testing Research and Applications Center
(NEETRAC is a research and testing center operated by the Georgia Institute
of Technology) and was “propagation through the epoxy resin insulation until
the dielectric strength of the remaining insulation was reduced to the point of
electric puncture.” The bushing may have “failed as a result of cantilever
loading on the bushing that initiated a crack” which later caused tracking
along the bushing interface. There was insufficient data to determine what
might have caused the cantilever cracking. It should be noted that in addition
to the failed bushing, all three phases of the associated network protector
fuses had operated and were “blown.” Otherwise, the transformer was found
to be in generally good condition. At this point in time, the network was in a
tenth contingency.
Figure 12: TM 6531 (1Q16)
71
VS 8807 – Transformer Failed Due to Overheating and Deenergized 1Q18
Structure Location
Date and Time of Failure
th
VS 8807
July 18 at 21:50
Connected Primary Feeder
Street Location
1Q18
th
Area
II
M&S Plate
th
69 Street near 38 Avenue
61AH
The 500 kVA transformer in vault VS 8807 failed on July 18th after being
operated at very high loads which caused the transformer hot spot winding
and top oil temperature to exceed their design limits.
The transformer in VS 8807 is located at the fringe of the network, and with
most of the other feeders in the area de-energized at the time of failure; the
transformer was subjected to heavy loading. The failure of the transformer
was detected after an attempt to re-energize feeder 1Q18 to service and it
opened automatically. The feeder restoration was expedited under a Section
9.
The transformer’s forensic analysis determined that the transformer failed as a
result of being operated well above its rating. It was overloaded and had
three faulted windings. The transformer initially faulted and the feeder
tripped and when an attempt to re-energize the feeder was made, the attempt
to re-energize caused damage to the transformer. The RT3 readings indicated
that the transformer load peaked at 200% of its nameplate and RMS indicated
that it operated at or above 127% of nameplate for a period of 23 hours prior
to its failure. It was the second transformer to fail in the area. A third
transformer failed in the area less than 24 hours later (VS 477, VS 8807, and
VS 7981). Refer to the portion of Con Edison’s M&S plate shown in Figure
13.
72
Figure 13: VS 7981 (1Q17) / VS 477 (1Q01) / VS 8807 (1Q18)
The transformer’s rating in PVL is 155% of nameplate load for normal load
cycle, and 170% of load under first contingency, and also at 170% of load
under second contingency. The Committee believes that these ratings may
not be appropriate based on the temperature and loads reported via RMS and
that the transformer was not able to operate at 170% of nameplate during first
or second contingencies. The RT3 curves indicate that the winding reached
the hot spot and top oil limits within hours of when it reached approximately
130% of nameplate.
Con Edison’s Distribution Engineering post event simulation predicted
possible overloaded secondary mains in Area II along both 69th Street and
38th Avenue and also on Roosevelt Avenue to the south. Secondary cable was
replaced at or near the intersection of 69th Street and 38th Avenue as part of
the Long Island City network event restoration effort.
The failure of 1Q18 to return to service meant the Long Island City network
was in a ninth contingency.
73
VS 0624 – Transformer Failed and De-energized 1Q19
Structure Location
Date and Time of Failure
th
Area
VS 0624
July 19 at 00:09
II
Connected Primary Feeder
Street Location
M&S Plate
1Q19
49th Street and 43rd Avenue
61AH
Due to the multiple contingencies, secondary mains had become highly
loaded in the area of 49th Street and 43rd Avenue and the load being supplied
by the transformer in VS 0624 grew steadily until it failed by overheating of
the coils. Refer to the portion of Con Edison’s M&S plate shown on Figure
14. At the time of failure, the transformer temperature had reached 214oC and
the load was 235% of nameplate. The transformer was identified as needing
cooling but was not cooled prior to its failure. On July 18th for nine hours
immediately prior to failure, the transformer had exceeded its hot spot
temperature. It had previously exceeded its top oil temperature for 5 hours on
July 17th. This General Electric G8SS unit was received in 1969,
reconditioned in 1999, and its ratings in PVL were 135%, 160%, and 170%
for normal operation, first contingency and second contingency respectively.
When the transformer in VS 0624 failed, it created a situation in the network
from 39th street along Skillman Avenue and 43rd Avenue extending to the
fringe of the network of the Brooklyn/Queens Expressway in which four
feeders and their associated transformers were out of service. This condition
expanded the local areas under multiple contingency and both highly loaded
Areas I and II merged into one large area.
The transformer in VS 0624 was the second transformer failure (VS 7995 was
the first) on adjacent M&S Plates (61AD and 62AC) indicating the area load
was beginning to exceed the capacity of local transformers. At 07:11 and
07:15, as load increased in the area, manhole fires and smokers were reported
in the general vicinity of this transformer. This suggests that the load dropped
by the transformer in VS 0624 was picked-up by other transformers in the
area that attempted to provide voltage and load support to the local area. This
exceeded the capacity of secondary mains in this vicinity and their insulation
was damaged and the mains began to fail and burn.
74
Figure 14: VS 339 (1Q15) / VS 7995 (1Q17) / VS 624 (1Q19)
VS 7981 – Transformer Failed Due to Overheating and Deenergized 1Q17
Structure Location
Date and Time of Failure
th
Area
VS 7981
July 19 at 21:49
II
Connected Primary Feeder
Street Location
M&S Plate
1Q17
62nd Street and Roosevelt Avenue
61AG
When this transformer failed, it placed the network in a 7th Contingency.
There were five transformers de-energized in the immediate vicinity of 62nd
Street and Roosevelt Avenue and secondary mains had become overloaded in
the area. The transformer in VS 7981 failed. At the time of failure, its RMS
was not reporting, however, from analysis of RMS readings for adjacent
transformers, the Committee believes that VS 7891 was loaded to
approximately 220% of its normal nameplate rating when it failed. The
transformer was not cooled prior to its failure. Analysis of the failed unit
indicated the cause of failure was overheating.
VS 7981 was the third transformer to fail on three adjacent M&S Plates
located in this highly loaded area during this Long Island City network event:
75
see Figure 13 (VS 7981, VS 8807, and VS 477). At 07:20 as load began to
increase in the area, manhole fires and smokers were reported in the general
vicinity of this failed transformer suggesting that the load previously supplied
by the transformer in VS 7981 was being picked-up by the remaining
transformers. The effort to provide voltage and load support for the local load
pocket, overloaded the secondary mains feeding the area to the point that the
insulation began to fail and burn.
Figure 15: VS 7981 (1Q17) and three transformers with three different load cycles
76
TM 6682 – Transformer Failed Due to Secondary Bushing
Failure and De-energized 1Q19
Structure Location
Date and Time of Failure
TM 6682
July 21 at 17:25
Connected Primary Feeder
Street Location
1Q19
Area
st
th
25
I
M&S Plate
th
Avenue between 46
th
47 Street
Street and
On the borders of M&S
plates 68AF and 68AG
The transformer that failed in structure TM 6682 was a 500 kVA unit
installed in 1977 at 25th Avenue (see Figure 16). Based on the transformer
forensic analysis, the transformer sustained an internal failure of all three
primary coils. The calculated RT3 winding hot spot reached 152 degrees C
(designed to 150 degrees C) and top oil reached 129 degrees C (designed to
125 degrees C). The unit leaked dielectric fluid through the low voltage
secondary bushing and into the network protector housing. The level of
dielectric fluid was found inside the transformer to be at the height of the
secondary bushings. The transformer failed when the live three phases of
high voltage components of the ground switch and tap changer arced to
ground.
RMS loading data was not available on the day of failure, July 21, 2006 and
does not correlate with temperature data at the time of the failure. RMS
stopped reporting reliably and indicated virtually no load on all three phases
of the transformer since 15:47 on July 20th. However, RMS did show that the
transformer was supplying loads above the second contingency rating of
156%. This unit supplied 169% on July 17th and about 160% on July 18th. It
exceeded its top oil temperature for 16 hours and hot spot winding
temperature for 8 of 11 hours on July 18th.
The PVL engineering ratings for the transformer were an operating limit of
118% for normal conditions, 139% under first contingency and 156% under
second contingency. The contingency in the network was well above a
second contingency during the high load period when, due to a stress
corrosion failure identified in the forensic analysis report, the secondary
bushing assembly failed and leaked. The leaks continued until the dielectric
fluid level dropped below the critical height exposing high voltage
components that then arced to ground and the transformer failed.
As part of the restoration process, secondary cables were replaced on 25th
Avenue, 46th Street and 47th Street.
77
The opening of 1Q19 placed the network into a second contingency with 2
feeders of the 22 in the network de-energized. The Long Island City network
operated at this second contingency for 13 hours until TM 6682 was live-end
capped and feeder 1Q19 was restored. Feeder 1Q15 remained de-energized
during the entire time 1Q19 was out of service.
Figure 16 – V 7813 (1Q20) / TM 6682 (1Q19) / VS 479 (1Q16) / VS 9819 (1Q18)
78
V 7388 – Transformer Failed Due to Overheating and Deenergized 1Q17
Structure Location
Date and Time of Failure
nd
V 7388
July 22
Connected Primary Feeder
Street Location
1Q17
at 20:35
Area
III
M&S Plate
st
31 Street and Ditmars Boulevard
On the borders of M&S
plates 72AF and 71AF
The transformer in vault V 7388 is a 500 kVA (see Figure 17) unit
manufactured in 1965 that was reconditioned at the Astoria shop in April of
2004 and re-installed in June of 2004.
Figure 17: V 7388 (1Q17)
Feeder 1Q17 was the first feeder to de-energize in the Long Island City
network event. It had four in-service operating failures between July 17th and
79
July 22nd 2006. The transformer in V 7388 failed nearly two days after the
feeder had been restored for the third time. RMS reporting for this
transformer was sporadic, however a review of the area and RMS data
confirm that the transformer operated above 80% of nameplate load for
approximately 12 hours before the feeder opened on July 18th. This is within
the assigned PVL ratings of 99% of nameplate for normal operation, 122%
for first contingency and 140% of nameplate for a second contingency. These
ratings are used for the determination if overload conditions would occur
under normal, first, and second contingency network operating conditions
respectively.
Transformer forensic analysis revealed the transformer failed as a result of
being operated well above its temperature rating. According to the calculated
RT3 temperatures, the winding hot spot reached 160 degrees C (it was
designed not to exceed 135 degrees C) and the top oil reached 137 degrees C
(it was designed not to exceed 125 degrees C). A primary fault developed
and the feeder tripped.
As mentioned previously, RMS data was sporadic for this unit when 1Q17
was restored on July 20th. The general portion of Area III (see Figure 3) in
which this transformer was located had reported manhole events and 10 of the
transformers supplied by 1Q21 along Ditmars had been live-end capped. On
July 22nd, the transformer did register RMS load readings in excess of 150%
of nameplate loading average for 4 hours before failing. The load was
unbalanced with “A” phase load at 203%, “B” phase load at 201% and “C”
phase load at 113% near the time of failure. Many of these readings are in
excess of the operational ratings assigned to the transformer by PVL.
The elevated level of contingency degraded the condition of the secondary
mains. This combined with the fact that a number of transformers were
removed from service during restoration seems to have contributed to the
high loads the transformer was seeing upon restoration along with the phase
imbalance. Specifically, 10 transformers in 1Q21 on Ditmars Blvd. from 26th
to 81st Street were live-end capped and dropped off the feeder early in the
Long Island City network event. These units were still de-energized when the
feeder 1Q17 (and the transformer in V 7388) was restored. Smoking
manholes were reported in the vicinity during the overall event, indicating
that the secondary mains experienced damage, creating the imbalance of loads
seen in the RMS load data.
80
Con Edison’s post event simulation predicted possible secondary overloads in
the immediate vicinity of this transformer and damaged secondary cable was
replaced as part of the restoration effort.
The transformer failure caused 1Q17 to open automatically. It was the only
feeder de-energized at the time and was restored approximately 8 hours later
after the transformer was live-end capped isolating the fault.
Why Transformers Failed
The Long Island City network is supported by 1194 transformers. Thirteen of
these transformers failed during the Long Island City network event and
additional transformers were replaced after the event as a result of Con
Edison inspections. Overheating contributed to ten of the transformer
failures. Two transformers failed due to corrosion and one transformer
primary bushing failed. A listing of the thirteen transformers is provided in
Attachment F. Transformers became highly loaded as a result of picking up
load from other transformers during contingencies beyond Con Edison’s N-2
design. Transformers picked-up load because:
•
•
•
Other (near-by) transformers were out of service
The secondary grid had broken-up such that load was supplied by
transformers located more electrically distant than transformers
that would normally supply the load
Multiple feeders were out of service and the system was operating
above its design capability
Transformers were out of service because:
•
•
•
Transformers failed while in-service and were isolated (live-end
capped)
Transformers were associated with a feeder that opened
automatically and de-energized
Transformers were associated with a spur of a feeder that was
isolated and de-energized
The failure of these transformers contributed to the propagation of the event.
Therefore, the Committee believes that important lessons about the event can
be understood by knowing how the transformers failed from a transformer
forensics perspective and, more importantly, by knowing what created the
conditions to which the transformers were subjected, leading to their failure.
81
Transformer Failures from a Forensics Perspective
From a component forensics perspective, transformer failures are summarized
as follows and described in more detail later in this section of the report:
•
High Internal Temperatures
o Four (VS 7995, VS 0339, VS 0479, and TM 6882) of the thirteen
units failed due to the dielectric fluid level dropping below the
high voltage components which then arced to ground. The cause
of these failures included corrosion of the transformer tank and
operation of the transformers at loads which caused the internal
temperatures to exceed its design. These transformers were not
de-loaded or cooled to reduce the temperature. In two instances
(VS 0339 and VS 0479), the internal pressure was high enough
that it caused the material adjacent to the radiator welds to fail,
resulting in significant loss of dielectric fluid. In the case of TM
6682, the temperature and loadings were such that when the low
voltage bushing assembly leaked due to corrosion, the oil level
dropped to a level that caused the high voltage components to arc
to ground. In the case of VS 7995, the transformer overheated due
to loss of dielectric fluid as a result of corrosion.
•
There were ten failures where the transformer forensic analysis report
attributed the cause to overheated coils as a result of the top oil and hot
spot winding temperature being exceeded. The multiple contingency
conditions caused the transformers to be loaded in excess of design. In
addition, based on transformer cooling information data, these units were
not cooled or de-loaded sufficiently to prevent the units from failing.
Self-Reinforcing Progression
The conditions leading to the failure of each transformer have been described
in detail previously in this section of the report. In summary, those conditions
can be described by this self-reinforcing logic:
• The progression began when transformers in a given area had
become de-energized
• Customer loads still existed in the area
• Secondary mains and transformers surrounding the area picked-up
the load of the de-energized transformers
82
• Secondary mains and transformers that picked-up this load became
highly loaded
• Secondary mains cable insulation and transformer oil temperatures
increased and became subject to overheating and failing
• When secondary mains and transformers failed, load shifted to other
mains and transformers that became more highly loaded and
became subject to failure which reinforced the progression
This cycle will continue until it is arrested by remedial actions such as
restoring feeders and reducing load or until secondary limiters operate or
secondary mains damage is extensive enough to isolate transformers from the
secondary grid.
There were 21 feeder outages of greater than two hours duration during the
Long Island City network event. Thirteen of these primary feeder failures
were caused by transformer failures. This caused loads to be shifted resulting
in more overheating and secondary grid break-up that isolated the shifted load
to transformers which continued the cycle.
Conclusions
Event Preconditions
•
Con Edison mobilized its resources prior to the event. Con Edison
assures proper maintenance of its transformers through use of
written specifications. Transformers that had been removed from
service on June 29th and July 11th were evaluated using the RMS
data available for nearby transformers in the immediate area. If as
a result of a contingency operation, a local area is not served by a
well-diversified intermesh of feeders and in-service transformers,
problems may develop requiring emergency repairs to restore the
local area to a well-diversified condition. If repairs are not
possible, special attention should be given to the local load pocket
to ensure that feeders, transformers, and secondary mains remain
within design limits. Recognizing the need for emergency repairs
or other special attention to ensure local pockets remain within
design limits, relies upon local area load pocket studies that alert
operators to closely monitor equipment that may become
overloaded during contingency operations and to inform decisions
regarding feeder restoration priorities and use of live-end capping.
83
Event Speed:

Designers provide the critical element of time for operators and field
crews to remedy equipment failure problems and prevent them from
spreading.

Designers do that by providing the appropriate number and size of
secondary mains, the number and size of primary feeders and the
number, size, and location of transformers to supply the forecasted
customer demand. As described in more detail in the Ratings and Load
Cycle Section of this report, “size” in this context refers to both the
number of the available cables and transformers and their available
capacity.

If the nearby secondary mains and transformers are already operating
near their design limits, under design condition, then the responders will
have very little time to prevent the cycle from starting. During the Long
Island City network event, after the de-energizing of 1Q17, 1Q16, and
the loss of 1Q21 and 1Q07, responders had very little time to prevent the
cycle of self-reinforcing events. Two additional feeders failed during
the evening of July 17th. Overnight the speed of the self-reinforcing
cycle was slowed by lower loads but beginning at approximately 08:30
on July 18th, as heat and load increased, the speed of the cycle was
greater than the ability of the responders to remediate problems and
break the cycle. From approximately 08:30 on July 18th to just past
midnight (about 16 hours) nine transformers failed causing eight
separate feeder de-energizing events. The Committee considers this
period to be an extremely fast moving progression.
Area Demand:

Lowering demand in an area prevents or slows the time it takes
components to overheat. This provides time for responders to restore
failed components and cool highly loaded transformers. During the
Long Island City network event Con Edison took early and effective
action to reduce load in the network. This helped prevent feeder
overloads that undoubtedly prevented additional failures. Network load
reduction efforts are more effective in the industrial area of the network.
Facility managers are normally ready and able to effect load reduction
quickly and effectively. Load reduction is much more difficult in
residential areas where many small load reductions are required to have
a significant contribution to the reduction of network load. However,
84
since a key driver of this event was the transformer failures and the
separation of the secondary grid due to local overload problems, an
important remedial action at this point in the cycle is to reduce local area
demand.
• Secondary Cable overheating is reduced through load reduction. Other
actions are available such as opening network protectors to reduce load,
rerouting load paths through temporary shunts, and relieving secondary
mains with portable generation. Those other actions should be considered
on a case-by-case basis but may carry disadvantages that make them
impractical some of the time. As mentioned previously, during the Long
Island City network event, demand was reduced on the network. A
targeted approach, especially early in the event, addressing local load
pockets of highly loaded secondary cable, may have helped to arrest the
cycle. However, reducing load in a specific area is difficult in the Long
Island City network because of the high percentage of residential
customers. Burning secondary cable insulation also communicated to
primary cables causing primary cables to fail.
Transformers Overheat:
• Preventing transformers from overheating is accomplished by ensuring its
ratings for normal, first, and second contingency operations are
appropriate and provide sufficient degree of operating margin. This is to
allow picking-up load while not quickly exceeding its design limits and
by implementing cooling actions well before the transformers approach a
critically overheated condition. During the Long Island City network
event some transformers heated quickly after picking up load.
Responders cooled various transformers effectively. Others that were not
cooled failed. One transformer that was cooled failed.
Feeder Restoration:
• In a multiple contingency events, feeder restoration is always a key
component in stopping a self-reinforcing progression. Con Edison
recognized this and has greatly reduced the feeder restoration times. In an
event such as the Long Island City network event, feeder restoration
during extreme contingencies is even more critical. Restoration also
needs to be planned and timed to avoid restoring a feeder into an
operating situation where its components could quickly become
overloaded. The self-reinforcing cycle may be arrested at this point by
85
restoring multiple feeders in a specific sequence or simultaneously. This
will help to avoid overloads that may lead to further failures of
transformers, cables or heat sensitive joints. During the Long Island City
network event, feeders were prioritized to reduce the number of network
contingencies as quickly as possible. This is normally an effective
prioritization. However, early in this event, field conditions were such
that local area load pockets had become under-supported by transformers.
When feeders were restored later, and transformers were energized to
supply those areas, the transformers picked-up too much load and became
overloaded. Additionally, when feeders were restored, new load paths
into those areas were created and a number of the secondary mains
overloaded and insulation failed.
• Remedial actions include carefully deciding the balance between the
advantages of quickly restoring a feeder by live-end capping transformers
and isolating feeder spurs, and the potential problems associated with
creating local pockets of load which may result in transformers
overloading and secondary mains load paths causing overheating. During
the Long Island City network event, feeder restoration was expedited by
live-end capping transformers and isolating feeder spurs. This action
benefited the Long Island City network as a whole, however, restoring the
feeders without those components contributed to the local load pocket self
reinforcing progression as described previously.
Recommendations
Event Preconditions
• Develop a procedure such that pre-event analyses include confirming that
all transformers that are out of service are accounted for
• Establish criteria which require that all transformers that are forecasted to
be loaded at or above a pre-determined value prior to the event are analyzed
to identify potential load pocket problems. These transformers should be
monitored during the event.
• Consider the impact of dropping un-faulted transformers on the next worst
event prior to dropping un-faulted transformers.
• Consider establishing a transformer installation team equipped with
vehicles, equipment, and material to replace transformers during the event.
• Establish clear criteria, as part of EO-10110 that requires appropriate
supervisory approval to downgrade a “heavy corrosion condition” to a
lower classification.
86
Event Speed
• Investigate the use of arc-proofing on secondary cables and crab joints or
the installation of fire shields between secondary crabs and primary cables
to limit collateral failure and the communication potential from fire
damage from one to the other.
Area Demand
• Develop procedures to reduce small commercial and residential customer
demand during periods in which a local high load pocket is subject to
expanding due to overloaded or highly loaded nearby transformers and
secondary mains.
Transformers Overheat
• Consider reducing the action-threshold which identified the specific points
that transformers are to be cooled.
• Conduct PVL studies as part of summer preparation for all networks to
identify transformer cooling candidates if multiple contingencies beyond
design criteria were to occur.
• Train additional field crews who can supplement crews normally assigned
to cooling transformers. Use the Long Island City network event as a
guide to determine staffing levels to ensure cooling can begin early
enough to prevent transformer overheating. Consider a three hour period
until a study can be performed to identify a suitable timeframe.
Feeder Restoration
• Revise EO-4095 to more specifically address details how to use knowledge
of “field conditions” when making decisions about how to restore feeders
in terms of priority, live-end caps, shunts, and partial restorations.
• Investigate if design modifications or different criteria are appropriate
during normal operation and contingency operation along network fringes.
The fringe area of the network is most vulnerable to directional network
support.
• Revise EO-4095 to ensure that special attention is directed to network
fringe areas and ensure the operators and designers consider new fringe
areas created during multiple contingency events.
87
Trouble Analyses
Overview
Con Edison’s Electric Operations Specification EO-4095 “Distribution System
Operation Under Contingency Conditions” specifies that its operating regions
must:
“Expedite feeder restoration based on field conditions and work status.
Restoration options include expediting cable pulling and splicing functions in
progress and live-en -capping of cable. Isolate the faulted leg of a bifurcated
feeder by opening the sectionalizing switch and re-energize the sound leg of the
feeder. It is imperative that every network feeder “Open-Auto” occurrence,
from June 1 to September 15, be treated on a HIGH PRIORITY BASIS”
Monday July 17th into Wednesday July 19th
Brooklyn/Queens region interviewees indicated that during the Long Island City
network event, the primary focus of Incident Command, Operations,
Engineering, and field crews was on the restoration of feeders and monitoring
feeder loading to avoid overloads that may have resulted in feeder deenergization due to overheating and failure. Their stated primary focus was to
expedite feeder restoration.
To “Expedite feeder restoration based on field conditions…” Incident
Command would have needed to know real-time field conditions sufficiently
well to ensure their efforts to restore feeders did not create transformer and
secondary mains problems that might lead to further feeder, transformer, and
secondary cable failures or to customer outages.
However, at the start of the event, Incident Command – believing that the
volume of trouble tickets from customers and the number of manhole smokers
and fires would be low – decided that it would be in the best interest of Con
Edison customers to route trouble tickets directly to the organization that
manages field crews and to the emergency desk of the operations center as is
normally done rather than to a special section of the Incident Command
organization typically referred to as the Trouble Analysis Group. At the time,
this decision appeared to be in the best interest of customers as it would ensure
that crews would be dispatched as quickly as possible to respond to these types
of trouble reports.
88
During the first days of the event, Incident Command was advised of the
number of trouble tickets but was provided little if any engineering analyses of
those tickets. The call volume was low and the number of manholes events
appeared low, especially when compared to the number of manhole events Con
Edison experiences during winter storms. As a result, Incident Command did
not identify localized problems with secondary mains and to what extent
customer’s service might have been affected by those problems.
Con Edison procedures call for the mobilization of the Brooklyn/Queens
engineers as part of the Incident Command structure. The engineers were
mobilized on Monday July 17th. During large-scale outages, trouble tickets are
routed to Brooklyn/Queens engineers who review network conditions, analyze
the trouble tickets and provide information to Incident Command useful in
determining restoration priorities.
The Brooklyn/Queens Region engineers’ function of analyzing secondary mains
and customer impacts provides predictive information to Incident Command.
The information assures that decision-making considers the likelihood of
primary feeders, network transformers, and secondary mains becoming highly
loaded, failing, thus impacting service to customers. The engineers also advise
Incident Command regarding issues such as heavily loaded transformers that
may be at their load limits. Additionally, the engineers identify if cooling
efforts are successful and if there are correlations between the information on
trouble tickets and events in the field such as cascading manhole fires as an
example.
Some of the decisions that rely upon the analyses of secondary mains trouble
during events such as the Long Island City network event included:
• When, where, and how far should localized load be reduced
• What equipment can be isolated by live-end capping of feeders and
when it should not be used to expedite feeder restoration
• Prioritizing which feeders should be returned to service first
• Estimates and impact of how much secondary load will be picked-up
when a feeder is restored
• What the secondary transformer loading upon feeder restoration is
estimated to be
• When specific feeders should be restored within a short time of one
another to minimize thermal stressing
• An estimate of where, how many, and what customers were being
impacted by an outage or restoration move
89
Based on information provided to the Committee, from Monday into early
Wednesday morning, the Brooklyn/Queens personnel monitored trouble ticket
volume to identify if the reports of manhole fires and smokers might indicate a
cascading event.
As can be seen in the table below, trouble tickets, and the number of reports of
smoking or on-fire manholes was low for the period. Calls in which the
customer service personnel referred the caller to seek the services of a non-Con
Edison electrician should be reviewed in the future. Under normal operations it
may be appropriate not to review these type of tickets however, during a
secondary event as was experienced in the Long Island City network event, had
these calls been forwarded to the Trouble Analysis group, the intelligence may
have aided the technicians in assessing the extent of the problem sooner. It may
have also given them information about how wide spread the problems were in
the field.
Trouble Tickets
Type of Report
17-Jul-06 18-Jul-06 19-Jul-06
Caller refered to an electrician
10
47
171
Subtotal
10
47
171
Manhole fire
1
2
9
Smoking manhole
4
17
56
Subtotal
5
19
65
Flickering lights
12
19
11
Low voltge
8
259
124
No lights
7
113
552
No lights in the area
5
41
332
Side off
8
399
136
Side off power needed
2
9
8
Subtotal
42
840
1163
Burning wire
1
2
8
Wire down
1
0
1
Miscellaneous electrical trouble
3
9
37
Other
1
4
9
Subtotal
6
15
55
Grand Total B Tickets
63
921
1454
20-Jul-06
103
103
2
7
9
5
99
443
307
123
20
997
0
0
52
10
62
1171
21-Jul-06 Total
136
467
136
467
3
17
11
95
14
112
6
53
135
625
547 1662
325 1010
131
797
19
58
1163 4205
5
16
3
5
73
174
16
40
97
235
1410 5019
Table 2: Trouble Tickets by Date and Type (The source of the data is Con Edison)
Certain messages from Con Edison to customers may have had the unintended
consequence of inferring to customers that Con Edison knew of the Customers’
problems and it was not necessary for the Customer to report the problem. Two
examples of messages that were provided at various times during the event and
90
that had the potential for inferring to customers that Con Edison knew of the
Customers’ problems include:
We are currently experiencing an 8% voltage reduction in our Long Island City
network area in Queens. Although this should have no impact on your service,
we’re asking anyone living within that area as follows: on the west, the East
River / on the east, the Brooklyn Queens Expressway / on the north, Long Island
Sound / and on the South, on Newtown Creek; To please cut back usage on all
non-essential appliances wherever possible.
Hundreds of Con Edison crews have been working to stabilize the electrical
system in northwest Queens’s neighborhoods of Long Island City, Sunnyside,
Woodside, Hunters Point, and Astoria. We appreciate the continued
conservation efforts of our residential and commercial customers in these areas.
The Brooklyn/Queens Region engineers were monitoring the system through
Con Edison’s Remote Monitoring Application (NetRMS) for overloaded
transformers or transformers reaching temperature limits. In situations where
RMS was not providing remote readings, the engineers dispatched field
personnel to conduct on-site checks then ordered cooling of suspected
overheated transformers specifying the priority as 1, 2, or 3. The engineers also
analyzed the network using various engineering support applications.
Engineering Support Applications Problems
Note: Below is a discussion of how certain engineering support applications
were utilized during the Long Island City network event. Greater detailed
explanations of these applications and their uses can be found in PVL, WOLF,
and RMS Sections of this report.
Con Edison specification EO-10110 “Inspection and Maintenance of Network
Type Distribution Equipment,” requires that its operating Regions do the
following:
“The RMS system of each network must be maintained to insure that at
minimum, 95% of the total number of units in the network is reporting
properly.”
At the beginning of the Long Island City network event, approximately 77% of
units in the network were reporting properly. The Remote Monitoring System
(RMS) is a tool used for both planning and operations. It is used by Con Edison
to monitor operating conditions on secondary transformers so that – in the event
91
of a highly loaded or overheating condition – Con Edison can have sufficient
data and warning to dispatch crews to de-load or cool the transformers. RMS
also provides key data into engineering support systems such as WOLF and
VDAMS. Information from RMS is also used to design and plan load relief.
Con Edison specifies that:
“If a distributed network experiences a second or higher contingency, review
WOLF exception reports for existing network conditions and monitor the
VDAMS to detect possible overloads on feeders, network transformers, and the
secondary grid” and “for network grid overloads, monitor its status, check for
reported smoking manholes, manhole fires, and customer low voltage
conditions.”
The state of the RMS data and the condition of the PVL model, in part,
complicated the response of Brooklyn/Queens engineering during this event.
Prior to the event, on July 12th, Brooklyn/Queens staff noticed incorrect
overloads being reported by WOLF when running the software.
Brooklyn/Queens engineers had attributed the errors to a PVL model version
control issue and speculated that old versions of the model were in use and not
appropriately updated.
Early during the event Brooklyn/Queens engineers tried to use the model but it
would not converge. It converged to the 4th contingency. WOLF stopped
working completely at the 6th contingency. The engineers started to get
erroneous results on 4th, 5th, and 6th contingency scenarios. In addition, when
Auto-WOLF was run during the event, the ”feeders out” report generated
incorrect data. The date and time reported were incorrect and the total load was
reporting lower than actual.
Therefore, the engineers began to make direct PVL runs. Switching to PVL
changed the way the model was run. PVL doesn’t utilize real time data as does
WOLF so the engineers compared the PVL data versus NetRMS. If the
engineers saw a conflict with NetRMS, the engineers would request that field
crews visit the location to obtain the data. This is normally referred to as a
switch check. As is discussed in the Ratings and Load Cycle section of this
report, neither PVL nor NetRMS were consistently providing accurate or
complete analyses of the event.
A finer resolution of prioritization is needed in ordering switch checks and
cooling checks. The Brooklyn/Queens methodology specifies three levels of
prioritization. During the event, a list containing dozens of high priority cooling
92
checks was generated, but the number of high priority checks were greater than
the available staff could address. A finer resolution of prioritization would help
to ensure the most effective cooling strategy is addressed by available resources.
The Committee noted that the format of the reports generated by the
engineering group for use by the Incident Command repeated ”canned text” and
little contingency specific information. The reports did not provide analyses
regarding the specific causes of the spreading secondary overloads.
As noted previously, when a multi-contingency state was being evaluated,
WOLF and other PVL-model based tools provided erroneous data, or the
models themselves did not sufficiently converge. Some erroneous data is
generated when utilizing a network load flow model in multi-contingency
situations. However, the state of the connectivity model at Brooklyn/Queens
Region made this issue more acute. The Committee was advised that the
accuracy of the model was suspect even when using a revised PVL model and
the engineers questioned the results of their analysis of the fifth contingency
situation on the night of July 17th.
Another issue with the model had an escalated impact during the event. As
feeders were restored and transformers live-end capped as part of the restoration
process, the actual network connectivity changed but the model can not be
updated real time to reflect those changes. The feeders have different
characteristics with the additional banks off. Generally, this difference is
limited. However, errors built-up in the assessment of the situation as the
number of transformers were removed from service increased during the
restoration process. This condition was critical for local load pockets of high
loads that were created and not observable through the engineering applications.
The outputs from many of the engineering applications report their results, such
as overloads, on an exception basis. While this is helpful in sorting out critical
information, the Committee believes that certain aspects of the day to day
system operation are not observed. For example, if a particular asset is
operating near or at its assigned limit of its rating, it would not be reported.
Actual Load Compared to Forecasted Load
During the Long Island City network event, the goal was to stabilize the
network. There was a major effort to reduce load and the Company was
successful in doing that. Long Island City network voltage was reduced by 8%
to get immediate load reduction. Major customers were contacted and advised
of the contingency condition and requested to reduce demand and/or switch to
93
emergency generators if possible. The NY ISO initiated a system reduction,
and customer appeals were requested through various means to reduce demand.
450
400
350
300
250
200
150
100
50
-
12
10
8
6
4
2
15:50
16:22
17:00
18:48
19:10
19:48
20:08
21:43
21:49
22:00
23:21
0:00
1:00
2:00
3:00
4:00
5:00
6:00
7:00
8:23
9:33
10:00
11:37
11:55
12:00
13:00
14:00
15:14
16:00
17:00
18:00
19:00
20:05
20:33
20:33
20:38
20:53
21:26
21:46
21:50
22:25
23:00
0:00
0:06
1:00
2:00
3:00
4:00
5:00
6:19
7:00
8:51
9:00
10:00
11:33
12:00
13:10
13:37
14:00
15:00
16:00
17:00
18:00
19:05
20:41
21:29
22:00
23:00
0:46
1:00
2:00
3:00
4:33
6:36
12:38
13:37
13:48
14:00
15:00
16:00
17:00
18:00
19:00
20:00
21:00
22:00
23:00
0:00
1:00
2:00
3:00
4:00
5:00
6:37
7:49
8:01
9:00
10:00
11:00
12:00
Demand (MW)
As can be seen by Figure 18 based on a post-event analysis using some data that
was not available real time, a reduction in demand was successfully achieved by
the tactics employed by the Company. Figure 18 also illustrates the difference
between the customer demand that Con Edison forecasted with and without its
demand reduction actions. Figure 18 suggests that where there is a departure
between the forecast and the actual load, customer service may have been
disrupted. Had detailed demand data (such as depicted in Figure 18) been
available during the Long Island City network event, it may have enabled Con
Edison to identify that customer outages were more widespread than estimated.
0
7/17/2006 12:00 7/18/2006 0:00 7/18/2006 12:00 7/19/2006 0:00 7/19/2006 12:00 7/20/2006 0:00 7/20/2006 12:00 7/21/2006 0:00
7/17/2006
7/18/2006
Contingency Level
Forecast
Date7/19/2006
7/20/2006
Forecast with Demand Mgmt
7/21/2006
Actual
Figure 18: Con Edison’s Contingency Levels (in bars), Demand Forecast (generally the
top line, Demand Forecast after implementation of Demand Management Actions
(generally the middle line), and Actual Demand (generally the bottom line) (The source of
this figure is Con Edison)
During the Long Island City network event, Incident Command could not
accurately differentiate between load reduced through the 8% voltage reduction,
load reduced as a result of their efforts to reduce customer demand, and load
which may have reduced as a result of customer outages. Since customer call
volumes appeared low, Incident Command assumed that efforts to reduce load
were successful and that the lower load in the network was almost entirely the
result of reduced customer demand.
94
Wednesday July 19th into Friday July 21st
On Wednesday July 19th, the Brooklyn/Queens engineers associated and
grouped trouble tickets to enable more productive dispatch of field crews to aid
in the restoration of customers. There were 1,163 trouble tickets regarding
customer service problems generated on Wednesday. There were 65 trouble
tickets from manhole events generated on Wednesday. The Committee found
that the engineering analyses were being done to more effectively dispatch
repair crews. They were not as effective in evaluating secondary loading
problems or identifying optimum feeder restoration priorities based on local
pockets of high loading.
It is noteworthy that due to the high number of feeders that were opening
automatically when restoration was attempted, on Wednesday morning Incident
Command ordered modified hi-pot tests to be performed on every feeder prior
to restoring the feeder. Doing so reduced the CIOA’s that had disrupted
restoration activities during Monday and Tuesday. This was a positive decision
in the management of the event. CIOA’s are discussed in the Feeder Processing
section of this report.
Incident Command had waived the hi-pot testing requirement at the start of the
event in order to expedite feeder restoration. Doing so is the generally accepted
practice at Con Edison during emergency conditions; it is consistent with their
General Rules as well as specification EO-4095 section 7.4. However, from the
start of the event until Wednesday, the number of CIOA’s contributed to the
spread of the event extending the length of time feeders remained out of service.
On Wednesday, Incident Command recognized the need to perform hi-pot tests
based on the unsuccessful cut-ins and this helped to control the course of the
event.
On Thursday July 20th, Con Edison’s Corporate Emergency Response Center
(CERC) directed that a detailed on-site survey be conducted on the evening of
July 20th. That survey indicated approximately 25,000 customers may have
been affected and were out of service. Follow-up field surveys were conducted
with more detailed instruction given to the personnel conducting the surveys to
get more accurate information on a street-by-street basis. This had a two-fold
benefit, it provided a more accurate estimate of the location and number of
outages, and it also aided in the evaluation of secondary damage. With this
information a more strategic recovery was formulated.
95
While the full CERC was mobilized on Thursday July 20th, executives and
senior managers were engaged in providing high-level oversight from the start
of the event on Monday night July 17th.
On Friday July 21st, Con Edison determined that restoration efforts were
blowing fuses and burning mains. The secondary mains loading had been
causing network damage. Con Edison corrected this situation by assuring that
secondary load was factored into their feeder restoration plans and began to
successfully restore service to customers. In the morning of Friday the 21st,
25,000 customers were out of service and on Tuesday the 25th, service to all had
been restored.
Conclusions
Secondary Trouble Analysis
• At the early stages of the Long Island City network event, engineers
reviewed transformer loadings, open network protectors, and blown
fuses. Engineers also monitored trouble ticket volume regarding low
voltage complaints, customer outages, and manhole events. As the
event progressed and the level of contingencies increased, local
pockets of highly loaded secondary were not identified adequately or
fast enough because the existing analysis tools limited engineers’
ability to effectively integrate information from various sources
regarding the level of degradation of secondary mains, blown limiters,
and lost services to customers.
• The decision to live-end cap some transformers placed significant
emphasis on the most expeditious way to restore the primary feeder.
The potential negative impact and consequences of highly loaded
secondary caused by live-end capping individual transformers and the
relationship between feeder restoration priorities and secondary load
pockets were given less weight than the benefit of rapidly restoring
the majority of the primary feeders to service.
Customer Calls
• Incident Command relied on the number of trouble tickets to serve as
a proxy for the customer service disruptions being experienced during
this event and the extent of secondary grid problems. This reliance
was based on historical experience but proved to be insufficient for
this event. Call volume was low but secondary grid damage was
96
occurring and customer impacts were higher than suspected. The
basis upon which Incident Command concluded that the relationship
between customer calls and customers impacted would be a one-toone relationship which is normally the case for a network outage,
turned out not to be appropriate during the multiple contingency event
which involved extensive secondary problems as experienced in the
Long Island City network event.
Recommendations
Secondary Trouble Analysis
• Identify all the sources of information that may assist analysis of
secondary mains trouble such as calls from customers, third parties,
community agencies, field crews, network maps, installed monitoring
systems, and engineering support systems. Then develop a procedure
and provide training to analyze all the streams of information to
identify remedial actions that would isolate the problem areas and
protect uninvolved network components from overload. As part of the
analysis, answer questions such as:
o When, where, and how far should localized load be reduced
o What equipment can be isolated by live-end capping of feeders
and when not to use this method to expedite feeder restoration
o Advise operators of the priority order of when feeders should be
returned to service first and why
o Estimate the impact of how much secondary load can be pickedup when a feeder is restored and its impact on facilities being
restored
o What might be the secondary transformer loading upon feeder
restoration
o When specific feeders should be restored within a short time of
one another to avoid excess thermal stressing
o Estimate of the location and number of customers impacted
during the outage and restoration
Customer Calls
• Develop and implement customer awareness programs to increase
outage calls from interrupted customers served by networks.
• Develop a process to ensure that when field crews receive customer
reports of outages or other service problems, those reports are added
to the information used to analyze secondary mains trouble.
97
• Revise voice scripts to reduce the likelihood of inferring to network
customers that Con Edison is aware of their service problem. State
explicitly that Con Edison needs their specific information to ensure
timely restoration of service.
• Develop criteria and a process to separately identify calls regarding
extremely low voltage trouble from calls regarding low voltage during
an 8% voltage reduction action.
• During periods of network trouble, include trouble calls from third
parties (NYPD, OEM, and FDNY) in the analysis of secondary mains
trouble.
• Review the calls that are currently coded as EDSCRE (referrals to
electricians) during a multiple contingency. Presently these may not
be evaluated, losing critical information that could be made available
to trouble analysis.
During this event, these represented
approximately 10% of the B tickets.
• Consider a research program to develop some type of empirical-based
algorithm or rule-of-thumb that can be used to estimate the locations
and number of customers who may be affected by a network problem
based upon the number of customers who actually called to report a
problem.
98
PVL, WOLF, and RMS
Overview
Note: Attachment D includes definitions of terms and acronyms used in this
section of the report.
Con Edison uses computer applications to support engineering and planning. A
few are heavily relied upon during emergency situations to provide operational
decision support and engineering analyses and have been very successful in
providing accurate and timely information to the Con Edison personnel who
operate the distribution system on a day to day basis. The simplified diagram in
Figure 19 is intended to assist in explaining the relationships of those
engineering support applications discussed in this section of the report.
Emphasis is placed on PVL, WOLF, and RMS. While the total number of
systems available to engineering and operations staff exceeds those discussed in
this report, this section highlights the applications that the Committee
determined had the most impact on the Long Island City network event
response.
WOLF
Auto-WOLF
Load Flow
Analysis from
PVL
Modified Operational
Data from Remote
Monitoring Estimator
Connectivity
Model
Operation
Data from
VDAMS
Asset Data
from CAJAC,
etc..
Remote
Monitoring
System
(located
elsewhere in
this report)
Default
Operational
Parameters
Figure 19: Simplified Diagram of PVL, WOLF, and RMS
99
Brooklyn/Queens Region uses a mix of corporate and locally developed
engineering applications. It has developed its own applications to augment, and
in some cases replace, what is provided at the corporate level. Over the years
each region has used a mix of corporate applications and tools developed within
the region to accomplish its work. While this becomes an incubator for solution
development and migration path for developed solutions across the company,
the different set of tools used from region to region can create logistical and
analytical issues as staff are moved between regions to deal with emergency and
other operational situations.
Brooklyn/Queens has compiled its local applications into what it calls its
Engineering Workstation. Local engineering staff indicated that ease of use and
accuracy are advantages that some of their locally developed applications offer
over other alternatives. However, a noted shortcoming was the inability to
archive engineering studies for later reference or investigation.
Poly Voltage Load Flow (PVL)
Con Edison’s Poly Voltage Load Flow (PVL) application is a balanced 3-phase
load flow application for secondary networks. It employs a Windows client and
UNIX server architecture. Win_PVL utilizes ASCII based models of
distribution system components extracted from mapping systems to simulate the
actual network (including substations, feeders, secondary mains, transformers,
switches, etc.). Win_PVL is the core planning and design tool used by regional
and customer engineering sections to forecast and identify system limitations or
deficiencies and develop reinforcement designs.
PVL is complex and contains hundreds of thousands of individual pieces of
modeling information for underground structures and feeders, including cable
connectivity, cable sizes of primary feeders, secondary mains, and service maps,
and utilizes cable ratings provided by Central Distribution Engineering.
Computer simulated load studies, typically performed with the PVL application,
are used to identify primary and secondary system reinforcement requirements.
PVL uses actual peak summer feeder and transformer load information
collected at the end of the summer load period. Typically, the loads are
adjusted for expected changes in connected customer demand, new customer
additions, planned reinforcement and then projected for the subsequent summer
period. PVL’s accuracy is highly dependent on actual network transformer and
high-tension loads. PVL produces projected normal and emergency loads for
each section of every feeder. The projected feeder and transformer loads are
then compared to their ratings to determine if a feeder or transformer is
100
anticipated to exceed its normal or emergency ratings and develop an
overloaded condition. If the feeder or transformer requires reinforcement, an
engineering layout containing the appropriate feeder reinforcement is
developed. It is authorized through a formal appropriations process and upon
approval issued to the construction group, in an effort to relieve the projected
overload prior to it actually being encountered on the system.
An integral element of PVL is the connectivity model. The model is where the
various elements of data are associated together to actually simulate the system.
The model requires accurate definitions of the feeder components and
equipment such as cable sections, joints, transformers and other related
equipment. In some cases, associated operational parameters also need to be
taken into account (such as equipment age and type, temperature ranges, etc.) in
order to apply proper operational ratings.
The connectivity model also drives other applications used to determine the risk
of feeder and network failure and it is also used to establish which are the
closest (electrically) near-bys to a particular transformer. Errors in this model
can impact the operation of engineering applications dependent upon accurate
system representation.
The Committee found errors imbedded in the secondary connections
represented in the most current version of the PVL model provided by
Brooklyn/Queens engineering and in daily use by the various departments in
Brooklyn/Queens who rely on it for planning and operations.
As an example, it was found that secondary cable was shown to be connected in
the latest Brooklyn/Queens model between manhole M11711 and service box
30111, with no connections between manhole M11711 and service box 30112.
In reality, six sets of cable are connected between M11711 and service box
30112 and while there are ducts that connect between M11711 and 30111, no
cable is physically laid in the ducts.
A secondary load flow analysis the Committee had conducted of the immediate
area surrounding M11711, with the connection errors included, predicted a
possible overload on the section of cable between service boxes 30111 and
30112. These errors would not allow an accurate portrayal of overloads in the
secondary analysis of the area surrounding this manhole. However, this is the
area of the initiating Long Island City event, starting with a secondary cable
fire. Based on the practices of Brooklyn/Queens Engineering, a Load Pocket
Analysis was performed when the transformer in V 9426 was live-end capped
101
on July 12th. The conducted analysis did not utilize the PVL model, nor was a
secondary analysis performed. A “near-by” assessment was performed and the
decision that the bank could be left off was made. That analysis, if conducted
with PVL, could have given indication of secondary overloads that would have
required additional analysis.
Another concern of the Committee involves how the PVL model treats overhead
mains ties and customer loads. When Brooklyn/Queens region conducts an
analysis of underground secondary mains loading, overhead ties are not
included in the connectivity model. Customer loads are represented as point
loads at the manhole riser location. This distorts the real electrical flow and can
affect the accuracy of the results. It also is an issue when restoration is
attempted as the overhead secondarys provide a path for load pickup. There
were several cases where overhead fires were encountered during the restoration
efforts as secondary supply was re-introduced into an area.
There are inherent limitations in the secondary PVL model created by how
individual customer loads are represented within the model. The current
version of PVL aggregates individual customer loads and spots them on a single
point near a transformer. This method of load spotting is effective in
representing primary feeder and transformer loading. This method of load
spotting is less effective in accurately representing secondary main loading. To
accurately represent secondary main loading in PVL for tactical situations,
individual customer loads need to be spotted at the individual customer service
point which is an increase in the level of complexity of the model. An ongoing
project is developing a method to spot individual customer loads at service
points but this was not available for use during the Long Island City event.
The Committee believes that the secondary PVL model constrained the ability
to dynamically analyze secondary load flows and could have been a
contributing factor to the difficulties experienced in managing the event. With
customer loads spotted at transformers and connectivity inaccuracies, the PVL
model will not accurately identify secondary mains overloads for tactical
situations.
Finally, while the PVL model was one of the first of its kind, its current state of
development presents a series of challenges to its users. The Con Edison
network is by far the largest underground urban network in North America. The
complexities of modeling such a network system meant that many of the early
computer tools available in the market would not operate properly for such a
configured system.
102
Con Edison addressed this technical challenge by developing its own network
load flow software (PVL). For years, its capability was unmatched by any other
application available in the marketplace. As the underlying operating systems
changed over the years, PVL was adapted to take advantage of some of the
newer features such as operating in a windows environment.
The reports from PVL primarily report by exception (i.e. overloads) as standard
reporting of all load conditions become quite voluminous. The nature of the
output reports is not intuitive. The Committee observed confusion by staff as to
the meaning of the columns displayed in PVL. They needed to research what
the codes and values represented by looking at a variety of specifications and
also talking directly with developers of the program.
The nature of the PVL user interface does not facilitate easy understanding of
the network or its state in the conducted assessment. PVL is cumbersome to
update and difficult to ensure accuracy of the thousands of connections
represented in the model. A graphical version of PVL has recently been
developed for primary network analysis and Con Edison plans to add secondary
network analysis capability.
WOLF
WOLF, World class Operation Load Flow, is a binary-based subset of PVL
used in real-time load flow analysis environments. It was developed in order to
analyze entire networks quickly, including results for the current (base) case and
all “next worse” cases of feeders which are currently in service. WOLF is run
manually by the user, while Auto-WOLF runs automatically for every network
feeder event (loss or restoration of each feeder) and that report is quickly
available to the regional control room operators. A visualization of ASCII load
flow output reports was created (Visual -WOLF) as an aid to operators in
assimilating the results of Auto-WOLF’s calculations quickly. WOLF and
Auto-WOLF are UNIX based systems with PC X-Windows user front ends.
Visual Auto-WOLF is a Windows Client / UNIX Server application. The
graphics display utilizes Scalable Vector Graphics (SVG).
Visual-WOLF presents WOLF real time load flow calculation results for
primary feeder sections and network transformers in a graphic environment.
Visual-WOLF helps visualize components at risk in next several events. It
places “halos” around banks used. It provides shortcuts that are beneficial for
the operators in the context of not having to review tabular data. The tool was
used by CERC but not the region during the Long Island City network event.
103
During contingency operations, WOLF reports are generated based on the
current load and forecasted peak loads given for the specific day. These loads
can be found on the Electric Distribution Information System (DIS). WOLF
generates two series of reports. One series is for the current load and the other
is based on the forecasted peak load. The exception reports are generated from
WOLF in order to see a summary of overloads on primary sections, network
transformers, unit station, secondary buses and any dropped loads. These
reports have been very effective in providing control room personnel fast
accurate information that can be used to remediate emergency conditions.
In Brooklyn/Queens, this exception report combined with the reports generated
from using Engineering Workstation provides the necessary information to
determine the ‘next worst feeder’ outage. It is noted in the WOLF methodology
that the user should not use the secondary mains portion of the report as the
model it is based upon is incomplete.
WOLF is also used to generate Overloaded Network Transformers by Element.
A detailed report is generated of all overloaded transformers on or ‘nearby’ the
target feeders. This report is used to identify and prioritize the overloaded
transformers.
WOLF is used again to generate an Overloaded Network Primary Sections
detailed report. The report identifies the overloaded primary sections for the
target feeders with ‘from’ and ‘to’ manhole IDs. This is used to determine the
overloaded primary sections.
A Next Worst Feeder from a Feeder perspective is then determined by
analyzing the ‘overloaded primary sections’ reports in WOLF. The next worst
feeder from a feeder perspective is the next feeder outage that would cause the
highest feeder overloads. The information is then provided in a report to
Incident Command.
Next, a Next Worst Feeder from a Transformer perspective is found by
analyzing the ‘overloaded transformer’ reports in WOLF. The next worst
feeder from a transformer perspective is the feeder outage that would cause the
highest transformer overloads. The information is then provided in a report to
Incident Command.
Brooklyn/Queens had developed its own contingency analysis methodology
designed for use during a summer month feeder contingency. It appears to have
104
been developed in 2004 and is a well thought out approach and is documented
in a report with complementary flow charts.
It describes the multiple feeder searches and WOLF analyses to be done in
parallel when triggered by a network contingency. These analyses were
intended to calculate the impacts of a feeder outage so that the next worst
scenarios and compromised system components can be provided to the regional
control center.
The Engineering Contingency Analysis Process is initiated when two or more
primary supply feeders are out of service in the same network. The steps taken
by the Engineering personnel are to provide the control center with the impact
on the ‘networks” current condition as well as the next worst feeder.
Part of the Brooklyn/Queens engineering contingency analysis methodology
involves a summer outage contingency rapid response. The summer outage
contingency rapid response, done correctly, as per Brooklyn/Queens analysis
documentation, should generate a quick and concise report of current network
status in response to a network contingency. An “as fast as possible response”
is characterized as within 10 to 20 minutes. It relies on a correct connectivity
model from PVL through WOLF. It also depends upon the availability of RMS.
Without WOLF or RMS, the report this process produces requires manual
calculations and reports from field crews on the transformer operating
conditions. In a fast moving event such as the Long Island City network event,
even with an accurate connectivity model, it is not clear the contingency
analysis reporting would be fast enough or accurate enough to be effective in
advanced multiple contingencies (i.e. a 6th contingency and beyond), but it
would certainly help the operators to assess the situation.
In the Long Island City event, a number of contingency reports were sent to the
Incident Commander. In addition, the Committee was advised that during the
event, the Incident Commander was verbally briefed on contingency analysis
reports as soon as they were available. E-mail follow-ups were later sent for
record purposes and reference. However, the e-mails were sent later than the 10
to 20 minutes referenced in the Brooklyn/Queens engineering documentation.
Data Presentation, Integration and Integrity
The Committee observed several issues around the presentation and integration
of operational data that is worth noting. One in particular deals with the
accuracy of displayed data and how it is represented. On two occasions while
the Committee was investigating the event, it encountered instances of NetRMS
105
showing differing data in the summary information window and in the
transformer specific information window of the application. Also, RMS
displays load data in chart or trace form that is sometimes not accurate. For
example, a particular transformer (in vault V 9426) continued to appear to be
loaded at approximately 45% when the load traces were observed in NetRMS.
However, this is after the time the transformer had been physically disconnected
from the feeder and live-end capped. When Brooklyn/Queens operational staff
was shown the traces, they assumed the transformer was closed and functioning
based on the load traces. The problem is that even a user very familiar with the
traces can be confused by the graphical representation of some data in NetRMS.
Another situation occurred when historical records were being sought for a
particular feeder in the Con Edison ECC Information Warehouse Feeder Detail
application. Two identical queries were run yielding two different sets of
historical actions.
Another observation deals with the non-uniform way symbolic data is
represented. The symbol used universally at Con Edison for a 1,000 kVA
transformer on feeder prints, M&S plates and planning maps is the exact symbol
used for a 500/560 kVA transformer in Con Edison’s Watchman application.
The screen indicates this in the legend, however since Watchman is a visual
display whose purpose is to provide the operator a quick snapshot of
transformer capability, it could easily mislead the observer. When the
Committee tested the premise the suspicion was confirmed.
Yet another issue was observed with ratings data. Ratings information for a
cable or transformer would, at times, be different in PVL or as displayed in
some of the reporting options in NetRMS, but no explanation of why different
ratings would be stored or displayed was found.
Applications such as PVL and WOLF report by exception to lessen the amount
of information an engineer or operator must review to determine an outcome or
conduct an analysis. While an aspect of this is certainly efficient, the lack of
presented data sometimes can mask a situation such as having other cables or
transformers at or near overload in a system. It also makes it more difficult for
an operator or engineer to develop a sense of the system that is being analyzed,
or to determine the quality/validity of the data being developed by the
application.
Attachment C includes a description of Computer Applications being used on a
limited basis at Con Edison that may be beneficial to consider for wider use.
106
Remote Monitoring System (RMS)
Overview
There are approximately 25,000 network transformers on the Con Edison
system. These transformers reduce the primary voltage to the 120 or 208 volts
used in most homes and businesses. The low voltage side of each transformer is
connected to the secondary grid through an automatic switch called a network
protector. Network protectors are designed to close when the feeder supplying
the associated transformer is energized and to open when the feeder is deenergized. Each transformer has a remote monitoring system (RMS) transmitter
that monitors the load on each of the three phases of the transformer and the
status of the network protector. The RMS transmitter sends the information in
near real time to a receiver in the substation, where the data is gathered and
forwarded to a central network data receiving system for the company as seen in
Figure 20.
SUBSTATION
RECEIVER
3
FRAME LINE
TX
PHONE
LINE
TX
1
TX
PICKUP COIL
5B21
TX
TX
2
4
VDAMS
USER
TX
USER
TX - TRANSMITTER
USER
Figure 20: Con Edison RMS System Overview (The source of this figure is Con Edison)
There are currently three design generations of RMS in use. The first
generation system was developed by Hazeltine and represents the highest
population. It is a tubular transmitter connected with an external wiring harness
and boot outside of the network protector housing. A second generation module
was designed and manufactured by BAE. It is a 'relay type' internal transmitter
that is installed inside the network protector housing. It transmits a higher
strength signal and is capable of more functionality. Examples of additional
functionality are the ability to provide voltage readings on all three phases.
Most recently, a third generation of modules has been manufactured by ETI.
These are also “relay type” transmitters. This is essentially the same as the
second generation with a little more functionality. A pressure sensor,
107
temperature gauge, and an oil level sensor has been incorporated to better
monitor the transformers and this module has become the new standard as
described in the programs listed below.
RMS uploads monitored data to VDAMS (Vax Data Acquisition Management
System), a dedicated computer system, which archives and presents the data to
end users directly and through additional applications such as NetRMS.
NetRMS is an Intranet-based system that provides network transformer status,
network protector status, and related details. In addition to current status
information, NetRMS provides various reporting and graphical tools that allow
users to view historical status and loading information about network
transformers and their nearby transformers.
The remote monitoring system for the Long Island City network is unique
because of the number of transformers. The network is comprised of 1194
network transformers fed by 22 distribution feeders. Since the number of
transmitters exceeds the design limit for the standard receiver; transmitters are
connected to different phases. This effectively increases the number of
transmitters that can function within the network. The Committee believes this
complexity was responsible for at least ten transformers not properly reporting
to the RMS system after being transferred from another feeder as part of
summer preparation work. Several of these transformers are located in the
vicinity of the initiating event on July 17th.
In January of 2006, Con Edison started deploying enhanced RMS modules
(known as the 3rd generation RMS modules) with more monitoring and
transmitting capability. Approximately, 765 sensors have been installed in
Queens. 552 of the sensors have temperature reading capabilities. At least 48
new modules have been installed in the Long Island City network.
The 3rd generation module deployment is part of a program called the Remote
Monitoring System Pressure Temperature and Oil Level (RMSPTO) program.
The RMSPTO program’s goal is to record and report measured pressure,
temperature, and oil level data from the underground transformers. The sensors
record real-time measured top oil temperatures of selected underground
transformers. The units report the data via the VDAMS system and the data is
displayed in VDAMS and NetRMS. This data is very useful especially during
extreme contingency operation and this work should be aggressively pursued.
This program is part of a larger program (Secondary Monitoring Program) to
help raise RMS reporting rates. Replacing 1st generation transmitters with 3rd
108
generation transmitters enhances reporting rates and allows analog oil
temperature and tank pressure data to be transmitted to the VDAMS system.
The 3rd generation (and 2nd generation) units have more output power than 1st
generation units and some locations that did not report consistently should yield
better reporting results. The addition of the sensors was to maximize the data
received by the system with the technology available today.
Overall, the reporting rate of the modules with temperature sensors is at
89.23%. Data capture began in the RMS system on January 11, 2006. In order
to verify the data coming in, contractors record visual data of the oil
temperature and compare the data with local computer readings. The
Committee understands that the approximately 48 new modules installed on the
Long Island City network were installed on assets of concern or perceived to be
of high risk and not with the priority of improving response rates.
Prior to the program, Con Edison identified “high risk” transformers by sending
crews to inspect those that were experiencing high loads. Also, old units or
certain transformer models were considered high risk. This method, using new
sensor and transmitter technology, is more accurate at identifying what is
actually going on inside a transformer.
For various reasons, the monthly availability of RMS in the Long Island City
network has ranged from 81% to 90% since 2002. For the first six months of
2006 its availability averaged 83%. Just before the event started, its availability
in the Long Island City network was 77%. RMS depends on individual
transmitters, installed at each of approximately 24,000 network transformers
across the system, to transmit via power line carrier certain information about
the current condition of the transformer. The signal is transmitted to a receiver
located at a remote location. Due to the harsh underground street environment
in which transmitters are required to work, their annual failure rate has been
about 6%. The ability of the receiver to effectively receive the signal from the
transmitters is also affected by the output of the transmitter as well as by signal
interference created in an urban environment. Fifty eight percent of the
transmitters are first generation transmitters and 38% are second generation
transmitters. Second generation transmitters have more output power and are
better able to propagate their signal from hard-to-report locations. Prior to this
event, Con Edison had already commenced deployment of a third generation
transmitter to improve Con Edison’s transformer information gathering ability.
RMS availability rates in Brooklyn/Queens began an annual decline in 2002 as
seen in Figure 21.
109
94
92
90
88
86
84
82
80
78
76
2006
2005
2004
2003
AU
G
U
SE
ST
PT
EM
BE
O
R
C
TO
BE
N
O
R
VE
M
BE
D
R
EC
EM
BE
R
E
N
JU
LY
JU
M
AY
2002
JA
N
U
AR
FE
Y
BR
U
AR
Y
M
AR
C
H
AP
R
IL
Reporting Rate
Brooklyn/Queens RMS5YearPerformance
Figure 21: Remote Monitoring System Availability
(The source of this figure is Con Edison)
As part of the summer preparedness, diagnostics were performed in March on
the feeder pick-up coils for the Long Island City network. Four were identified
as needing replacement to permit full signal sensitivity and resolve some of the
units not reporting on feeders 1Q02, 1Q11, 1Q20 and 1Q21. Replacement of
these coils requires an outage to the feeder. One of the feeder pickup coils was
replaced (1Q20) prior to the Long Island City network incident. The three
remaining coils were not replaced, although records indicate that these feeders
had been taken out for scheduled work at least once before July 17, 2006.
As part of its Long Island City event analysis, Con Edison attempted to examine
the replacement of individual transmitters, but could not find any records and
was told none existed. Index ID numbers are programmed into each transmitter
module by field personnel at the time of installation and are kept in an index file
database and in the static field database that coordinates with the VDAMS
computer system. These databases must match in format and data for the
information to be communicated properly between all systems. The databases
do match; however, a running tally was not kept of the ID number assignments
resulting in their inability to update the file.
On a broader scale, the Property Records group indicates that in at least 390
locations, new transmitters were used by Brooklyn/Queens region from January
to July of 2006. In addition, Brooklyn/Queens also received an unspecified
number of reconditioned RMS transmitters from Astoria. These were used to
address some of the approximately 2,400 units not reporting across the entire
110
region, but the allocation to the Long Island City network could not be
accurately determined.
The lack of available real time data from RMS was problematic before and
during the event. The fact that many transformers were not reporting actual
data made it difficult to determine if pockets of high load existed before the
event. When actual data does not exist, some applications estimate load data
and utilize the estimate in lieu of actual data. This is done without the explicit
knowledge of the operator of some of the engineering and planning applications
(WOLF, cable thermal models, etc.). In certain instances it can actually
complicate an analysis as the operator may think a result is based on actual
monitored data when the application has estimating data based on a subset of
other known data points, combined with historical data.
It is believed that below a certain level of availability of RMS data, estimation
of transformer load for non-reporting units becomes unreliable. The Committee
was not able to discern what the exact availability rate threshold is that causes
such a problem. Con Edison Information Resources (IR) estimates the usable
RMS availability requires a threshold at 85%. The Committee has been
informed that a study is being conducted to more precisely determine the
threshold value. IR built RME Estimator to compensate for RMS availability
issues by calculating estimates as described above. However, IR is concerned
about validity of results when the compensating systems need to provide a large
portion of estimated data.
In reviewing RMS availability historical data, trends can be observed. One
particular trend worth noting involves large networks with significant residential
components. These types of networks tend to rank higher (more of a concern)
in Jeopardy rankings, and RMS reporting for these types of networks typically
are lower. In some cases availability is below the 85%. Because of the
difficulty of quickly dropping load in an emergency in residential areas, as
opposed to large commercial customer load, it is more critical to have high
availability of RMS to understand such loading situations. Con Edison has
substantially increased RMS funding in the last several years to improve
reporting rates and increase functionality.
Limited actual RMS data combined with flawed connectivity models in PVL
have the potential to develop seriously flawed analyses when used in a critical
operating situation or for summer preparation activities and capital investment
option analysis.
111
The Committee believes that the state of the PVL model, combined with the
overall reporting rate in RMS, contributed to the secondary problems
encountered.
Conclusions
Non Standard Engineering Applications
• Brooklyn/Queens uses unique applications that it has developed which
have certain benefits to Brooklyn/Queens but may be problematic in
certain ways including:
o Engineering studies are not all archived for future referral
o Technical staff can not easily migrate across organization
boundaries without needing to learn a new set of applications
o Analysis of the region during emergencies is limited by the use
of non-standard applications
o Data quality controls, version controls, application testing, and
the economies of scale thereof are undermined from a Company
perspective
Poly Voltage Load Flow (PVL) Connectivity Model
• Errors are imbedded in the secondary connections represented in the
most current version of the PVL model. With improper connectivity,
the PVL model will not always accurately identify overloads for
tactical situations or summer peak preparation. Nor will it always
correctly identify electrical near-bys, converge as effectively, or
provide an accurate stream of data which is needed when managing a
fast moving event such as the Long Island City network event.
Several locations in the Long Island City network have secondary
mains and street ties with capacities less than their nearby
transformers. Overhead secondary mains are not included in the PVL
model and this can lead to incomplete and inaccurate underground
secondary mains loading
PVL Application
• User interface and report generation in PVL is cumbersome and at
times cryptic. This impacted the ease of use, the ease of interpreting
results and full understanding of the data calculated and presented.
112
Load Pocket Analysis
• The transformer in V 9426 failed on July 11th. Based on the practices
of Brooklyn/Queens Engineering, a Load Pocket Analysis was
performed at the time. The conducted analysis did not utilize the PVL
model; it utilized the Load Pocket Analyzer or Brooklyn/Queens
version of it, nor was any secondary analysis performed. A “near-by”
assessment was performed of local transformers and the decision that
the bank could be left off was made.
Data Presentation, Integration, and Integrity
• Data is not always consistent between certain applications nor are
naming conventions or engineering symbols. Those inconsistencies
could lead to errors and inefficiencies.
RMS Receiver Database
• When transformers associated with a feeder relief project were
electrically moved to another feeder to meet the demand forecast for
the summer of 2006, at least ten RMS transmitters did not report
properly after transfer. This was a result of the failure to update the
database index file.
RMS Availability
• At the start of the event on July 17th, RMS instantaneous availability
was approximately 77% compared to the monthly availability rate of
95% which is required by Con Edison specification.
RMS Related Record Keeping
• Part of the Long Island City RMS availability issues are attributed to
record keeping. As units have been installed and modified in the field,
the unique situation in the Long Island City network of having to
closely manage ID numbers, transmitter phase and index and
addressing assignments has become problematic for staff. Targeted
surveying of the installed base along with improved record keeping
will be needed to ensure ID issues do not prevent achieving the
desired availability rate.
113
Recommendations
Non Standard Engineering Applications
• Standardize engineering support applications and centralize quality
controls, version controls, application testing, application interfacing,
and new application development.
Poly Voltage Load Flow
• Establish specific criteria and develop a procedure that assures the
implementation of actions and ensures that network secondary
connections are accurately represented in the PVL model. Modify the
PVL connectivity model to include overhead secondary mains and
place customer loads appropriately.
• Complete the development of and deploy an enhanced PVL model or
the procurement of a commercial equivalent of an enhanced PVL. To
be most useful to today’s engineers, PVL should be more visual and
preferably linked to a Geographical Information System (GIS). It
should seamlessly integrate with other existing applications to
coordinate protection, forecast load growth and provide a variety of
analyses in a visual format.
• Investigate commercial software that exists today to see if they can be
applied to a network as large as Con Edison’s, that will allow one to
point and click at a section of a feeder to quickly reconfigure the
feeder and secondary connectivity and perform revised load flows.
PVL is not currently capable of doing this for secondary networks.
Secondary Mains and Street Ties
• Conduct a study in all networks to determine if the ratings of
transformers match the ratings of the secondary street ties and
secondary mains cables. Based upon the results, develop and
implement a plan to either reinforce the street ties and secondary
mains as needed or lower the rating on the transformer to match the
secondary mains and street ties.
Load Pocket Analysis
• Develop and implement a standard Company procedure to ensure that
“local load pockets” can be analyzed in real time when network
components such as feeders, cable sections, transformers and
secondary mains are out of service.
114
Exception-Based Analyses
• Develop criteria and new output reports for PVL and other analyses to
provide designer’s earlier warning, instead of just seeing “overloaded
conditions.” Create a threshold below the overload condition so that
designers and operators see components that are approaching overload
or a threshold of concern.
• Standardize mapping symbols utilized in the visual tools used by
operators and designers.
Data Presentation, Integration, and Integrity
• Develop and implement processes and standards to ensure that data
are consistent between engineering support applications, as well as
naming conventions and engineering symbols.
• Revisit the corporate data integration and presentation approach and
strategy. Confusion will be reduced and operational efficiency will be
improved.
RMS Receiver Database
• Require that testing of RMS reporting is completed after transformer
additions or when transformers or transmitters are electrically moved
from one feeder to another feeder.
• Ensure that the RMS database index file is appropriately updated.
RMS Availability
• Identify minimum RMS reporting requirements below which
operators will know that the information is suspect or not sufficiently
accurate to use for operating decisions. Institute proper management
controls and audits to ensure compliance with those requirements.
• Enforce present criteria for RMS reporting as specified in Con Edison
specifications or review the 95% reporting requirement. Test the
system to determine the appropriate level or determine other ways to
measure the effectiveness of the RMS reporting rate.
• Ensure networks with high transformer counts and large pockets of
residential load are prioritized accordingly in all efforts undertaken to
improve reporting rates.
• Suspect pickup coils should be replaced as soon as practical. With
restoration work underway, pickup coils should be replaced as needed
as feeder outages are scheduled. This work should be placed in the
feeder repository to ensure the DO knows that it is outstanding and
may combine it with planned work or when a feeder opens auto.
115
RMS Related Record Keeping
• Develop a new RMS ID and transmitter phase database, or update the
existing database, by conducting a field survey of units without proper
records. Further, embed ID, transmitter phase and status checking
into normal routines when visiting such equipment for maintenance,
inspection or other routine purposes.
116
North Queens Substation
Situation
The Long Island City network is supplied from a 27kV substation in the
northern section of Long Island City called North Queens Substation. The
North Queens Substation is supplied by five 138kV transmission feeders and
contains five transformers converting the voltage from 138kV to 27kV. The
transformers feed four separate bus sections that are connected through bus tie
breakers with a maximum of eight breaker positions on each bus. Four to six of
the breakers are used for network feeders while the others are used for nonnetwork load, capacitor banks or are spare positions. There are presently three
spare positions available.
The original breakers were “fixed position” in this vintage (1950) substation;
more modern substations contain rack out breakers. North Queens Substation
was in the process of converting from fixed to rack out breakers with 12 out of
22 positions completed.
The station equipment can be operated remotely from the System Operations
Control Center located in Manhattan, from a local controller at the NQS, or
automatically by protective relaying. There are red and green lights monitoring
the status of each circuit breakers trip coil and the close coil respectively. All
control of the breaker is supplied by a redundant DC battery source at the
substation.
Relay protection is designed to operate for faults on the feeder to which the
breaker is attached. The feeder relay system consists of three single-phase nondirectional over-current relays, one ground non-directional relay and where
needed one phase-balance relay plus its associated timer.
The over-current relays contain an instantaneous element and an inverse-time
element. The ground relay has only an inverse-time element. The phase balance
is a single three-phase relay that energizes a timer. The tripping output of all of
these relays, except the phase-balance relay, which trips through the timer, are
connected in parallel to energize the associated feeder circuit breaker trip coil.
The 27kV network feeder protection systems are designed to detect any faults at
the farthest end of the longest feeder while the substation is experiencing a
contingency of either a transformer or supply feeder. There is no protection for
an open phase fault.
117
Protective relays take corrective action in the shortest practical time with due
regard to selectivity, dependability and security. The feeder relays also provide
a partial back–up function for faults on the High Tension Customer equipment,
on unit substation secondary equipment, spot network secondary equipment and
any other connected load on a feeder.
The relay settings are generally not applied to prevent equipment or conductor
overload. Overload is intended to be prevented by proper planning, system
design and operation.
Should a feeder breaker fail to trip when a relay detects a fault condition the
minor bus section will be opened by bus overload relays, over-current relays set
to coordinate with the normal tripping time of feeder relays and breaker opening
time.
There is no sequence of events recording devices in the distribution substations.
There has been a program to install master-point Power Quality Node (PQ
Node) recording devices in the distribution substation in recent years that will
provide fault information. The PQ Node device had just been made fully
operational at the North Queens Substation on July 17th, the day of the Long
Island City event.
The Improper Operation of the Circuit Breaker 1Q21
Operations at the North Queens Substation during the Long Island City event
contributed to the severity of the event. The loss of the Bus Section 3S at 18:48
on July 17 was due to a fault on feeder 1Q21 which was a B phase to ground
fault that was detected by relays. However, the 1Q21 relay failed to trip the
feeder breaker resulting in a correct operation of the bus overload relay (B phase
target). This operation put the Long Island City network immediately into a
fifth contingency from a second contingency.
The mis-operation at the North Queens Substation is attributed to 1Q21 not
tripping for a B phase fault due to a cable failure. The breaker (34W) failure to
trip was caused by a misalignment of the secondary contact fingers on a
retrofitted rack-out breaker. This rack-out breaker replaced a fixed breaker as
part of a breaker replacement program undertaken to improve reliability at a
number of Con Edison substations.
The breaker (34W) failure to trip was caused by the failure of the secondary
movable contact fingers on the breaker to make solid contact with the secondary
118
stationary contacts on the breaker cubicle wall. This effectively opened the trip
circuit to feeder breaker 1Q21 (34W). A control wiring problem described
below prevented operators from discovering this problem.
The problem can be attributed to the installation of the replacement movable
contact assembly by a contractor without the necessary support bar in March of
2006. During installation, the support bar shown in Figure 22 was not installed.
Figure 22: Merlin Gerin secondary contact finger support assembly (Source is Con
Edison)
119
This resulted in the lower movable contact assembly having insufficient
pressure to make contact with the stationary contacts. This only impacted the
lower contacts as the support for the upper contact was provided by the breaker
frame.
Con Edison was in the process of improving the reliability of a series of circuit
breakers with the installation of the movable contacts. This improvement called
for the installation of half moon shaped contacts in lieu of fingers for the
movable contact assembly on each breaker. The half-moon contacts are, as the
name implies, semi-circular contacts that mount to the movable breaker. They
mount to a metal piece that has a guide with a tapered end. The guide catches a
track that is mounted to the cubicle and spring mechanism and allows the
vertical movement of the auxiliary contact block on the movable breaker to
move up or down so the guide travels through the track. The use of the track
and guide aligns the movable contacts on the breaker with the stationary
contacts of the cubicle. This improves the auxiliary contact consistency
between the breaker and the cubicle as shown in the following picture.
Figure 23: (The source of this figure is Con Edison)
Feeder breaker 1Q21 (34W) was originally installed in March of 2002 as part of
Con Edison’s replacement program for over-dutied breakers. The previous
breakers were fixed into position and the new breakers were of a “rack out”
design providing flexibility for maintenance and replacement. This also enabled
more feeder processing options through the use of ground and test devices.
120
After Con Edison diagnosed the secondary contact problems they began
replacing the secondary movable contacts. Feeder breaker 1Q21 had its
contacts replaced in March of 2006. Con Edison reports the breaker had
previously operated correctly for a fault on April 18, 2006 and returned to
operation on April 20, 2006 with no further operation until the July 17th misoperation.
Feeder breaker 1Q21 (34W) was removed from service on September 14, 2006
at which time the support bracket was installed.
The mis-alignment of the secondary contact on breaker 34W resulted in the
opening of the protective relay trip path. The following is the schematic of the
breaker wiring prior to the installation of the jumper shown between TB-24 and
TB-3, which was added during the event as a temporary fix to the wiring
problem. The contact on TB24 was open due to the contact misalignment
causing an open in the tripping contact circuit. Also there are two red lights
with separate current paths, one of which was off while the other was on
causing more uncertainty during the restoration effort.
The secondary contact shown in green (#24) was open therefore the protection
relay contacts shown in the circles did not have a trip path. It should also be
noted that 1Q81 breaker 34E also did not trip for this event due to the same
contact mis-alignment issue.
121
Figure 24: North Queens 27kV Retrofitted M&G SF-6 Breaker Trip Circuit (1Q21) (The
source of this figure is Con Edison)
Another issue with this control circuit design is that the Red light circuit which
is normally in parallel with the tripping circuit is not. Operators use the light as
an indication that there is a trip path for the relays, here there is none. In this
case, the red light circuit is not in parallel with the relay tripping contacts but
has a separate path that utilizes separate contact figures. As installed it does not
monitor the tripping circuit and trip coil, it provided misleading information to
the operator indicating that everything was in order should a trip be initiated by
a protective relay.
Field modifications were made at the time of the event when it was realized that
the proper trip circuit did not exist.
The following is the permanent wiring design to correct for contact
misalignment. This insures monitoring of the relay tripping path for all
protective relays and was completed on September 14, 2006.
122
Figure 25: North Queens 27kV Retrofitted M&G SF-6 Modified Breaker Trip Circuit
(1Q21) (The source of this figure is Con Edison)
Relay Performance
At Con Edison, circuit breaker relay calibrations and trip tests are to be
performed on 27kV feeders every six years. At the North Queens Substation,
relay calibrations have been completed on 24 of the 31 breaker positions. The
Committee has been advised that all North Queens Substation relay calibrations
and tests will be on schedule with the next calibration to occur in 2007 and
2008. Trip tests are on schedule for all North Queens Substation busses.
A PQ Node is a high data-rate recording device intended to be utilized to
capture waveforms to analyze the quality of power being delivered. Recently,
Con Edison has installed PQ Nodes in various substations including North
Queens Substation as part of a research project.
The PQ Node is connected so that it monitors the current from one of the banks
and is also connected to the station bus to monitor voltage. As such, it “sees”
123
approximately a quarter of the station load under normal circumstances. The
PQ Node is connected to enable observation of phase to phase and phase to
ground faults. These are the most common faults and the PQ Node should be
able to capture the specifics of such a fault as connected. In the search for the
fault after the event, voltage is applied to all three phases.
During the Long Island City network event there were seven occasions where a
feeder breaker was closed in an attempt to restore the feeder to service and they
immediately tripped out. The operation is referred to as a cut-in open- auto
(CIOA) operation. There were four confirmed instances during the Long Island
City network event during which “cut-in open-auto” can be attributed to inrush
currents caused by the magnetizing current of the transformers connected to the
feeders and the levels seen were above the settings of the Phase Instantaneous
relays. These conditions were confirmed by a review of the PQ Node data for
those operations post event
There was also a test performed after the event in which a recording
oscillagraph was connected into the current circuits at North Queens Substation
when feeder breaker 1Q02 was being returned to service on August 22, 2006
that confirmed that the magnitude of transformer magnetizing current was
above the Phase Instantaneous settings of the relays.
The Instantaneous Phase relay settings on the feeder breakers were all set at
4000 ampere pickup. This is the settings for most of the feeders on the system,
the only exception are for some feeders in Manhattan with short feeder lengths.
As stated previously, the relay settings are developed to detect all types of faults
at the farthest end of the longest feeder. There is no protection for open phase
faults. The relay settings are designed to open the breaker in the shortest
practical time while still coordinating with other protective relays on the feeder.
The relays also provide backup protection for connected loads on the system
such as High Tension customers, unit substations and their connected loads on
the feeder. There is a slight margin to compensate for inaccuracies in current
transformers, relay calibration etc.
There are three phase relays that have instantaneous over-current elements and
time over-current elements and a ground relay with time over current element
only. The feeder relay package also contains a phase unbalance relay that
protects for unbalanced faults at the multi-bank or unit substation transformer or
secondary 4KV bus and as backup for unbalanced faults on the secondary 4KV
system.
124
The phase instantaneous relays are set to be the expected tripping relay for all
faults. There maybe some high impedance faults that will trip with time delay
but the majority of the faults are expected to clear instantaneously to; reduce the
energy released at the fault location, reduce the time that equipment and cable
experience high fault currents and to minimize power quality impact on the
customers.
There were four confirmed incorrect relay operations on four different feeders
during the Long Island City event.
•
•
•
•
1Q07 on July 17th at 19:09.
1Q18 on July 18th at 23:55.
1Q17 on July 19th at 08:49.
1Q14 on July 23rd at 19:47.
This feeder connects to 64 transformers.
This feeder connects to 50 transformers.
This feeder connects to 52 transformers.
This feeder connects to 47 transformers.
The operation of 1Q07 occurred on July 17 at 19:09 shortly after Bus Section
3S was returned to service and individual feeder restoration began. This
operation added complexity to the situation and confusion at a point when the
network was trying to recover from a 5th contingency. Feeder 1Q07 was not
restored for another 16 hours. The magnitude of the inrush currents exceeded
the instantaneous phase over-current relay operation points.
To review this issue in greater detail and using the feeder operations of feeder
1Q14 on July 23rd at 11:56 and at 19:47 are examples that reveal the following:
Feeder 1Q14 was manually opened on July 23rd in order to repair various liveend capped transformers from the previous outage. When attempting to return
the feeder to service on July 23rd a cut-in open- auto occurred due to the
magnitude of inrush current. The PQ Node captured the operation and
engineers calculated the inrush current to be approximately 6,000 amperes.
There was no conclusive evidence of a fault on the North Queens PQ Node
chart. Relay operation appears to have been caused by high transformer
magnetizing inrush after feeder energization and possibly some cold load pickup.
Con Edison has revised the inrush current setting to a higher set point which
would prevent the relay operation for inrush currents. It should be noted that
the Long Island City network has the highest number of transformers per feeder.
125
Relay Targets
The instantaneous relay targets are electro-mechanical and have a history in the
industry of not providing an indication when the relay operates due to their
design and reset function.
The majority of the relay operations were phase instantaneous operations as
expected, however, for the following events there were no relay targets. The
relay targets should be tested to insure functionality.
• 1Q01 on July 17th at 21:56
• 1Q07 on July 20th at 13:39
However, the PQ Node data indicated that the proper instantaneous tripping
occurred in each event. The PQ Node device, as it is fine-tuned in the North
Queens substation, should be utilized as an additional source of information for
the station operator and district operator as it provides a confirmation of both
the existence of fault current and the fault type.
Breaker Operations
Aside from the failures of breakers 1Q21 and 1Q81 all other breakers operated
to clear all faults during the Long Island City event. There were two breaker
operations in which the clearing times were longer that what should have
occurred. The following breakers should be inspected to insure that the breaker
is functioning properly.
• 1Q20 on July 18th at 05:52 for a 6,600 ampere fault
• 1Q19 on July 21st at 17:26 for a 16,000 ampere fault
Conclusions
Secondary Contact Fingers
• Circuit breaker 1Q21 secondary contact fingers were misaligned
preventing protective relays from tripping the breaker of a feeder
fault.
Red Light Circuit Wiring
• The red light circuit did not monitor the tripping circuit and trip coil,
and it provided misleading information to the operator.
126
In Rush Settings on other Networks
• In addition to the Long Island City network, there are other networks
in the system that have large numbers of transformers connected
including networks in Queens, Brooklyn, Bronx, Westchester (Granite
Hill). Those networks may also have potential problems with
instantaneous relay operations causing cut-in open-auto operations
due to inrush current.
Electro-mechanical Relay Speed
• If Con Edison adjusts electro-mechanical relay settings to prevent inrush current from transformer magnetization, the resulting fault
clearing times in some cases may be too long to protect public safety
and equipment.
Relay Protection Engineering Approvals
• Feeder relay settings are not subject to periodic review as other relay
systems. The Central Engineering Section does not routinely receive
notifications of all changes that occur to feeders. They do receive
information when a unit substation, auto loop, grounding transformer,
and/or shunt reactors are added to a feeder. The annual relief and
reliability program which moves and adds transformers to feeders
does not receive review. Based on the Long Island City network
event, the change to network feeders should include a review of relay
settings. It should be noted that the issue is not as simple as
increasing the relay trip settings based on added transformer capacity.
Even if such a review of relay settings would have been done, it may
not have resulted in settings that would have addressed other factors
such as the level of harmonics, the point on the sine curve at which the
breaker closes, when the transformers were de-energized, etc. If this
would have occurred as the changes to the network occurred, then the
incorrect relay operations during the Long Island City event may have
been avoided. A notification system should be established or a
procedure issued that provides the parameters for when relay settings
require review.
Electro-mechanical Relay Targets
• Electro-mechanical targets have a history in the industry of not
operating consistently. The failure to operate impacted restoration
efforts during the Long Island City event as 1Q01on July17 at 21:56
was closed 7 minutes after the open automatic and resulted in a cut-in
open-automatic because of no target being found and the assumption
127
that the feeder would return to service. There also were no targets for
the 1Q07 operation at 13:37 on July 20th, however, in this occasion the
feeder was not returned to service. The relay targets are not always a
reliable source for determining the existence of a fault. If operating
practice permits closing a breaker when there is no positive indication
of a fault (i.e. voltage dip, report of a manhole incident etc.) existing
aside from the breaker opening another source of information such as
the PQ Node data should be utilized by the Station Operator and
District Operator prior to re-closing the feeder breaker.
1Q20 and 1Q21 Clearing Time
• Circuit breakers 1Q20 and 1Q19 had long clearing times observed on
July 18th and July 21st for 1Q20 and 1Q19 respectively.
Recommendations
Secondary Contact Fingers
• Evaluate the effectiveness of the replacement contact arrangement
being installed on circuit breakers similar to 1Q21 and if that
arrangement is found to be an optimal solution to the misalignment
issue, review the criteria currently in use that determines at what point
the replacement is installed. Adjust the criteria as necessary to
prevent a future occurrence similar to the Bus Section 3S trip that
occurred during the Long Island City network event.
Red Light Circuit Wiring
• Expedite the modification of this wiring scheme on all circuit breakers
in which the scheme is similar to that of 1Q21 where the Merlin Gerin
Model SF2 breakers were used to retrofit the cubicle.
• Establish a review process that includes the Relay Protection Section
for all modifications and alterations of the relay protection system.
In Rush Settings on other Networks
• Review all feeders on the Con Edison system to determine the total
MVA of the transformers connected and establish a procedure for
changing the settings based on new transformers or equipment being
added or transferred.
• Establish an annual review of the number of transformers per feeder
and perform periodic reviews of feeder relay settings.
128
Electro-mechanical Relay Speed
• Study the impact of raising the instantaneous current relay protection
setting to determine if tripping times adequately protect the equipment
and the public. If not, consider replacing electro-mechanical relays
with micro-processor relays whose settings can distinguish between
inrush and fault current to maintain fast clearing times.
Relay Protection Engineering Approvals
• Develop and implement a procedure that requires all operating regions
to notify the Relay Protection Engineering Section of specific changes
to the number or size of transformers in a timely manner. Require the
Relay Protection Engineering Section to take actions to ensure the
relay setting adequately considers inrush current from magnetization
such that “no fault” cut-in open-autos are prevented.
• Investigate if a procedure that requires periodic testing or calculation
of inrush current from magnetization is required and assure that relay
settings are coordinated accordingly.
Electro-mechanical Relay Targets
• Review and modify as appropriate the Con Edison maintenance
procedure that should have prevented the failure of the electromechanical relay targets on 1Q01 on July 17th and 1Q07 on July 20th.
• Develop a method for the PQ Node equipment to provide a station
alarm when fault current is detected, thus providing the District
Operator and Station Operator with a confirmation that a fault existed
even when a relay target is not present.
Breaker Clearing Time
• Inspect and test feeder circuit breakers 1Q20 and 1Q19 (including
breaker timing tests) to correct the apparent long clearing times
observed on July 18th and July 21st.
129
Feeder Processing
General
This section reviews each feeder that opened automatically during the Long
Island City event. The analysis reviews the key decisions that were made and
the impact that those decisions had on the feeders’ restoration times, how they
affected other restoration efforts, the impact of damage and the effect on
contingency levels. The ability of the substation to process feeders and their
restoration will be quantified. The analysis also focuses attention on the
impacts that feeder restoration decisions had on the secondary system. The
analysis is based on the Committee’s review of data gathered from employee
interviews, the Con Edison Rapid Restore System, District Operating logs,
feeder maps, secondary planning maps, M&S Plates, PQ Node data, and event
summary reports and e-mails.
This section analyzes 17 distinct operations that required field repair work. An
event is defined as a feeder opening automatic. In the majority of cases,
involved damaged apparatus that had to be repaired or isolated from the feeder
before the feeder could be restored to service. The duration of the event
commenced at the initial automatic feeder operation and ended when the feeder
was successfully restored to service. It included delays and any test failures or
repairs that did not successfully result in the restoration of the feeder. The
description of these 17 events addresses the critical feeder processing activities
until the network returned to a state of having no feeders out of service on the
morning of July 21st.
Definitions
The Committee recognizes that the use of jargon can be confusing and make
following the sequence difficult. However, in this section it is almost
unavoidable. To aid the reader, a list of definitions is provided as Attachment
E. The table of terms will be utilized throughout the report and in particular this
section as they are related and utilized in the processing of feeders during the
restoration effort.
Events
Event # 1: July 17th at 15:50 Feeder 1Q17
Fault locating and standard operating procedures were being followed at the
beginning of the Long Island City network event until 21:57 at which time a
130
Section 9 was invoked due to the network reaching a 5th Contingency at 18:47.
Because of the fire in the connecting underground conduits to manhole M
11711, it was determined restoration would be expedited by installing a live-end
cap to isolate the cable and temporarily remove the faulted cable sections. In
addition to clearing the damaged section of cable, this action also removed 2
network transformers on feeder 1Q17 from service that were connected east of
the faulted cable section. The Section 9 was in effect for 29 minutes and had no
impact on other work. The ammeter clear test was also waived to expedite the
return to service of the feeder.
Prior to this event there were 25 transformers out of service. An additional two
transformers from feeder 1Q17 were removed during this event. During the
period from 15:50 to 16:22 when 1Q17 opened there were approximately 75
transformers out of service in the Long Island City network.
This
approximation takes into account banks-off, feeders out, and transformers liveend capped. It does not account for open network protectors or blown limiters.
Event # 2: July 17th at 16:22 Feeder 1Q16
The opening at 16:22 of 1Q16 created a second contingency and was caused by
the same secondary cable fire that damaged 1Q17 in the conduit run connected
to manhole M 11711. The invoking of the Rule Book, Section 9 was
appropriate as the work was in the same area as 1Q17 and overlapped the
existing Section 9. However, the strategy of clearing the damaged section of
cable by utilizing a live-end cap, as had been done with 1Q17, triggered an
analysis of dropping additional transformers in the immediate area and the
effect on the secondary system. The Committee was advised that the initial plan
was to install a primary shunt on 1Q16 and not drop off any of its associated
transformers. However, later in the evening the bus section tripped deenergizing three additional network feeders and 159 transformers. Based on
this trip-out, it was decided to live-end cap 1Q16, thereby restoring 58
transformers on 1Q16 and temporarily dropping off three transformers.
The Section 9, under which 1Q16 restoration was operating, did add a 1 hour
and 56 minute delay to the restoration efforts on 1Q07 by delaying the
“establishing a condition” phase. There also was an alive on back feed
condition (ABF) on feeder 1Q16 requiring a 40 minute period before beginning
the processing of this feeder. While the period of time is referred to as a delay
the back feed must be cleared before the substation personnel can proceed to
process the feeder.
131
Once the live-end cap was completed, an attempt to close the breaker for 1Q16
was made. However, the first of thirteen cut-in open-auto conditions occurred.
This particular condition was caused by a failed Raychem 3W-1W (three-way to
one-way) joint in manhole M 1188. This type of joint has been identified as a
heat sensitive joint that has a higher failure rate in hot weather. Expedited
procedures were utilized in this restoration effort, and as permitted under
Section 9, the ammeter clear test was waived to expedite feeder restoration.
The use of live-end caps while expediting restoration of the feeder 1Q16 did add
additional load on the transformers and secondary in the area by removing an
additional 3 transformers on 1Q16. While the greater network contingency
would have been reduced from a second to a first, a local second contingency
would still have existed and due to other transformers that had been dropped
prior to July 16, the local condition would not have been improved. This
coupled with the fact that the initial fault was caused by a burning secondary
around manhole M 11711 and one block to the south provided a strong
indication that the secondary in the immediate area needed attention.
Note: From 16:22 through 18:47 on July 17th, there were approximately 136
transformers removed from service due to 1Q16 and 1Q17 being out of service.
Event # 3: July 17th at 18:47 Feeder 1Q21
The failure of feeder 1Q21 was identified as a cable failure supplying TM 804.
The feeder breaker at the North Queens Substation failed to open and clear the
fault. The problem with the breaker failure was determined to be a DC control
wiring issue as discussed in the North Queens Substation section of this report.
There was an unsuccessful attempt to reenergize the feeder approximately 15
minutes after the initial open automatic operation. To expedite feeder 1Q21’s
restoration, a Section 9 was invoked to disconnect the faulted cable by installing
live-end caps in manhole M 2820, dropping TM 804. The Section 9 was in
effect for 3 hours and 39 minutes and negatively impacted the restoration of
feeder 1Q07. An attempt to restore feeder 1Q21 to service failed with a cut-in
open-auto operation at 2:49 on July 18th. The restoration process was further
delayed for 6 hours due to the limitations with the test bus design at North
Queens Substation which permits the concurrent processing of only one feeder
per bus section. During this period of time feeders 1Q07 and 1Q01 fault
restoration was underway with conditions being established on both feeders.
Twelve of twenty two circuit breaker positions at North Queens Substation have
been retrofitted with “rack out” type circuit breakers including feeder breakers
1Q21, 1Q07 and 1Q01. These positions have the capability to use ground and
132
test devices to provide an alternate means of feeder restoration. However, when
the circuit breakers were put back into service, the Ground and Test Devices
(G&T) were not available and could not be certified for use. Therefore they
were not available during the Long Island City network event. Had they been
available, they may have reduced the processing time at North Queens
Substation and provided the ability for more concurrent processing than was
done during the incident.
The fault on feeder 1Q21 was located and identified as an Elastimold 2W-1W
(two-way to one-way) paper to poly joint. This type of joint has a history of
being heat sensitive and failing in the type of weather that was being
experienced. The fault was located in manhole M 14669 on the cable supplying
transformer VS 7987. To expedite the return to service of feeder 1Q21, it was
decided to utilize a “Known Point Splice” (KPS) at manhole M 10705 to isolate
the fault in manhole M 14669. The decision was made to use the KPS to save
time and eliminate steps in the feeder processing sequence. The actual field
work was completed in one hour. However, using the KPS in addition to
isolating transformer VS7987 also meant that nine additional transformers
would remain out of service on a cable spur when feeder 1Q21 was restored at
the North Queens Substation. While the feeder would be restored in the greater
network it leaves the local area vulnerable.
An attempt was made to close the 1Q21 breaker at 17:11, resulting in a cut in
open auto condition. A fire was reported in manhole M 2554 which is on the
route of this feeder. This fire caused by burning secondary cables, also caused
damage to an Elastimold 2W-1W X-E joint and other cables in the manhole.
The damage required the replacement of three sections of cable from manhole
M 2554 in three different directions and manholes. The cable replacement and
splicing required 12 hours to complete. Since the feeder 1Q21 was never
returned to service from the initial operation, it is difficult to be sure of the exact
sequence described for the identification of the faults. However, based on the
known facts the Committee believes the sequence describes what actually
occurred.
Note: During the time from 18:47 through 23:21 on July 17th, there were a
maximum of 293 transformers removed from service due to 1Q16, 1Q17, 1Q21,
1Q07, and 1Q15 being out of service, including banks off just prior to the event.
Event # 4: July 17th at 19:09 Feeder 1Q07
The de-energizing of feeder 1Q07 was caused by the clearing operation of Bus
Section 3S operating due to a fault on 1Q21 at 18:47. Since the feeder breaker
133
on 1Q21 did not clear, the fault was properly isolated by the back up relay
which caused the bus section to trip. The 1Q07 feeder breaker was closed at
19:09 on July 17th and a cut in open auto operation occurred. Examination of
the PQ Node data indicated that this was caused by an inrush current condition
and lack of coordination with the instantaneous over-current relay settings.
These settings were low enough to cause the relay to operate on the inrush
currents due to the magnetizing current of the large number of network
transformers (62 transformers at the time) on this feeder.
Standard operating procedures were followed including “establishing a
condition”, which includes the use of a higher voltage (60kV) to locate the fault.
An Elastimold (2W-2W) joint was found damaged after the test.
The restoration effort was delayed on two different occasions for a total of 4
hours and 41 minutes, preventing the “establishing of a condition” phase due to
Section 9s being called on feeders 1Q21, 1Q16 and 1Q17. Waiving of the AC
test to expedite restoration was prudent considering the network was still in a 6th
Contingency at 11:36 on July 18.
Event # 5: July 17th at 21:43 Feeder 1Q20
Feeder 1Q20 opened auto on July 17th at 21:42 with 8,996 amps of fault current
and an A phase target. The breaker was closed at 21:55 and an A phase fault of
6,912 amperes was recorded by the PQ Node and calculated by Con Edison
Distribution Engineering. This put the network back into a fifth contingency.
The breaker was closed without establishing a condition because of the elevated
contingency level. The closure at 21:55 resulted in a cut in open auto and it was
then determined that transformer V 7813 had failed. The transformer was
isolated by installation of live-end caps in manhole M 2584. The ammeter clear
test was waived. There was an attempt to close the breaker in at 00:28 on July
18th. This was unsuccessful and another cut-in open-auto resulted. The fault
was located utilizing a capacitance discharge of 20KV for 6 seconds as a test.
However, there was an 8 hour and 46 minute delay due to test bus conflicts in
processing the feeder for work. The fault was identified to be a Raychem 3W1W P-X joint in manhole M 1699. The work was delayed due to a conflict with
crews working on feeder 1Q02 for 54 minutes. The feeder was repaired and
was returned to service at 13:09 on July 19th.
Note: From 21:43 through 21:49 on July 17th, there were approximately 302
transformers removed from service due to feeders 1Q16, 1Q17, 1Q20, 1Q21,
and 1Q07 being out of service.
134
Event # 6: July 17th at 21:49 Feeder 1Q01
Feeder 1Q01 opened automatically on July 17th at 21:49 with 4140 amperes of
fault current and a C phase target. The feeder breaker was re-closed at 21:56
with no targets reported but experienced an A&C phase fault of 6,720 amperes
as calculated by Con Edison Distribution Engineering. This put the network
into a sixth Contingency. It is not apparent why the breaker would have been
closed without locating a fault, as relay targets gave indications that there was a
fault. The re-closure at 21:56 resulted in a cut-in open-auto. Feeder processing
was conducted from the pothead cubicle using standard operating procedures
and located the fault as the transformer in VS 477. Defective cable was also
found in manhole M 820 which also contains 1Q02, which is damaged and fails
in M 820 the following day, July 18th at 08:23. The fact that both faults are in a
common manhole delays 1Q02 restoration. A caution on 1Q01 is applied at
15:36 on July 18th.
The failed transformer in VS 477 was isolated and cleared by installing live-end
caps in manhole M 225. The damaged cable in manhole M 820 was also
repaired. The ammeter clear test was waived and there were delays
encountered, a 14 minute delay due to equipment conflict, a 36 minute delay
due to defective equipment and a 7 minute operational conflict before the
breaker was closed at 20:53 on July 18th.
Note: From 21:49 through 23:21 on July 17th there were approximately 355
transformers removed from service due to feeders 1Q01, 1Q16, 1Q17, 1Q20,
1Q21, and 1Q07 being out of service.
Event # 7: July 18th at 08:23 Feeder 1Q02
The 1Q02 feeder breaker originally opened at 19:48 on July 17th. A C phase
fault of 10,432 amperes was recorded on the PQ Node located in the North
Queens Substation and calculated by Con Edison Distribution Engineering. The
feeder remained alive on back feed and a successful close in of the 1Q02 feeder
breaker was performed 20 minutes later. The feeder remained in service until
08:23 on July 18th when a C phase fault with 10,224 amperes was recorded and
calculated by Con Edison Distribution Engineering. Two faults were identified
a cable failure in manhole M 820 mentioned above and a 2W-1W X-E joint
failure in manhole M 14503. The joint was damaged due to a secondary
burnout. The burnout caused external damage to both the cable and the joint.
Again the feeder remained alive on back feed causing a delay of 2 hours and 11
minutes to process. A caution was invoked due to the fault being in a common
manhole (M 820) with 1Q01. The Section 9 on feeder 1Q17 did not negatively
impact restoration. A live-end cap was installed in manhole M 820 to drop the
135
damaged section of cable which also dropped TM 6389. The joint in manhole
M 14503 was also repaired and an attempt to close the 1Q02 feeder breaker was
made. However, it resulted in a cut-in open-auto at18:54 on July 18th. Targets
indicating a B phase fault, and later confirmed by PQ Node data, showed a B
phase fault of 5,472 amperes. In M 1889 another failed (Raychem 3W-1W P-E)
joint was found. The feeder was processed from the pothead compartment
utilizing the capacitance discharge of 20KV for 6 seconds. A delay of 2 hours
and 31 minutes was encountered due to Section 9s on feeders 1Q16 &17. There
were also test bus conflicts at North Queens Substation resulting in an
additional delay of 1hour and 58 minutes. The feeder was returned to service at
19:04 on July 19th.
Note: From 08:23 through 09:33 on July 18th there were approximately 357
transformers removed from service due to feeders 1Q01, 1Q02, 1Q07, 1Q16,
1Q20, and 1Q21 being out of service.
Event # 8: July 18th at 11:54 Feeder 1Q17
Feeder 1Q17 opened automatically on July 18th at 11:54, and 4,400 amperes of
fault current was recorded at the PQ Node and calculated by Con Edison
Engineering and B&C phase relay targets were reported.
A Section 9 was invoked and live-end caps were installed in manhole M 8407 to
drop VS 6300 and HTV 9769. There were delays of 1hour and 26 minutes to
restoration due to test bus conflict and caution placed on the feeder due to a
manhole fire on the run of the feeder. The feeder was processed using the
capacitance discharge from the pothead compartment to expedite restoration and
work around the test bus conflict. An attempt to close the feeder breaker on
1Q17 at 20:52 was not successful and the feeder cut-in open-auto.
A three phase fault of 11,868 amperes was recorded on the PQ Node and
calculated by Con Edison Distribution Engineering, with all three phases
recording relays targeting. This fault was found to be the transformer in VS
7995. The transformer had lost dielectric fluid and the high voltage components
on all three phases in the transformer arced to ground. The transformer was
isolated by installing live-end caps in VS 7995. The transformer would later be
changed out on an isolated feeder hold-off. A 49 minute delay was caused by a
Section 9.
An attempt to close the feeder breaker on 1Q17 was unsuccessful at 08:49 on
July 19th when another cut-in open-auto occurred. Later analysis of the PQ
Node data indicates there was no fault current and a review of the PQ Node data
136
(post event) concluded that the cut-in open-auto operation was caused by higher
inrush current than the relay settings accounted for.
A modified hi-pot was performed and an Elastimold 2W-1W P-E was identified
as the failure in manhole M 11730 and isolated TM 810 and a cable section.
The feeder was returned to service at 20:41 on July 19th.
Event # 9: July 18th at 15:14 Feeder 1Q18
Feeder 1Q18 sustained a three phase fault of 9,148 ampere magnitude as
recorded by the PQ Node and calculated by Con Edison Distribution
Engineering, which was cleared by B phase relaying. The fault was caused by
the failure of the transformer in VS 9819. To expedite restoration, a Section 9
was utilized at 20:19 and the transformer was isolated from the main feeder at
21:25 and the 1Q18 feeder breaker was restored to service at 21:46. There were
no other conflicts on the restoration. The Long Island City network returned to
an sixth contingency. With this event, approximately 344 transformers or 28%
of the transformers were out of service from 15:14 to 20:05 on July 18th.
The Feeder remained in service for only four minutes and opened automatically
at 21:50 due to a C phase fault of 5,290 ampere magnitude recorded by the PQ
Node and calculated by Con Edison Distribution Engineering with C phase
instantaneous over-current and ground time over-current relay targets. The
feeder remained alive on back feed for two hours and 52 minutes and the feeder
breaker was closed at 23:57 on July 18th with a cut-in open-auto occurring.
Review of the PQ Node data post event indicated that the cut-in open-auto was
the result of lack of coordination with the magnitude of actual inrush currents.
There was no conclusive evidence of fault current although there was a Ground
relay target. The feeder again remained alive on back feed which was cleared
11:06 minutes later. Standard restoration procedures were then followed
incurring a 2hour and 7 minute delay in processing. The fault was identified as
the transformer in vault VS 8807. The fault was isolated by installation of a
live-end cap to disconnect the transformer from the main run of the feeder.
Feeder 1Q18 was returned to service at 00:46 on July 20th.
Event # 10: July 18th at 20:32 Feeder 1Q12
Feeder 1Q12 sustained a three phase fault of 11,620 ampere magnitude, as
recorded by the substation PQ Node and calculated by Con Edison Distribution
Engineering, that was cleared by A and C phase relaying. The fault was
identified as the failure of the transformer in TM 838. The fault was isolated by
installing a live-end cap in manhole M 2539. This disconnected one section of
primary cable and the failed transformer in TM 838. The transformer was
137
overheated due to the level of contingency being experienced in the Long Island
City network. A delay was encountered due to the hi-pot test equipment being
defective. This caused a 1 hour and 36 minute delay in the restoration effort. A
modified hi-pot was utilized after the repairs. However, the feeder failed on test
at 11:15 on July 19, 2006. The feeder was re-tested with a modified hi-pot at
12:11 on July 19, 2006 and passed at 12:38 on July 19, 2006 and returned to
service. At this point the network was in the 8th Contingency with
approximately 441 or 37% of the transformers were out of service.
Event # 11: July 18th at 20:33 Feeder 1Q15
Feeder 1Q15 sustained an A and B phase fault of 9,128 ampere magnitude, as
recorded by the substation PQ Node and calculated by Con Edison Distribution
Engineering, which was cleared by B phase relaying. At this point in time the
Long Island City network was in a 9th contingency with approximately 480 or
40% of the transformers were out of service.
A caution was placed on the feeder due to a fire on the feeder run and a caution
was also called due to crews doing restoration work on 1Q02. These caused a 2
hour and14 minute delay in restoration activities. The feeder fault was
identified as the failure of the transformer in VS 339. The transformer failed
when internal pressure caused the radiators to leak dielectric fluid and an
internal flashover occurred. The feeder failed a modified hi-pot at 16:06 on July
19th. There was a 1 hour and 15 minute delay in processing due to conflicts on
the test bus. Live-end caps were used to drop Riker Island on July 20th at 11:35
and the live-end caps that were installed to isolate VS 339 were re-made. The
feeder 1Q15 was restored to service at 13:48 on July 20th.
Event # 12: July 18th at 20:38 Feeder 1Q16
Feeder 1Q16 sustained a three phase fault of 11,698 ampere magnitude, as
recorded on the substation PQ Node and calculated by Con Edison Distribution
Engineering, which was cleared by A, B and C phase relaying. At this point in
time the network was in a 10th Contingency with approximately 538 or 45% of
the transformers were out of service from 20:38 through 20:53 on July 18th. At
21:49 an attempt to close the Feeder breaker on 1Q16 was made and a cut-in
open-auto occurred as recorded on the PQ Node and in the System Operations
data. There was a three phase fault of 13,830 amperes magnitude that was
cleared by A, B and C phase relays. The fault was identified as the failure of
the transformer in VS 479. This transformer was very highly loaded and the
internal pressure in the transformer caused a rupture in the radiator resulting in a
loss of dielectric fluid leading to a flashover. A cable failure in manhole M
8405 was also located, which was caused as a result of a secondary burnout.
138
The transformer was isolated as well as a section of cable by installing live-end
caps in manhole M 16182. The feeder was processed from the pothead cubicle
at North Queens Substation to expedite feeder restoration. A 44 minute delay
was incurred due to equipment issues at the substation. The feeder was tested
with a modified hi-pot and failed the test four times during the restoration
process. These occurred at 19:45 and at 23:49 on July 19th, and at 8:12 and
19:43 on July 20th. Delays encountered due to test bus conflicts totaled 3 hours
and 12 minutes.
A faulted section of cable and the transformer in TM 6285 were removed from
the main run of feeder 1Q16 by installing a live-end cap at 3:45 on July 20th.
The transformer in TM 6531 and the section of cable in manhole M 5550 were
also removed from the main run of feeder 1Q16 at 17:25 on July 20th by the
installation of live end caps. The feeder passed a modified hi-pot of 30KV for 5
minutes at 7:00 on July 21st and the feeder breaker for 1Q16 was cut in at 07:49.
Three transformers were removed from service in the restoration efforts on
1Q16 adding to the strain on the remaining transformers in this area of the
network. The network was operating in a tenth contingency resulting in very
high network transformer loads and secondary mains also being subjected to
overloads.
Event # 13: July 18th at 22:25 Feeder 1Q19
Feeder 1Q19 sustained a failure and opened auto on July 18th at 22:25 with
5,060 amperes of fault current as recorded by the substation PQ Node and
calculated by Con Edison Distribution Engineering, and C phase and ground
relay targets. The feeder remained alive on back feed and the feeder breaker
was closed at 23:59. The 1Q19 feeder breaker opened automatically 7 minutes
later at 00:06 on July 19th with three phase overcurrent of 7,028 amperes
recorded by the PQ Node with B, C and Ground relay targets being reported.
The operation was attributed to the failure of the transformer in VS 624. The
transformer was overloaded and sustained a failure. The feeder again remained
alive on back feed, which was cleared in 8 hours and 4 minutes, after
application of a ground at the station. Processing of the feeder was pursued
from the pothead compartment to expedite the restoration process. The failed
transformer VS 624 was isolated from the main feeder by the installation of
live-end caps in manhole M 1929. There was a 1 hour and 3 minute delay
encountered due to the test bus conflicts at North Queens Substation. The
Feeder 1Q19 was restored to service at 04:32 on July 20th.
139
Event # 14: July 19th at 08:51 Feeder 1Q14
Feeder 1Q14 sustained a three phase fault of 11,220 ampere magnitude, as
recorded by the substation PQ Node and calculated by Con Edison Distribution
Engineering, that was cleared by A and C phase relaying. At this point in time
the network was operating in a 9th Contingency with approximately 478 or 40%
of the transformers were out of service from 08:51 through 11:33 on July 19th.
The feeder was processed from the pothead compartment to expedite feeder
processing and restoration. Delays were caused by a caution of 50 minutes and
equipment conflict of 55 minutes due to the test bus at North Queens
Substation. An Elastimold 2W-1W X-E joint was identified as the failure in
manhole M 908. The fault was isolated by installing a live-end cap removing
the joint and various sections of cable including two transformers (located in
TM 923 and VS 7684) from the main run of the feeder. A modified hi-pot was
performed on the feeder and it was restored to service at 06:35 on July 20th.
Event # 15: July 19th at 11:33 Feeder 1Q01
The feeder 1Q01 sustained a B phase fault of 4,880 ampere magnitude, as
recorded by the substation PQ Node and calculated by Con Edison Distribution
Engineering, which was cleared by B phase relaying. At this point in time the
network returned to a 10th Contingency with approximately 530 or 45% of the
transformers were out of service from 11:33 to 13:10 on July 19th. The feeder
remained alive on back feed for 21 hours and 4 minutes before a station ground
was applied and the feeder processing began. Standard fault locating procedures
were utilized. There were delays caused by the alive on back feed condition and
test bus conflicts of 21 hours and 17 minutes and defective test equipment
delays of 36 minutes. The operation was identified as cable failure in manhole
M 1892 due to external damage from a defective duct. The cable was repaired
and the feeder was tested by a modified hi-pot test at 20:22 on July 20th. The
failure was identified to be a 1W-1W X-E joint in manhole M 1593. The fault
was repaired and the feeder was retested with a modified hi-pot test and passed
at 07:26 on July 21st. The feeder was closed back into service at 07:59 on July
21st.
Event # 16: July 19th at 21:29 Feeder 1Q17
Feeder 1Q17 experienced an A phase fault of 4,470 ampere magnitude, as
recorded by the substation PQ Node and calculated by Con Edison Distribution
Engineering, that was cleared by the feeder breaker at North Queens Substation
with an A phase relay target recorded. At this point in time the network
returned to a 7th Contingency with approximately 389 or 32% of the
transformers were out of service from 21:29 on July 19th through July 20th at
00:46. The feeder remained alive on back feed for 6 Hours and 3 minutes until
140
it was cleared by placing a 3 phase ground at North Queens Substation. A
caution caused a 1 hour and 29 minute delay due to a manhole fie on the run of
the feeder. A tentative fault was identified as a failure of the transformer in
vault VS 7981. The faulted transformer was isolated from the main run of the
feeder by installing live-end caps in manhole M 1183. The feeder 1Q17 was
given a modified high potential test and returned to service at 12:37 on July
20th.
Event # 17 July 20th at 13:37 Feeder 1Q07
Feeder 1Q07 experienced an A phase fault of 4,180 ampere magnitude, as
recorded by the substation PQ Node and calculated by Con Edison Distribution
Engineering, that was cleared by the station breaker. There were no relay
targets reported. At this point in time the network returned to a 4th contingency
with approximately 263 transformers out of service from 13:37 through 13:48
on July 20th. The open automatic operation was identified as a defective
Elastimold 2W-1W P-X joint in manhole M 523. A caution was placed on
1Q07 due to a fire on the run of the feeder. The defective Elastimold joint was
isolated by installing live-end caps in manhole M 523. This isolated the
transformer in V 3385 and two sections of cable from the run of the feeder.
Repairs were completed and a modified hi-pot performed. Feeder 1Q07 failed
the hi-pot test and another Elastimold 2W-1W P-X joint was identified as the
fault in pull box 783. The fault was isolated by installing live-end caps in pull
box 783 which removed the shunt reactor in TM 632 from the main run of the
feeder. The feeder was processed from the pothead compartment to expedite
feeder restoration. A modified hi-pot of the feeder was successful and the
feeder 1Q07 was restored to service at 06:36 on July 21st. There were no
reported delays in the restoration process even though the pothead compartment
was used to perform the processing with some equipment conflicts recognized.
System Operations Rapid Restoration Procedure
System Operations Department utilized their Rapid Restoration of Distribution
Facilities procedure during the Long Island City network event, excluding
efforts to restore the North Queens Substation bus section. A rapid restoration
function is guided by System Operations procedure S011-5-9 which states the
following in section 3.0 paragraph 3.1 as to when rapid restoration pertaining to
underground distribution facilities are to be applied:
Experience has demonstrated that in a majority of cases when underground
distribution feeders (including aerial cable) open automatically accompanied
by relay targets this usually indicates the presence of a permanent fault.
Therefore, as a matter of policy, these feeders should not be reenergized (except
141
as indicated in paragraph 3.2, 3.3 and 3.4) without making further attempts to
locate and isolate the fault.
Section 3.2 and 3.3 define the cases for consideration of events in the
underground and section 3.4 apply to overhead cables. There is reason to
believe that sections 3.2 and 3.3 could be interpreted to apply to the Long Island
City event.
3.2
In some cases the loss of a single underground feeder will result in an
interruption of customers because the alternate supply feeder is out of service.
If the fault in this situation is not visible and the alternate feeder cannot be
restored, then it may be advisable to attempt a restoration prior to locating the
fault through test, after discussion with the Electric Operations Control Center
Shift Manager.
3.3
In those cases where multiple feeders have opened auto in unrelated bus
sections and have resulted in an interruption of customers, the District
Operator shall attempt restoration via supervisory control.
There were certain guidelines established during the event such as if there were
no relay targets and the feeder was alive on back feed, rapid restoration should
be attempted because of the multiple contingencies. Due to the extreme
contingency an attempt would also be made to restore a feeder with relay targets
reporting in order to reduce the number of feeders out of service. System
operations cited their processes and past experience as the logic to attempt a cutin. The seven events in Table 3 comprised the application of the rapid
restoration procedure. The actions taken during this event did not restore
feeders rapidly and the resulting cut-in open-autos had roughly similar fault
currents to the original open automatic operations.
142
Feeder
Date
OA
Time
Cut-in
Fault
Target
Fault Current
(in Amperes)
OA
CIOA
Alive
on
Back
feed?
Failed
Equipment
1Q02
7/17/06
19:48
20:08
C-IOC
10,432
0
Yes
None
1Q20
7/17/06
21:42
21:55
(CIOA)
A-IOC
8,996
6,912
No
Transformer
1Q01
7/17/06
21:49
21:56
(CIOA)
C-IOC
4,140
6,720
No
Transformer
1Q13
7/18/06
20:04
21:46
AB-IOC &
G –TOC
8,930
0
Yes
None
1Q16
7/18/06
20:37
21:50
(CIOA)
B-IOC
11,698
13,830
No
Transformer
1Q18
7/18/06
21:50
23:55
(CIOA)
C-IOC & GTOC
5,290
Inrush
Yes
Transformer
1Q19**
7/18/06
22:24
23:59
C-IOC &
GTOC
5,060
7,028
Yes
Transformer
* Feeder opened 7 minutes later with 3 phase fault current of 7,028 Amperes due to a failed transformer.
Table 3: Rapid Restoration Feeders -- Excludes substation bus related restoration – See
North Queens substation section
Cut-in Open-autos
As shown in Table 4, there were thirteen cut-in open-auto operations during the
Long Island City network event. Some overall observations of these operations
are:
• There were 6 cut-in open-auto operations where no form of hi-potting
was performed during the restoration efforts, including 2 that were
restored under rapid restoration procedures by System Operations
• There were 4 cut-in open-auto operations attributed to the magnitude
of inrush current exceeding the setting of the relay
• Feeder 1Q14 was one of the cut-in open-auto operations attributed to
the magnitude of inrush current exceeding the setting of the relay. It
was the only feeder subjected to a full hi-pot test. It is believed that
the test caused the fault that was found, a failed joint on July 23rd at
19:47
• There was limited use of hi-pot tests until after Wednesday morning,
July 19th. After adopting the policy of conducting modified hi-pot
tests for all feeders that opened automatically, only 2 additional cut-in
open-auto operations occurred
143
Looking back on the event, and with the benefit of information learned during
the investigation of the event, the Committee is of the opinion that had Con
Edison’s relay settings been set higher than the magnitude of the inrush current,
and if hi-pot testing (which is typically waived during summer months to
expedite feeder restoration) or other fault locating procedures were employed
prior to attempting to restore feeders that indicated a faulted condition, at least
seven (1Q01, 1Q07, 1Q14, 1Q16 on July 18th at 12:50, 1Q17 on July 19th at
08:49, 1Q18, 1Q20 on July 17th at 21:42) of the cut-in open-auto operations
might have been avoided.
144
CIOA
Details
Inrush?
Yes/No
ABF?
Yes/No
Faulted
Component
Fault Mag
of Prior
OA
CIOA
Fault
Magnitude
Hi-Pot?
1Q07
July 17th
19:09
Yes
No
Joint
N/A
*
None
1Q20
July 17th
21:55
No
No
Transformer
8996
Amps
6912 Amps
None
1Q01
July 17th
21:56
No
No
Transformer
6720 Amps
None
1Q16
July 17th
23:33
No
No
Joint
8800 Amps
4540 Amps
None
1Q21
July 18th
2:47
No
Yes
Joint
(CIOA)
10,424
Amps
20,252
Amps
Modified
1Q20
July 18th
5:52
No
No
Joint
(CIOA)
6912 Amps
6,604 Amps
Modified
1Q21
July 18th
17:09
No
No
Joint
(CIOA)
20,252
Amps
12,140
Amps
None
1Q02
July 18th
18:54
No
Yes
Joint
10,224
Amps
5,472 Amps
Modified
1Q17
July 18th
20:53
No
No
Transformer
4,400
Amps
11,868
Amps
None
1Q16
July 18th
21:50
No
No
Transformer
11,698
Amps
13,830
Amps
Modified
Yes
Transformer
5,290
Amps
*
Modified
1Q18
July 18th
23:55
4140 Amps
Yes
1Q17
July 19th
08:49
Yes
No
Joint
(CIOA)
11,868
Amps
*
Modified
1Q14
July 23rd
19:47
Yes
No
Joint
N/A
*
Full
*The peak inrush current cannot be determined from PQ Node data as inrush peaks may not be coincident
with station load peaks.
Figure 26: Cut-in Open-Autos July 17- July 23, 2006
145
Conclusions
Partial Restoration
• The restoration efforts throughout the Long Island City network event
can be characterized by a strong focus on restoring the feeders. The
thinking of all involved in the restoration rightly emphasized that the
strength and backbone of the network is the condition of the primary
feeders that supply the network transformers and its underlying
secondary grid. However, the same attention given to the primary
should be given to the secondary system in order to develop an
understanding of the event and impact on the network. While it
lowered the overall network contingency level, it resulted in removing
transformers from service that weakened the secondary grid system
during subsequent contingency operating conditions.
• Throughout the Long Island City network event the use of live-end
caps to restore primary feeders removed approximately 31 additional
transformers from service. In all, there were 13 transformer related
failures. In addition, 6 feeders failed due to collateral damage
associated with secondary fires. The transformer failures and feeder
failures due to secondary burnouts indicate that while the focus was
on the primary system, more attention should have been directed to
the secondary grid.
• Of particular concern is the work performed on feeder 1Q21. When
1Q21 failed, there were three faults subsequently identified. To clear
them in one instance on 1Q21, ten transformers were removed as part
of the work performed. The decision to utilize a known-point-splice
was made at an early point in the restoration effort but a third fault
required that 3 sections of cable needed to be replaced. The decision
was made to repair the cable but the concurrent repair to return these
transformers to service was not done. During that time, plans could
have changed to drop only 2 transformers via a live-end cap. This
would have restored 8 of the 10 transformers on the spur to service.
The lack of these transformers, later in the Long Island City event,
contributed to high loading of the transformer in VS 7388 which
subsequently failed and caused the loss of feeder 1Q17.
146
Expedited Restorations
• Another concern that developed during this event was the re-closing
of feeder breakers without confirmation of the fault by establishing a
condition and positively identifying the fault. While this was
successful in three cases, in retrospect, it was counterproductive
overall since the majority of the restoration attempts closed back into
faulted conditions. This was attempted on seven different occasions.
• The relay settings on the instantaneous phase relays created
considerably more complexity throughout the process, as did the 3S
bus section trip out at North Queens Substation. The 3S bus section
opening removed three network feeders from service simultaneously
and caused the network to rapidly escalate from a second to a fifth
contingency. The bus outage initiated the use of “Section 9s” and the
re-closing of feeder breakers utilizing rapid restoration procedures,
which states “It is possible that equipment damage, and consequent
increased exposure to loss of load, may occur when attempting to
restore a facility. However, there are benefits to be achieved if
customer inconvenience can be reduced when successful restoration
can be accomplished” . In all instances, the decision to initiate rapid
restoration requires a careful balancing of the two factors. The
Committee contends that in all instances a careful balancing of these
two factors is always required. The procedure in Section 3.0 of SO115-9 also recognizes that on underground feeders that open
automatically with relay targets usually indicate the presence of
permanent faults. “Therefore, as a matter of policy, these feeders
should not be reenergized.” The procedure allows exceptions based
upon customer outages however, locating and isolating the fault is the
preferred course of action.
• Section 9, another provision of the “Rule Book”, was invoked on five
occasions with mixed results. The use was effective in the early
outages but contributed to over twenty six hours of delay due to
operating conflicts on feeder restorations later in the Long Island City
network event.
• Application of high voltage and modified hi-potting was seldom used
in the early restoration effort. Modified hi-pot test were used at
reduced voltage of 30 kV for a 5 minute period approximately half
way through the event in an attempt to reduce the incidences of cut-in
open-automatic operations which was effective after it was instituted
147
as only two additional cut-in open-auto
Additional faults were generally located with
not have been operating failures, but like
potential proof testing program, hi-potting
incipient faults.
operations occurred.
hi-potting which may
the Company’s high
most likely located
Substation Implications
• Another factor that contributed to delays in restoration was the testing
capabilities at the North Queens substation. This substation utilizes a
test bus configuration such that only one feeder can be processed or
tested per bus section at a time. This restriction caused almost 18
hours of delay to restoration of various feeders during the Long Island
City event. The substation operators removed pothead plates and used
other methods in an effort to work around the limitation of the North
Queens Test Bus. This helped but delays were still encountered.
• In addition there were numerous alive on back feed conditions that
occurred during the network contingencies resulting in over sixty-four
hours of delay, where network protectors did not open. To clear the
back feeds, grounds were applied at the substation. As the
contingencies increased, and secondary damage increased as well,
application of substation grounds became less effective and may, in
fact, have caused additional secondary separation by causing limiters
to clear.
148
Restoration and Processing Strategy
Figure 27: Long Island City network outage case 16 (The source of this figure is Con
Edison)
• The Committee recommends that the restoration of feeders be
approached more strategically. Specifically, that isolated feeders not
be restored to service in certain situations but deliberately held out of
service, so that there is less likelihood of a transformer or feeder
becoming overloaded. An example best describes the case.
• The Long Island City network was in an eighth contingency on July
18th with feeders 1Q01, 1Q20, 1Q17, 1Q18, 1Q21, 1Q02, 1Q13, 1Q12
out of service. Looking at Figure 27 above labeled “Case 16,” the
network would benefit significantly if 1Q21 and 1Q02 were restored
together rather then if either were restored alone.
• In the north east corner of the network 5 feeders in 3 adjacent feeder
bands are out of service. In the Southeast corner of the network 6 of 8
149
feeders in four adjacent feeder bands are out of service, 1Q18, 1Q21,
1Q20, 1Q02, 1Q01, 1Q17. Restoring any single feeder could in these
load pockets, subject the transformers and the feeder to very high
loads. The restored facilities would attempt to pick up the load and as
happened during the Long Island City network event pick up too
much load and fail. Instead the restoration of the feeders should be
prioritized and if possible at least feeders 1Q02 and 1Q21 be restored
together. This would reestablish one feeder in each of two adjacent
feeder bands and more equally distribute the load on the facilities
being restored.
• While the transformers might still be overloaded there is a much
greater probability they would stay in service. In addition, the
secondary mains cables would also be in a better position to share the
load as multiple pathways from several directions would be
established.
Recommendations
Partial Restoration
• While it is impossible to predict what contingencies will occur during
any event, a review of local area impact and what the next worst event
would be for that local area should be considered in all decisions to
drop multiple network transformers.
• The secondary cable loading, and the ability of the secondary mains
and supporting transformers for the next worst contingency, should be
evaluated and included in the decision making process.
Substation Implications
• Ground and Test equipment at North Queens Substation and other
substations of like design, should be commissioned as soon as
possible after breaker commissioning.
• The rack out breaker replacement program at North Queens
Substation and other substations of like design should be completed as
quickly as possible. Acceleration options should also be evaluated.
Expedited Restorations
• Review criteria outlined in System Operations procedure SO11-5-9
and provide specific instruction regarding closing feeder breakers on
underground feeders that have positive indicators of faulted
150
conductors. Criteria should restrict this practice except for extreme
emergencies.
• Repair decisions for expedited feeder restoration should include
consideration of all outstanding isolated cable.
• The action of closing in breakers without following the standard
feeder restoration process should be thoroughly reviewed. If
considered for future use, specific procedures on when such actions
can take place should be well defined.
Restoration and Processing Strategy
• Isolated feeders should not be restored to service in certain situations
but deliberately be held out of service so that there is less likelihood of
a transformer or feeder becoming overloaded.
• Since contingencies can change quickly, strategies need to be
reviewed during the course of any event and modified as appropriate.
• The implementation of the “Rule Book” Section 9, while it may
expedite one feeder’s restoration, may delay or negatively impact the
processing of the other feeders out of service. This factor should be
considered when adopting a restoration strategy.
151
Recommendations
This section of the report summarizes all of the recommendations found
throughout the report.
The Committee identified direct and root causes of the event, and provides
recommendations that are pertinent to those causes. Further, the Committee
offers additional recommendations relating to broader issues which provide
opportunities for Con Edison to take actions to improve overall performance
and reliability, and to reduce the possibility of network outages occurring in the
future. The recommendations are summarized below as those that are pertinent
to the direct and root causes and those that relate to broader issues.
Recommendations Pertinent to Direct and Root Causes
Ratings and Load Cycles
• Conduct a study to determine the expected time to failure in a multicontingency event. The study should determine the time required to
enable remedial actions to prevent /control overloads and failures. If
insufficient time is available, consider and develop appropriate
engineering solutions.
• Evaluate if feeder ratings, estimated loads on associated feeder bands,
and individual feeders should be considered in planning network
reinforcement.
• Investigate and develop criteria for application of “band relief” which
may suggest relief be implemented earlier than only relieving
individual overloads.
• Review the transformer ratings being assigned for normal, first, and
second contingency operation and ensure that the secondary street ties
and mains cables are within their ratings during the corresponding
operation.
• Evaluate load curves currently applied to transformers and confirm the
curves are appropriate. If not, revise them so that the proper curve is
used and the PVL model is updated. Specific attention should be
given to nearby transformers.
Event Preconditions
• Establish criteria which require that all transformers that are
forecasted to be loaded at or above a pre-determined value prior to the
152
event are analyzed to identify potential load pocket problems. These
transformers should be monitored during the event.
Event Speed
• Investigate the use of arc-proofing on secondary cables and crab joints
or the installation of fire shields between secondary crabs and primary
cables to limit collateral failure and the communication potential from
fire damage from one to the other.
Transformers Overheat
• Train additional field crews who can supplement crews normally
assigned to cooling transformers. Use the Long Island City network
event as a guide to determine staffing levels to ensure cooling can
begin early enough to prevent transformer overheating. Consider a
three hour period until a study can be performed to identify a suitable
timeframe.
Secondary Trouble Analysis
• Identify all the sources of information that may assist analysis of
secondary mains trouble such as calls from customers, third parties,
community agencies, field crews, network maps, installed monitoring
systems, and engineering support systems. Then develop a procedure
and provide training to analyze all the streams of information to
identify remedial actions that would isolate the problem areas and
protect uninvolved network components from overload. As part of the
analysis answer questions such as:
o When, where, and how far should localized load be reduced
o What equipment can be isolated by live-end capping of feeders
and when not to use this method to expedite feeder restoration
o Advise operators of the priority order of when feeders should be
returned to service first and why
o Estimate the impact of how much secondary load can be pickedup when a feeder is restored and its impact on facilities being
restored
o What might be the secondary transformer loading upon feeder
restoration
o When specific feeders should be restored within a short time of
one another to avoid excess thermal stressing
o Estimate of the location and number of customers impacted
during the outage and restoration.
153
Secondary Mains and Street Ties
• Conduct a study in all networks to determine if the ratings of
transformers match the ratings of the secondary street ties and
secondary mains cables. Based upon the results, develop and
implement a plan to either reinforce the street ties and secondary
mains as needed or lower the rating on the transformer to match the
secondary mains and street ties.
Secondary Contact Fingers
• Evaluate the effectiveness of the replacement contact arrangement
being installed on circuit breakers similar to 1Q21 and if that
arrangement is found to be an optimal solution to the misalignment
issue, review the criteria currently in use that determines at what point
the replacement is installed. Adjust the criteria as necessary to
prevent a future occurrence similar to the Bus Section 3S trip that
occurred during the Long Island City network event.
Red Light Circuit Wiring
• Expedite the modification of this wiring scheme on all circuit breakers
in which the scheme is similar to that of 1Q21where the Merlin Gerin
Model SF2 breakers were used to retrofit the cubicle.
• Establish a review process that includes the Relay Protection Section
for all modifications and alterations of the relay protection system.
In Rush Settings on other Networks
• Review all feeders on the Con Edison system to determine the total
MVA of the transformers connected and establish a procedure for
changing the settings based on new transformers or equipment being
added or transferred.
Recommendations Relating to Broader Issues
Ratings and Load Cycles
• Utilize available breaker positions at the North Queens substation to
establish new feeders to reduce the average normal and emergency
feeder loading and improve diversity on the Long Island City network.
• Review the use of ratings in PVL and establish criteria for when the
use of these ratings should be applied, specifically during operating
and planning conditions (see Attachment D for definitions of
engineering applications).
154
• Review the transformer ratings that are presently assigned in PVL and
ensure they are appropriate for the actual load cycle for the area
during peak conditions.
Summer Preparation
• Use the operations analysis of the Brooklyn/Queens Engineering
Department’s compliance with CSP 5-3-19, EO-2072, and EO-2048
as a guide and consider similar analyses of engineering departments in
other operating regions. Re-analyze Brooklyn Queens Engineering to
ensure all recommendations have been implemented.
• Share the operations analysis among all regional engineering groups.
Consider following future engineering analyses with field audits or
analyses of the completed work.
Event Preconditions
• Develop a procedure such that pre-event analyses include confirming
that all transformers that are out of service are accounted for
• Consider the impact of dropping un-faulted transformers on the next
worst event prior to dropping un-faulted transformers.
• Consider establishing a transformer installation team equipped with
vehicles, equipment, and material to replace transformers during the
event.
• Establish clear criteria, as part of EO-10110 that requires appropriate
supervisory approval to downgrade a “heavy corrosion condition” to a
lower classification.
Area Demand
• Develop procedures to reduce small commercial and residential
customer demand during periods in which a local high load pocket is
subject to expanding due to overloaded or highly loaded nearby
transformers and secondary mains.
Transformers Overheat
• Consider reducing the action-threshold which identified the specific
points that transformers are to be cooled.
• Conduct PVL studies as part of summer preparation for all networks
to identify transformer cooling candidates if multiple contingencies
beyond design criteria were to occur.
155
Feeder Restoration
• Revise EO-4095 to more specifically address details how to use
knowledge of “field conditions” when making decisions about how to
restore feeders in terms of priority, live-end caps, shunts, and partial
restorations.
• Investigate if design modifications or different criteria are appropriate
during normal operation and contingency operation along network
fringes. The fringe area of the network is most vulnerable to
directional network support.
• Revise EO-4095 to ensure that special attention is directed to network
fringe areas and ensure the operators and designers consider new
fringe areas created during multiple contingency events.
Customer Calls
• Develop and implement customer awareness programs to increase
outage calls from interrupted customers served by networks.
• Develop a process to ensure that when field crews receive customer
reports of outages or other service problems, those reports are added
to the information used to analyze secondary mains trouble.
• Revise voice scripts to reduce the likelihood of inferring to network
customers that Con Edison is aware of their service problem. State
explicitly that Con Edison needs their specific information to ensure
timely restoration of service.
• Develop criteria and a process to separately identify calls regarding
extremely low voltage trouble from calls regarding low voltage during
an 8% voltage reduction action.
• During periods of network trouble, include trouble calls from third
parties (NYPD, OEM, and FDNY) in the analysis of secondary mains
trouble.
• Review the calls that are currently coded as EDSCRE (referrals to
electricians) during a multiple contingency. Presently these may not
be evaluated, losing critical information that could be made available
to trouble analysis.
During this event, these represented
approximately 10% of the B tickets.
• Consider a research program to develop some type of empirical-based
algorithm or rule-of-thumb that can be used to estimate the locations
and number of customers who may be affected by a network problem
based upon the number of customer who actually called to report a
problem.
156
Non Standard Engineering Applications
• Standardize engineering support applications and centralize quality
controls, version controls, application testing, application interfacing,
and new application development.
Poly Voltage Load Flow
• Establish specific criteria and develop a procedure that assures the
implementation of actions and ensures that network secondary
connections are accurately represented in the PVL model. Modify the
PVL connectivity model to include overhead secondary mains and
place customer loads appropriately.
• Complete the development of and deploy an enhanced PVL model or
the procurement of a commercial equivalent of an enhanced PVL. To
be most useful to today’s engineers, PVL should be more visual and
preferably linked to a Geographical Information System (GIS). It
should seamlessly integrate with other existing applications to
coordinate protection, forecast load growth and provide a variety of
analyses in a visual format.
• Investigate commercial software that exists today to see if they can be
applied to a network as large as Con Edison’s, that will allow one to
point and click at a section of a feeder to quickly reconfigure the
feeder and secondary connectivity and perform revised load flows.
PVL is not currently capable of doing this for secondary networks.
Load Pocket Analysis
• Develop and implement a standard Company procedure to ensure that
“local load pockets” can be analyzed in real time when network
components such as feeders, cable sections, transformers and
secondary mains are out of service.
Exception-Based Analyses
• Develop criteria and new output reports for PVL and other analyses to
provide designer’s earlier warning, instead of just seeing “overloaded
conditions.” Create a threshold below the overload condition so that
designers and operators see components that are approaching overload
or a threshold of concern.
• Standardize mapping symbols utilized in the visual tools used by
operators and designers.
157
Data Presentation, Integration, and Integrity
• Develop and implement processes and standards to ensure that data
are consistent between engineering support applications, as well as
naming conventions and engineering symbols.
• Revisit the corporate data integration and presentation approach and
strategy. Confusion will be reduced and operational efficiency will be
improved.
RMS Receiver Database
• Require that testing of RMS reporting is completed after transformer
additions or when transformers or transmitters are electrically moved
from one feeder to another feeder.
• Ensure that the RMS database index file is appropriately updated.
RMS Availability
• Identify minimum RMS reporting requirements below which
operators will know that the information is suspect or not sufficiently
accurate to use for operating decisions. Institute proper management
controls and audits to ensure compliance with those requirements.
• Enforce present criteria for RMS reporting as specified in Con Edison
specifications or review the 95% reporting requirement. Test the
system to determine the appropriate level or determine other ways to
measure the effectiveness of the RMS reporting rate.
• Ensure networks with high transformer counts and large pockets of
residential load are prioritized accordingly in all efforts undertaken to
improve reporting rates.
• Suspect pickup coils should be replaced as soon as practical. With
restoration work underway, pickup coils should be replaced as needed
as feeder outages are scheduled. This work should be placed in the
feeder repository to ensure the District Operator knows that it is
outstanding and may combine it with planned work or when a feeder
opens auto.
RMS Related Record Keeping
• Develop a new RMS ID and transmitter phase database, or update the
existing database, by conducting a field survey of units without proper
records. Further, embed ID, transmitter phase and status checking
into normal routines when visiting such equipment for maintenance,
inspection or other routine purposes.
158
In Rush Settings on other Networks
• Establish an annual review of the number of transformers per feeder
and perform periodic reviews of feeder relay settings.
Electro-mechanical Relay Speed
• Study the impact of raising the instantaneous current relay protection
setting to determine if tripping times adequately protect the equipment
and the public. If so, consider replacing electro-mechanical relays
with micro-processor relays whose settings can distinguish between
inrush and fault current to maintain fast clearing times.
Relay Protection Engineering Approvals
• Develop and implement a procedure that requires all operating regions
to notify the Relay Protection Engineering Section of specific changes
to the number or size of transformers in a timely manner. Require the
Relay Protection Engineering Section to take actions to ensure the
relay setting adequately considers inrush current from magnetization
such that “no fault” cut-in open-autos are prevented.
• Investigate if a procedure that requires periodic testing or calculation
of inrush current from magnetization is required and assure that relay
settings are coordinated accordingly.
Electro-mechanical Relay Targets
• Review and modify as appropriate the Con Edison maintenance
procedure that should have prevented the failure of the electromechanical relay targets on 1Q01 on July 17th and 1Q07 on July 20th.
• Develop a method for the PQ Node equipment to provide a station
alarm when fault current is detected, thus providing the District
Operator and Station Operator with a confirmation that a fault existed
even when a relay target is not present.
Breaker Clearing Time
• Inspect and test feeder circuit breakers 1Q20 and 1Q19 (including
breaker timing tests) to correct the apparent long clearing times
observed on July 18th and July 21st.
Partial Restoration
• While it is impossible to predict what contingencies will occur during
any event, a review of local area impact and what the next worst event
would be for that local area should be considered in all decisions to
drop multiple network transformers.
159
• The secondary cable loading, and the ability of the secondary mains
and supporting transformers for the next worst contingency, should be
evaluated and included in the decision making process.
Substation Implications
• Ground and Test equipment at North Queens Substation and other
substations of like design, should be commissioned as soon as
possible after breaker commissioning.
• The rack out breaker replacement program at North Queens
Substation and other substations of like design should be completed as
quickly as possible. Acceleration options should also be evaluated.
Expedited Restorations
• Review criteria outlined in System Operations procedure SO11-5-9
and provide specific instruction regarding closing feeder breakers on
underground feeders that have positive indicators of faulted
conductors. Criteria should restrict this practice except for extreme
emergencies.
• Repair decisions for expedited feeder restoration should include
consideration of all outstanding isolated cable.
• The action of closing in breakers without following the standard
feeder restoration process should be thoroughly reviewed. If
considered for future use, specific procedures on when such actions
can take place should be well defined.
Restoration and Processing Strategy
• Isolated feeders should not be restored to service in certain situations
but deliberately be held out of service so that there is less likelihood of
a transformer or feeder becoming overloaded.
• Since contingencies can change quickly, strategies need to be
reviewed during the course of any event and modified as appropriate.
• The implementation of the “Rule Book” Section 9, while it may
expedite one feeder’s restoration, may delay or negatively impact the
processing of the other feeders out of service. This factor should be
considered when adopting a restoration strategy
Network Shutdown
• Consider ways to reduce the significance and negative consequences
to the community of shutting down the network by making the
network smaller or effectively smaller. This should include, but not
be limited to, considering a new substation to split the network and the
160
•
•
•
•
•
consideration of primary and/or secondary grid sectionalizing
capability.
Consider changing the design criteria, now based solely on station
capacity, to include local demographics and the community impacts of
a network shutdown.
Consider splitting the Long Island City network to reduce average
feeder loading, length and shut down impact to the community.
Investigate and as appropriate develop detailed plans to manually
sectionalize the secondary into predetermined sections.
Develop a training program to ensure high levels of analytical skills
and strategic decision-making for Incident Commanders and others
involved in the process of evaluating multiple contingency situations
and deciding the most effective tactical responses.
Organize the Incident Command team so that the unique responsibility
for the analysis and remediation of each of the three considerations
provided in EO-4095 (and listed below) are clearly assigned in the
structure of the response team.
o Overloads on primary feeders can not be eliminated, or
o There are reports of cascading manholes on fire, or
o Network transformers are loaded beyond the allowable limits
when cooled.
Network Size
• Utilize “Jeopardy” analysis as part of the criteria in evaluation of the
need for reinforcement and prioritization of relief projects.
• Investigate what role the Jeopardy application should play in the
evaluation of a possible network shutdown decision.
161
Attachment A - Committee Members
The Committee members and their biographic summaries follow:
Robert W. Donohue (Chairman)
Mr. Donohue has over 43 years of experience in the electric utility industry. Since
2003, he has provided consulting services, assessments and operational expertise to a
variety of utilities across the Country. These services include evaluations of
underground network systems, network planning and investment decision-making, and
post-event investigations of outage management after significant outage events. Mr.
Donohue has served the electric power industry on the Research Advisory Council for
the Electric Power Research Institute (EPRI). He was Chair for EPRI’s Power
Delivery Council, its Distribution Business Council, and is a senior member of the
Institute of Electrical and Electronics Engineers. He served as Chairman of EPRI’s
Power Delivery Reliability Initiative and Vice Chairman of Edison Electric Institute’s
(EEI) Transmission and Distribution Committee.
Mr. Donohue has held management and senior executive positions for Consolidated
Edison Company. Prior to his retirement in early 2003, as Senior Vice President of
Electric Operations, Mr. Donohue was responsible for all electric distribution
operations, maintenance, engineering, construction, planning, and customer services for
Con Edison’s territory which serves a population of over 9 million people through
nearly 120 thousand miles of underground and overhead line.
Wade P. Malcolm
Mr. Malcolm has nearly 25 years of experience in the electric utility industry. He was
most recently the Vice President of Power Delivery and Markets for the Electric Power
Research Institute, where he had held a variety of management and officer positions in
his eleven years of employment. Mr. Malcolm guided the development of an industry
portfolio of collaborative research and development and the development of a variety
of technical services supporting power delivery operations. Mr. Malcolm has
supervised and provided oversight to major outage investigations globally and has been
instrumental in the development and commercialization of emerging technologies used
to improve reliability such as the PQ Node, PQ View software and reliability
benchmarking methodologies.
Previously, Mr. Malcolm was the President and CEO of Powel Group, Inc. energy
software and services company. Mr. Malcolm was also a Vice President at Utility.com,
the first internet utility. He was a Principal in the Chemicals and Energy Practice of
SRI Consulting and spent nine years in a variety of technical assignments at PECO
Energy. He is a Senior Member of the IEEE and a Registered Professional Engineer.
162
Edward N. Neal
Mr. Neal has over 35 years of experience in the electric utility industry. Since 2004, he
had provided management and technical consulting services to electric and
telecommunications utilities. Mr. Neal has provided these services in the areas of
utility operations, maintenance, construction, power plant development, renewable
energy and regional transmission organization markets. Mr. Neal is a licensed
professional engineer in the commonwealth of Pennsylvania and has served on the
Power Delivery Group Council of the Electric Power Research Institute, the Committee
on Power Delivery of the Association of Edison Illuminating Companies, and the
Executive Board of the East Central Area Reliability Council.
Mr. Neal held various engineering, management, and senior executive positions for
Duquesne Light Company while employed there from 1970 to 2004. Mr. Neal’s
responsibilities included substation, transmission, distribution, telecommunications,
underground system operations, distribution operations, and business development.
Ron Williams
Mr. Williams has nearly 30 years of experience in the electric utility industry. Since
1998, as Vice President of EXL Consulting, Inc, then as President of PDR&C, Corp.,
he has consulted throughout the United States in the areas of utility reliability, outage
management, asset management, and the use of information technologies for electric
distribution reliability and efficiency improvement. This includes investigations of
investments in underground distribution infrastructure, practices of distribution
operations and maintenance organizations responsible for underground networks, and
evaluations of the performance of utilities during large-scale outage events. Mr.
Williams has conducted investigative studies and advised numerous utility clients and
served a coalition of thirty utilities as consultant for the Electric Power Research
Institute’s Power Delivery Reliability Initiative.
Previously, Mr. Williams held various leadership positions at Pacific Gas & Electric
Company including Plant Manager of fossil fuel and geothermal power plants,
Manager of the San Francisco Division, Manager of PG&E’s Electric Distribution
Information Technology Program, Manager of Organization Planning and
Development, and Executive in Residence at PG&E’s corporate Learning Center.
163
Attachment B - Pre-Event Banks-Off and Open
Mains
Banks-Off – July 17, 2006
M&S 58Z
The transformer in TM5937 on 1Q17
M&S 60AA
The transformer in V3174 on 1Q15
M&S 61AA
The transformer in TM6602 on 1Q04
The transformer in TM6701 on 1Q04
M&S 61AC
The transformer in TM0831 on 1Q18
M&S 61Z
The transformer in V1769 on 1Q08
The transformer in V5041 on 1Q09
M&S 62Y
The transformer in V3073 on 1Q07
M&S 63AE
The transformer in VS7307 on 1Q20
M&S 63AF
The transformer in TM5865 on 1Q14
M&S 63Y
The transformer in TM6360 on 1Q07
M&S 63Z
The transformer in V0768 on 1Q23
M&S 65AC
The transformer in TM2244 on 1Q19
M&S 65AE
The transformer in VS 5023 on 1Q14
M&S 65Z
The transformer in V0599 on 1Q05
M&S 66AC
The transformer in VS7659 on 1Q12
M&S 66Z
The transformer in TM5783 on 1Q08
M&S 67AB
The transformer in V7859 on 1Q20
164
M&S 67AE
The transformer in V9426 on 1Q17
M&S 68AF
The transformer in VS5447 on 1Q19
M&S 69Z
The transformer in VS9847 on 1Q21
M&S 72AH
The transformer in VS8283 on 1Q02
M&S 72AI
The transformer in VS8640 on 1Q15
M&S 73AI
The transformer in V1086 on 1Q14
M&S 76AF
The transformer in V4175 on 1Q06
Open Mains- July 17, 2006
M&S 67AB
SB593 - SB595
M&S 47AG
MH12380 – VS8283
M&S 68AJ
MH20777 – MH10569 – SB33221
M&S 67Z
SB53252 – SB50096 – SB20371
M&S 66Z
SB64868 – SB53045 – SB53046
165
Attachment C - Limited Use Computer
Applications
Several applications and tools exist that appear to be known well enough to be
used by only a few individuals. While Jeopardy, Monitor and Contingency
applications have potential value in responding to an event such as the Long
Island City network event, there seems to be differing opinions of the value and
readiness of some of these applications. This creates confusion and inefficiency
during an event. Jeopardy, Monitor and Contingency are well known
applications at Con Edison and appear to be useful tools, as does Visual WOLF.
However, the Committee was advised that only a limited staff is capable of
effectively running the applications. Some of the additional representative
applications are described below.
STAR
STAR (System Trouble Analysis and Response) is a computer system that
analyzes problems and tracks jobs on the electric distribution system. It receives
SCADA information on the status of field equipment and displays it on control
center maps. By analyzing additional information received from customer calls,
STAR is able to identify the causes of system trouble. It then creates jobs for
corrective work and allows operators to prioritize and track these jobs through
to completion.
STAR understands where customers are connected to the distribution system.
When problems on the system affect customers, it uses this information to
quickly identify the number and names of all customers affected. It also keeps
customer logs for every job, regardless of whether or not the customers have
called the Company.
STAR has a historical database that collects job and equipment information.
This information is used in reports that are automatically generated by STAR. It
is also available to users of the system for other reports and analysis.
STAR is very precise in estimating customers out for radial systems. It was
used as an experiment for the Long Island City Network by the Information
Resources Department at the CERC during the event. Its results indicated a
customer outage count higher than initially estimated in the Brooklyn/Queens
region. The discrepancy in customer outage count was in part the driver to
conduct actual customer outage surveys and assessments later in the event. It
166
requires an accurate model and accurate data input maintenance in order to
effectively estimate customer outages.
CuFLink
CuFLink is a back office system that combines data from the Customer
Information System and Mapping systems (VISION, EDFIS and Brooklyn
Mapping). It places customers onto electric and gas services. CuFLink also
utilizes the customer billing data to calculate demand imposed by the customer.
The quality of CuFLink’s results varies. Its accuracy seems better for those
Regions that have implemented another application underdevelopment known
as STAR. This is because the regions implementing STAR have spent months
upgrading their models to ensure STAR would provide optimum results.
Load Pocket Analyzer
Load pocket analyzer predicts loading in an area based upon feeder data
provided concerning banks off, open mains, shunts, etc. and develops a scoring
or assigns weights (not a detailed analysis). Initially developed in the Brooklyn
region, it was built for engineering to look at the system prospectively each
morning. This is to help identify switch checks on network transformers with
suspected blown fuses, open network protectors, etc. However, its effectiveness
is limited when the area of concern spans 2 or 3 M&S plates.
NetDVD
It is a pilot project for Manhattan Control Center’s (MCC) data visualization
and display applications. It is under development. The project’s goal is to
develop an advanced, web-based map visualization tool that allows Control
Center operators to quickly analyze and respond to distribution system
problems.
Users will be able to directly associate data with symbols superimposed on the
Company’s facility maps, such as M&S plates and feeder maps. The newly
created outage maps are based on information extracted from other computer
systems, and provide a dynamic interface to MCC applications such as STAR
(CufLink) and NetRMS.
A version is now being used by the Queens Long Island City Restoration
Organization for restoration and for recording “as found” information in the
Long Island City Network.
167
NETCAP and DYNCAP
NETCAP is an application being developed by CYME International for Con
Edison as a replacement to an older, in-house created cable rating application,
CARP. The application creates cable section ratings based upon PVL data
models and load flow results. PVL then uses the cable section ratings to
establish and publish feeder ratings.
The application assembles manhole cross-sections and equips the conduits with
cables specified within PVL Feeder (.nf) and Secondary (.usn) data models.
NETCAP then translates and incorporates Primary (.plf) and Secondary (.slf)
load flow results for each cable in the previously established manhole crosssection. This is performed for each cable equipped manhole wall (cable
sections) in the subject network. Thermal calculations are then performed using
published network load cycle curves which result in a cable section rating for
each cable in the network being analyzed. PVL then uses this information to
establish the limiting section on each feeder and calculates the overall "Feeder"
rating.
CYME International T&D is a power engineering solutions provider and offers
an extensive line of power engineering software analysis tools for transmission,
distribution and industrial power systems.
DYNCAP is a real-time, dynamic cable ampacity rating program under
development which will calculate a 24 hour thermal timeline for each feeder
section in a network that has experienced an event (feeder contingency).
Sections which are found to exceed their rating over the next 24 hour period
will be accumulated in an exception report, available to specified electric
operations personnel.
The dynamic application will rate all feeder sections remaining in service in the
subject network after each event. The application will reach a determination for
the top twenty sections for each feeder in service within 15 minutes of when it is
triggered to run. The dynamic application will compare the new (dynamic)
rating with the WOLF load flow output and report which, if any, of the top
twenty sections are either currently above or will be above its thermal limitation
in the next 24 hours. The estimated time, for those feeder sections which will be
thermally overloaded over the next 24 hours will also be reported by this
application. Additionally, the application will be responsible for calculating
168
section ratings upon the return of each feeder so as to “reset” the individual
thermal status.
The dynamic application will, upon completing calculations for the top twenty
sections, continue to calculate the thermal impact of the event on the remaining
feeder sections and report any thermal violations in the same manner it did for
the initial twenty sections. The calculations will only continue beyond the top
twenty if processing is not pre-empted by another system event in the same
network. Any additional events in the subject network will cease current
processing in favor of a new set of calculations to include the impact of the
latest event.
DYNCAP was built after NETCAP and is a dynamic thermal model. It relies
on WOLF to run in order to predict thermal impact and overloaded
transformers, cables and joints, and how long the assets can sustain the thermal
strains. As such, DYNCAP tries to take a prospective look. It was in place for
all areas this summer for use as an engineering tool. Operators will be trained
on its use next. The Information Resources Department envisions it will be
used for supporting operations such as feeder restoration.
169
Attachment D - Engineering Applications Defined
Engineering
Application Name
CAJAC:
Purpose of Engineering Application
Cable And Joint Analysis Control System - CAJAC
is used to keep a record of all feeder cable and joint
failures that occur in the electric distribution system.
Connectivity Model:
An integral element of PVL is the Connectivity
Model. The model is where the various elements of
data are associated together to actually simulate the
system. The model requires accurate definitions of
the feeder components and equipment such as cable
sections, joints, transformers and other related
equipment. In some cases, associated operational
parameters also need to be taken into account (such
as historical failure rates, equipment age and type,
temperature ranges, etc.) in order to apply proper
operational ratings.
CONTINGENCY:
A planning tool used to assess the reliability of
feeders and networks. A twenty year period is
evaluated in a five hour run time. The model
develops relative reliabilities and ranks networks.
CuFLink:
CuFLink combines data from the Customer
Information System and Mapping systems (VISION,
EDFIS and Brooklyn Mapping).
DYNCAP
A dynamic thermal model. It relies on WOLF to run
in order to project thermal impact and overloaded
transformers, cables and joints, and how long the
assets can sustain the thermal strains.
Electric Distribution DIS is an intranet system that provides an overview
Information
System status or 'health' of the Electric Distribution System.
(DIS)
The system provides a view of the network loading
for each area and the status of the feeders within
each network. The system also provides drill-down
capability to more detailed information
Engineering
A mix of corporate and locally developed
Workstation
engineering
applications
in
use
in
the
Brooklyn/Queens Region. It has developed its own
applications to augment, and in some cases replace,
170
Load Pocket Analyzer
NETCAP
NetDVD
PVL, Win_PVL
RME
what is provided at the corporate level.
Load pocket analyzer predicts loading in an area
based upon feeder data provided concerning banks
off, open mains, shunts, etc. and develops a scoring
or assigns weights (not a detailed analysis).
NETCAP is an application being developed by
CYME International for Con Edison as a
replacement to an older, in-house created cable
rating application, CARP. The application creates
cable section ratings based upon PVL data models
and load flow results. PVL then uses the cable
section ratings to establish / publish feeder ratings.
This is a pilot project for Manhattan Control
Center’s (MCC) data visualization and display
applications. The project’s goal is to develop an
advanced, web-based map visualization tool that
allows Control Center users to quickly analyze and
respond to distribution system problems
PVL (Poly Voltage Load flow) is a suite of
programs used in the analysis and design of the Con
Edison electrical distribution system, the most
important of which is the core Power Flow
application. It calculates balanced, three phase
power flows on primary & secondary network
systems and their radial source feeders for normal
and contingency conditions. Auto-Loops may also
be modeled but only as three phase systems. The
distribution system is assumed to be a balanced
three-phase system, single or two-phase lines or
demands cannot be specified. The user is able to
select which case(s) are to be calculated (i.e.:
normal, specific contingencies, or ALL cases) and
can choose from a wide array of reports with which
to view the analysis results
The Remote Monitoring Estimator (RME) is a
UNIX based application developed to compensate
for deficiencies in the data reported by Hazeltine's
Remote Monitoring System (RMS). This
functionality is required to support the "Advanced
Contingency Analysis" capabilities of WOLF, which
came into being as a result of the reviews into the
171
RMS, NetRMS
RT3
STAR
VDAMS
Watchman
Visual-WOLF
WOLF, Auto-WOLF
heat related Washington Heights events in the
Summer of 1999
Remote Monitoring System to monitor the
approximately 25,000 network transformers on the
Con Edison system. NetRMS is a networked
version of report generator.
Real Time Transformer Top Oil Temperature –
Computer program that estimates network
transformer or 4 kV unit substation transformer top
oil temperature based on actual loading conditions
from the RMS reading of the past 48 hours.
System Trouble Analysis and Response - STAR is a
computer system that analyzes problems and tracks
jobs on the electric distribution system
Vax Data Acquisition and Management System –
System that collects data from the Remote
Monitoring System (RMS) on network transformer
loading and from the system that monitors primary
feeders and substation transformer loading
A visual display in the NetRMS system whose
purpose is to provide the operator a quick snapshot
of transformers loading and capability
Presents WOLF real time load flow calculation
results for primary feeder sections and network
transformers in a graphic environment
World class Operation Load Flow – Load flow that
collects present network field conditions and runs all
combinations of first contingencies to predict
possible overloads on primary feeders, network
transformers and secondary mains. It can also
predict the same for the next worst second or higher
contingency
172
Attachment E - Feeder Processing Listing of
Definitions
Word or Phrase
Definition
(ABF) Alive on Energization of electrical equipment from a source other
back feed
than the normal source after the primary supply has been
disconnected.
(AC) Alternating The Application of a low voltage alternating current
Current Ammeter through an ammeter in series with a current limiting
Clear Test
resistor to indicate the presence of short circuits or
grounds on the feeder conductors or on transformer
secondarys.
Banks Off
Transformers that are not in service.
Caution
A term used to alert all personnel working on a feeder
that an event such as a fire that can delay feeder
processing or that a feeder may be re-energized.
CIOA or cut-in- Term used to indicate that when an attempt to energize a
open-auto
feeder and return it to service is made. The feeder again
immediately opens automatically.
Contingency
A scheduled or unscheduled event which results in the deenergizing of a primary source of power supply to an
electrical distribution system component.
Customer Outage
When a network customer’s supply is interrupted. In the
case of multi-phase customers, when one or more phases
are interrupted.
Energized
The act of connecting a piece of equipment to a source of
electrical potential.
Equipment: Electric cables, wires, buses, conductor runs,
and apparatus (i.e. network transformers and network
protectors).
Equipment
Electric cables, wires, buses, conductor runs, and
apparatus (i.e. network transformers and network
protectors).
Establish
a The application of high voltage potential for the purpose
Condition
of creating or establishing a low impedance path to
ground at a fault location.
Fault
A permanent or transient condition which causes
equipment not to perform in its required manner.
Feeder
The Electric Cable, Station Equipment, and connected
173
apparatus between the controlling generating station,
substation, high-tension customer installation, back feed
transformer, underground step down transformer and one
or more of the following points:
•
The controlling circuit breaker or disconnecting
device in the same or in another generating station or
substation or high-tension customer installation
•
The first circuit breaker or disconnecting device in
the street
•
The transformer or insulated end cap at the ends
of the cable if there is no circuit breaker or disconnecting
device
(G&T) Ground and A breaker specifically designed to install a protective
Test Device
ground on a feeder cable. The breaker is also equipped
with a test position that allows the application of various
test voltages.
Grounded
Connected to earth or to some extended conducting body
that serves as earth, whether the connection is intentional
or not.
Grounding: Application of a low impedance path or short
circuit between all conductors and ground.
(Hi-pot)High
The application of high voltage on a feeder, transformer
Potential Test
or other apparatus to test its electrical integrity.
(IATC) Individual Current used for both cable identification tests and phase
Action
Tracing identification tests.
Current
In Service
A term used to indicate equipment is connected to the
power system and fulfilling its designated function.
Isolated:
Disconnected from the system by the opening of
switches; disconnecting potheads, cutouts, or links; or by
cutting or disconnecting conductors.
(KPS)
Known A location where the feeder designation and phase
Point Splice
markings when provided are maintained by management
personnel in the responsible operating group, immediately
after each change in the electrical connections. In the
special case of permanent identification via badging on
pipe type feeders phase markings are not required. Work
on pipe type feeders shall commence at the location
without standard identification as long as connections are
not broken which exposes the copper.
(LEC)
Live-End A sealed housing installed on a primary cable end, which
174
Cap
Network Protector
allows the cable to be energized at system voltage.
A three phase automatic circuit breaker in series with a
set of fuses on the secondary side of network transformers
that will open or close automatically on demand.
Open Mains: A section of secondary cable isolated from
the secondary grid.
Outage: Equipment is disconnected from the electrical
system and is not fulfilling its designated function.
Open Mains
A section of secondary cable isolated from the secondary
grid.
Outage
Equipment is disconnected from the electrical system and
is not fulfilling its designated function.
PQ Node
A power quality monitoring device that is capable of
providing load and voltage measurements at a very high
sampling rate
Secondary Cable, Generally 120/208 volt cables being supplied from the
Secondary Grid
secondary winding of a higher voltage transformer
Section 9
A section of the Company’s “Rule Book,” General
Instruction Governing Work on System Electrical
Equipment, which permits specific work to proceed on
high tension cables without performing positive
identification of the faulted cable.
175
Attachment F - Failed Transformers from the Long
Island City network event
176