The Role of Software in the
August 14, 2003 Blackout and
Subsequent Investigation
Second Carnegie Mellon Conference in
Electric Power Systems:
Monitoring, Sensing, Software and Its Valuation for the
Changing Electric Power Industry
January 11, 2006
Carnegie Mellon University
Eric H. Allen, New York ISO
Blackout Team Investigator
Summary of August 14 Blackout
● Impacts
ƒ
ƒ
ƒ
ƒ
ƒ
ƒ
8 states/2 provinces
Over 50 million people
60-65,000 MW
96+ hours to restore
Manufacturing disrupted
531 generators tripped
− 19 nuclear generators
at 10 plants
● Statistics
ƒ Line trips began at 3:05 PM
ƒ Cascading began at 4:06 PM
ƒ Instability began at 4:10:37 PM
− Lasted approximately 12 seconds
ƒ Thousands of discrete events
2
Outage Sequence of Events
Transmission Map Key
ONTARIO
3
Transmission Lines
765 kV
500 kV
345 kV
230 kV
East Lake #5 Generator Trip: 1:31:34 PM
ONTARIO
ONTARIO
4
Transmission Lines
765 kV
500 kV
345 kV
230 kV
Power System High Level Sequence
● Premature failure of three 345 kV lines
ƒ
starting at 3:05 PM, three 345 kV line outages within 36
minutes due to tree branches under conductors
● Northern Ohio 138kV cascade began
ƒ
started 3:39 PM - caused by above premature failures
● Northern Ohio 345kV high speed cascade of
overloaded lines 4:05:57 - 4:09:07 PM
ƒ
accelerated by Zone 3 directional distance relays
● Eastern Interconnection Separates by 4:11PM
5
Chamberlin-Harding (3:05:41)
6
(3:05:41)
7
Hanna-Juniper
(3:32:03)
(3:05:41)
Star- S. Canton (3:41:35)
8
(3:32:03)
Sammis-Star
(4:05:57.5)
9
Major Path to Cleveland Blocked after Loss of
Sammis-Star 4:05:57.5 PM
Remaining
Paths
10
345 kV Lines Trip Across Ohio to West
4:08:59 - 4:09:07 PM
ONTARIO
11
Generation Trips 4:09:08 – 4:10:27 PM
ONTARIO
12
345 kV Cascade Moves North into Michigan
Argenta-Battle Creek lines trip 4:10:36 – 4:10:37PM
13
New Phase Begins - “Transient Instability”
Three 345 kv Lines trip from 4:10:37.5 - 4:10:38.6 PM
14
NY to Ontario 345kV Line Flows at Niagara
Progressively Worsening Stability Conditions
New York to Ontario 345 kV Line Flow at Niagara
(does not include 230 kV line flow)
2000.00
380.0
Niagara KV
1800.00
360.0
1600.00
340.0
East Lima - Fostoria
Central 345 trip
1400.00
Thetford-Jewel, Hampton Pontiac,
& Perry - Ashtabula 345kV lines trip
320.0
Argenta - Battle Creek
double circuit 345 trip
300.0
1000.00
Sammis-Star 345 trip
280.0
800.00
ON TARIO
260.0
600.00
PA301&2 MW
240.0
400.00
220.0
Transmission Lines
200.00
765 kV
500 kV
345 kV
15
16:10:48
16:10:42
16:10:37
16:10:31
16:10:26
16:10:20
16:10:01
16:09:56
16:09:50
16:09:45
16:09:39
16:09:33
16:09:28
16:09:23
16:09:17
16:09:11
16:09:06
16:09:00
16:08:55
16:06:55
16:06:49
16:06:44
16:06:38
16:06:33
16:06:27
16:06:22
16:06:16
16:06:11
16:06:05
16:06:00
16:05:54
16:05:49
16:05:43
230 kV
0.00
200.0
KV
MW
1200.00
Power Transfers Shift at 4:10:38.6 PM
16
Eastern Michigan (Detroit)
“Transient Instability” Phase Begins
MW/MVAr
kV
300
4000
MW
3000
250
kV
2000
200
1000
150
0
100
MVAr
50
-1000
-2000
16:10:38
16:10:40
16:10:42
16:10:44
Time - EDT
17
16:10:46
0
16:10:48
Detroit Units Slip Poles
Keith-Waterman (J5D) 230 kV - Tie Line
800
Keith-Waterman Trips
at 16:10:43.2
600
Classical
Stability
400
200
#4
0
37
38
39
40
41
42
43
-200
#1
#2
#3
-400
Severe Voltage Depression
in Downtown and Southern
Detroit Region
-600
Toledo/Cleveland Island
Separates from Detoit
Detroit Area Generation Pulls
Out of Synch and Slips 2 Poles
as Frequency Increases to ~62 hz
Significant Generation Loss and/or
Transmission Seperation in Detroit
-800
Seconds from 16:10
18
Remaining Detroit Generation
Slips 2 Poles as Frequency Fallsat
44
V
P
Q
Frequency in Ontario and New York
19
Penna–New York Separation 4:10:39 to :44 PM
20
Northeast Island Completes Separation from
Eastern Interconnection 4:10:43 – 4:10:45 PM
North of Lake
Superior
21
End of the Cascade
Area affected by blackout
Service maintained in
isolated pockets
22
Key Findings
● Inadequate system planning and
design studies, operations
planning, facilities ratings, and
modeling data accuracy
● Operating with insufficient
reactive margins
● More effective system protection
and controls could slow or
minimize spread of cascading
outage
● Problems from prior blackouts
were repeated
23
The Old - Repeated
● The three “T’s”
ƒ Tools – for the operator
to monitor and manage
the system
ƒ Trees – vegetation
management to prevent
tree contacts
ƒ Training – operators
need to provided training
and drills to be prepared
to respond to system
emergencies
24
The New
● Failure of Tools
Information Technology support –
communications
ƒ “Game Over”
ƒ
● Failure of “Defense in Depth”
● Generation Protection
Consideration of performance
during dynamic and extreme low
voltage events
ƒ Coordination of plant controls with
the transmission system
ƒ
25
Multiple Concurrent Software Failures
1. FirstEnergy alarm failures
2. FirstEnergy remote EMS terminal failures
3. MISO state estimator failures for L/O
Bloomington-Denois Creek 230 kV and
Stuart-Atlanta 345 kV
4. MISO topology processor failure – did not
recognize Harding-Chamberlin 345 kV
line trip when breakers were open
26
Control Area (CA) Computer Failures
● 2:14 PM - alarm logger fails and operators are not aware
● 2:41 PM - server hosting alarm processor and other
functions fails to backup - IT staff is auto-paged.
● 2:54 PM - backup server fails (2 of 4 servers down)
ƒ 2 servers continue to function but with long refresh (~59 second)
ƒ selected CA data continues to be sent to other control centers
ƒ unidentified Harding - Chamberlin 345kV line outage at 3:05 PM
● 3:08 PM - IT warm reboot of servers appears to work but
alarm process not tested and still in failed condition
● CA Operators are now ‘blind’ & lose situational
awareness
● Reliability Coordinator (RC) computers are still working.
27
“Reliability Coordinator (RC) Diagnostics”
● MISO Reliability Coordinator (RC) for CA
ƒ
“state estimator” failed due to data errors, & periodic
automatic restart turned OFF
ƒ
monitoring tools did not have real-time line information
to detect growing overloads
ƒ
not within contingency limits from 3:06 - 4:06
ƒ
28
−
“missed” Harding - Chamberlin line outage 3:05 PM
−
did not have wide area map-board for the above
−
state estimate & contingency analysis not successful.
To monitor the power system, MISO was using
software that was considered “under development and
not fully mature”
Modeling & Studies Work
● Successful Forensic Dynamic Analysis
ƒ Able to duplicate measured system response
ƒ
ƒ
ƒ
ƒ
ƒ
29
through 16:11
Far beyond expectations
Significant modeling lessons learned
Validation of Sequence of Events
Understanding of Eastern Interconnection
behaving as a single system
Set stage for further study of “what ifs”
Simulated 138 kV Line Loadings
% of Normal Ratings (Amps)
200
180
Dale-W.Can
138 kV
160
W.Ak-PV
Q22 138 kV
140
Cham-W.Ak
138 kV
120
E.LimaN.Fin 138 kV
CantC Xfmr
100
80
W.Ak-PV
Q21 138 kV
60
Babb-W.Ak
138 kV
Dale-W.Can
138 kV
W.Ak 138 kV
Bkr Failure
Cham-W.Ak
138 kV
30
E.Lima-N.Fin
138 kV
Outages
CantC Xfmr
W.Ak-PV Q21
138 kV
Babb-W.Ak
138 kV
E.Lima-N.Lib
138 kV
Clov-Torrey
138 kV
Star-S.Cant
345 kV
0
Hanna-Jun
345 kV
20
Hard-Chamb
345 kV
40
E.LimaN.Lib 138 kV
Clov-Torrey
138 kV
Dale-W.Can
138 kV
W.Ak 138 kV
Bkr Failure
Cham-W.Ak
138 kV
E.Lima-N.Fin
138 kV
Outages
CantC Xfmr
W.Ak-PV Q21
138 kV
Babb-W.Ak
138 kV
E.Lima-N.Lib
138 kV
Clov-Torrey
138 kV
Star-S.Cant
345 kV
Hanna-Jun
345 kV
31
Hard-Chamb
345 kV
0
CantC-Tidd
345 kV
100
Star-S.Cant
345 kV
80
Hard-Chamb
345 kV
20
Sammis-Star
345 kV
120
60
Hanna-Jun
345 kV
40
% of Normal Ratings (Amps)
Simulated 345 kV Line Loadings
140
Ontario-Michigan Voltage and Flows
Measured from Lambton
MW/MVAr
kV
300
4000
MW
3000
250
kV
2000
200
1000
150
0
100
MVAr
50
-1000
-2000
16:10:38
16:10:40
16:10:42
16:10:44
Time - EDT
32
16:10:46
0
16:10:48
Simulated Ontario-MI Data
33
Power Flow Modeling Problems
● Line Rating Discrepancies
ƒ
Sammis-Star 345 kV line rating different for PJM, AEP,
MISO, FE
● Topology Errors — Some dating back to 1991
ƒ
No feedback loop in MMWG/SDD case creation
process
● Power Factor
ƒ
Optimistic reactive power loads
ƒ
Not verified with actual capability
ƒ
Systemic problem of model and data quality control
● Generator Reactive Limits
● Model Quality
● Powerflow Program Version Discrepancies
34
Dynamics Modeling
● Accurate Models Necessary
ƒ One family of exciter models was found to
ƒ
ƒ
ƒ
ƒ
have an unstable step response
Some units’ models were unstable due to bad
data in the models
Generators dispatched beyond real and
reactive limits
Incorrect machine MVA base values
No dynamics data for most small units
− Generic generator data used in the MMWG case
ƒ No standard of formats for powerflow and
dynamics data
35
Dynamics Modeling
● Program deficiencies
ƒ Program does not allow modeling of
underfrequency load shedding (UFLS) and
undervoltage load shedding (UVLS) on the
same bus
ƒ Some models caused the program to freeze
or crash
● Program features
ƒ Program allows a simulation to be stopped
during the middle of a run and restarted at a
later time
36
Dynamics Modeling Innovations
● New Models Developed
ƒ Simulate phenomena not
normally represented in
standard dynamics program
models
ƒ Modified dynamic load model
−More responsive to voltage
deviations
ƒ Modified existing generator
models
−Represent generator mechanical
overspeed protection systems
37
Dynamic Analysis Finding
● Cascade arrested by dropping all of
Cleveland load prior to loss of Argenta –
Battle Creek / Argenta – Tompkins 345 kV
ƒ Nothing then measurable to suggest taking
such action
ƒ Angular separation analysis showed possible
trigger (~90 second window)
38
Maximum Power Transfer Diagram
Sending
Receiving
XS = 0
XL
VS
E S = 1.0 PU
R
VS × VR × sin Θ
P=
XL
XR = 0
VR
E R = 1.0 PU
⎛ XL
Θ = Arc sin ⎜⎜
⎝ VS VR
The maximum real power that a transmission
line can transfer occurs when the voltage angle
across the line reaches 90º.
39
⎞
⎟
⎟
⎠
Angular Separation Analysis
Generalize Power Transfer Equation to Zonal
PSR = PSR max × sin Θ
300
-0.01
40
-0.005
Θ
1.5
200
1
100
0.5
0
0
0.005
0.01
0.015
0.02
0
0.025
-100
-0.5
-200
-1
-300
-1.5
Simulation Angular Measurement Points
41
Relative Phase Angle
Angular Separation Analysis
0
-10
-20
-30
-40
-50
-60
-70
-80
-90
-100
-110
-120
-130
-140
-150
-160
-170
Normal Angle ~ -25º
15:05:00 15:32:00 15:44:00 15:51:00 16:05:00 16:06:01 16:09:05 16:10:38
Time (EDT)
42
Reference:
Browns Ferry
Cleveland
West MI
16:05:50 to 16:08:52
Western MI
Sammis - Star
Detroit
East Lima –
Fostoria Central
NJ
NY
Cleveland
43
Muskingum –
Ohio Central Galion
Ontario
15
Degrees
16:08:50 to 16:10:50
Detroit
out of
Synch
NY
Western MI
40
Degrees
NJ
NY-PJM
Separation
Cleveland
Separation
44
Cleveland
Ontario
Dynamic Recording Devices
● Non-synchronized Disturbance
Recorders
ƒ Monumental forensic analysis
required
● Dynamic data supplied in multiple
proprietary formats
ƒ Some files had format errors
● Ad-hoc software written to extract
frequency and line power flows
(real and reactive) from digital fault
recorder (DFR) data
45
Digital Fault Recording Devices
● All such devices should …
Be GPS time synchronized.
Include a frequency trace.
Be periodically checked for calibration and operability
Have all data provided in IEEE/ANSI Comtrade
standard C37.111-1999 format
ƒ Have file names in the IEEE common naming
convention.
ƒ
ƒ
ƒ
ƒ
● Add digital readout of frequency requirement to
the Comtrade standard.
● The disturbance data recording system should
facilitate data storage and archiving capabilities.
46
NY Phasor Measurements (PMUs)
4000
NY to IMO
3000
PJM to NY
NE to NY
Ont to Mi
Power Flow (MW)
2000
1000
0
-1000
-2000
-3000
-4000
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
3.
4.
6.
5.
7.
8.
9.
0.
1.
2.
3.
5.
4.
6.
7.
8.
9.
0.
:4
:4
:4
:4
:4
:4
:4
:5
:5
:5
:5
:5
:5
:5
:5
:5
:5
:0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
:1
:1
:1
:1
:1
:1
:1
:1
:1
:1
:1
:1
:1
:1
:1
:1
:1
:1
16
16
16
16
16
16
16
16
16
16
16
16
16
16
16
16
16
16
time (EDT)
NY Separates from NJ
Ontario splits from West NY
NY and NE Separate
Ontario reconnects
w ith West NY
47
63
62
61
60
59
58
57
56
NY West
NY East
Lambton
16
:1
0:
43
.0
16
:1
0:
44
.0
16
:1
0:
45
.0
16
:1
0:
46
.0
16
:1
0:
47
.0
16
:1
0:
48
.0
16
:1
0:
49
.0
16
:1
0:
50
.0
16
:1
0:
51
.0
16
:1
0:
52
.0
16
:1
0:
53
.0
16
:1
0:
54
.0
16
:1
0:
55
.0
16
:1
0:
56
.0
16
:1
0:
57
.0
16
:1
0:
58
.0
16
:1
0:
59
.0
16
:1
1:
00
.0
Frequency (hz)
Split complete betw een East and West NY
time (EDT)
Ontario Power System Dynamic Recorders (PSDRs)
16:10:36
5,000
16:10:38
16:10:41.9
16:10:38.6
16:10:39.5 16:10:42
16:10:45.2
16:10:48
New York into Ontario
New York into New England
3,000
Ontario into Michigan
2,000
1,000
0
-1,000
-2,000
Frequency (Hz)
Ontario reconnects with West NY
16:10:50
Split complete between
East and West NY
Ontario splits from West NY
NY and New England separate
Time
NY separates from NJ
Cleveland separates
from Toledo, islands
NY separates from PA
1630 MW Detroit generation trips
64
63
62
61
60
59
58
57
16:10:30
16:10:40
Start of split between
East and West MI
16:10:30
to
16:11:00
Detroit, Cleveland separated from W. MI.
Cleveland cut off from PA
PJMinto New York
-3,000
16:10:30
48
16:10:56
16:10:49
4,000
Power Flows (MW)
16:10:50
16:11:00
NY-West
Lambton ONT-MI
NY-East
16:10:40
Time
16:10:50
16:11:00
View Into Detroit from Lambton
49
System-Wide Analysis & Tools
● Power flow and dynamic analyses point
toward
ƒ Need for situational awareness of line
outages and their influence on operational
security
ƒ Need for monitoring locational reactive
reserves and resultant voltage profiles
ƒ Improved situational awareness for operators
though early warning system based on zonal
angular separation
− Need for phasor measurements for wide-area
50
Questions?