August 14, 2003 Blackout
Summary Based on Interim Report of the
United States – Canada Power Outage Task Force
November 19, 2003
U.S.-Canada Interim Report
● Released November 19, 2003
● Result of an exhaustive bi-national
investigation
 Working groups on electric system, nuclear
plant performance and security
 Hundreds of professionals on investigation
teams performed extensive analysis
● Interim report produced by the teams and
accepted by the bi-national Task Force
2
Overview
●
●
●
●
●
●
3
Overview of power system and reliability
Pre-outage conditions on August 14
Trigger events and start of cascade
Wide area cascade
Root causes
Next steps
Power System Overview
4
Reliability Overview
● Balance generation and demand
● Balance reactive power supply and demand
● Monitor flows and observe thermal limits
● Observe power and voltage stability limits
● Operate for unplanned contingencies
● Plan, design and maintain a reliable system
● Prepare for emergencies
Reliably operate the system you have!
5
3 Interconnections / 10 NERC Regions
6
NERC Control Areas
7
NERC Reliability Coordinators
8
Footprints of Reliability Coordinators in Midwest
9
NERC Immediate Response to Blackout
● First hours
 Worked closely with NERC Reliability




Coordinators
Identified what had tripped and extent
of outage
Assessed restoration efforts
Maintained open line with
DOE/FERC
Communicated with DHS, White
House, and NRC
● First days
 Assigned project manager
 Established Steering Group with
industry executive experts
 Began organizing investigation teams
 90+ volunteers + entire NERC staff
10
Investigation Organization Overview
U.S – Canada
Task Force
Steering Group
Investigation
Team Lead
Project Planning and
Support
MAAC/ECAR/NPCC
Coordinating Group
Root Cause Analysis
Cooper Systems
NERC & Regional
Standards/Procedures
& Compliance
Sequence of Events
Restoration
Operations - Tools,
SCADA/EMS
Communications Op
Planning
Frequency/ACE
NPCC
Data Requests and
Management
MEN Study
Group
System Modeling and
Simulation Analysis
System Planning,
Design, & Studies
MAAC
ECAR
11
Investigation
Process Review
Vegetation/ROW
Management
Transmission System
Performance,
Protection, Control
Maintenance & Damage
Generator Performance,
Protection, Controls
Maintenance & Damage
Data Gathering and Analysis
● Three fact-finding meetings
 August 22
 September 8-9
 October 1-3
● Onsite interviews and inspections
● Secure database of outage information
● Extensive corroboration of data to
determine facts
● Analysis by teams of technical experts
12
Root Cause Analysis
● Logical structure for investigating complex
●
●
●
●
problems
Identifies changes, conditions, actions, or
inactions at each causal step
Starts with final event and drills back through
each branch of causal tree.
Asks “why?” at each step.
Accurate, reliable, defensible understanding of
the root causes.
Successfully used to investigate root causes of PJM voltage stability condition
in July 1999 and established history in nuclear and defense industries.
13
Root Cause Analysis Phases
16:15
BLACKOUT
16:06
Initial Focus
Sammis – Star
Star – South Canton
Hanna – Juniper
Chamberlin - Harding
15:05
Pre-Existing Conditions
E.g. voltages, wide- area transfers,
line and generator outages, etc.
14
August 14 Conditions Prior to Blackout
● Planned outages
 Cook 2, Davis Besse nuclear plants
 East Lake 4, and Monroe 1
● Transfers high to northeast U.S. + Ontario
 Not unusually so and not above transfer limits
● Critical voltage day
 Voltages within limits
 Operators taking action to boost voltages
● Frequency
 Typical for a summer day
● System was within limits prior to 15:05,
15
on both actual and contingency basis
Warm But Not Unusual for August
16
August 14 Imports to Northeast-Central Compared
to 6/1 to 8/13/2003
Exports
6000
Max Imports
Max Exports
Average
4000
14-Aug
0
Imports
-6000
16:00
-8000
Hour (EDT)
17
22:00
-4000
20:00
-2000
18:00
16:00
14:00
12:00
10:00
8:00
6:00
4:00
2:00
0:00
MW
2000
Voltages Prior to 15:05 EDT August 14
18
Frequency Typical for Summer Day
19
Blackout was NOT Caused by
●
●
●
●
●
●
Heavy wide-area transfers
Low voltages, voltage collapse
Lack of IPP voltage/reactive support
Frequency anomalies
Cinergy outages starting at 12:08
East Lake 5 trip at 13:31
 Contributing factor to later events, but not by
itself causal to the blackout
● DPL Stuart-Atlanta trip at 14:02
 Contributing factor to loss of MISO real-time
monitoring, but not electrically significant
20
Outage Sequence of Events
Transmission Map Key
ONTARIO
21
East Lake 5 Trip: 1:31:34 PM
ONTARIO
2
1
22
East Lake 5 Exciter Failure Causes Trip
23
Stuart Atlanta Trip: 2:02 PM
ONTARIO
24
MISO State Estimator and Reliability Analysis
● MISO state estimator and contingency
analysis ineffective from 12:37 to 16:04
 State estimator not solving due to missing
information on lines out in Cinergy then DPL
 Human error in not resetting SE automatic
trigger
● Using Flowgate Monitoring tool to monitor
conditions on previously identified critical
flowgates
25
FirstEnergy Computer Failures
● 14:14 Alarm logger fails and operators are not aware
 No further alarms to FE operators
● 14:20 Several remote consoles fail
● 14:41 EMS server hosting alarm processor and other
functions fails to backup
● 14:54 Backup server fails
 EMS continues to function but with very degraded performance
(59 second refresh)
 FE system data passed normally to others: MISO and AEP
 AGC function degraded and strip charts flat-lined
● 15:08 IT warm reboot of EMS appears to work but alarm
process not tested and still in failed condition
● No contingency analysis of events during the day
including loss of East Lake 5 and subsequent line trips
26
Phone Calls to FirstEnergy
● FE received calls from MISO, AEP, and PJM
indicating problems on the FE system but did
not recognize evolving emergency
 14:32 AEP calls regarding trip and reclose of Star-S.





27
Canton
15:19 AEP calls again confirming Star-S. Canton trip
and reclose
15:35 Calls received about “spikes” seen on system
15:36 MISO calls FE regarding contingency overload
on Star-Juniper for loss of Hanna-Juniper
15:45 FE tree trimming crew calls in regarding
Hanna-Juniper flashover to a tree
PJM called MISO at 15:48 and FE at 15:56 regarding
overloads on FE system
Chamberlin-Harding (3:05:41)
28
Chamberlin-Harding Indication of Ground Fault Due
to Tree Contact as Measured by DFR at Juniper
29
(3:05:41)
30
Hanna-Juniper
(3:32:03)
Hanna Juniper Confirmed as Tree Contact at
Less than Emergency Ratings of Line
31
Effects of Ambient Conditions on Ratings
32
(3:05:41)
Star- S. Canton (3:41:35)
33
(3:32:03)
Situation after Initial Trips 3:05:41 – 3:41:35
ONTARIO
34
Canton Central – Tidd
(3:45:41)
35
138 kV Lines Overload and Cascade Near Akron
Simulated 138 kV Line Loadings
200
Dale-W.Can
138 kV
% of Normal Ratings (Amps)
180
W.Ak-PV
Q22 138 kV
160
Cham-W.Ak
138 kV
140
E.LimaN.Fin 138 kV
120
CantC Xfmr
100
Dale-W.Can
138 kV
W.Ak 138 kV
Bkr Failure
Cham-W.Ak
138 kV
36
E.Lima-N.Fin
138 kV
Outages
CantC Xfmr
0
W.Ak-PV Q21
138 kV
20
Babb-W.Ak
138 kV
E.LimaN.Lib 138 kV
E.Lima-N.Lib
138 kV
40
Clov-Torrey
138 kV
Babb-W.Ak
138 kV
Star-S.Cant
345 kV
60
Hanna-Jun
345 kV
W.Ak-PV
Q21 138 kV
Hard-Chamb
345 kV
80
Clov-Torrey
138 kV
0
Dale-W.Canton 138 kV
W.Akron 138 kV Breaker
Chamberlin-W.Akron 138 kV
E.Lima-N.Finlay 138 kV
20
Sammis-Star
120
Canton Central Transformer
W.Akron-Pleasant Valley 138 kV
Babb-W.Akron 138 kV
HannaJuniper
60
16:05:55 EDT
15:51:41 EDT
15:32:03 EDT
Star-S.Canton 15:41:35 EDT
40
HardingChamberlin 15:05:41 EDT
37
E.Lima-New Liberty 138 kV
80
Cloverdale-Torrey 138 kV
100
% of Normal Ratings
138 kV Cascade Contributes Further
to Overload of Sammis-Star
140
Sammis-Star
(4:05:57.5)
38
Sammis-Star Zone 3 Relay Operates
on Steady State Overload
39
Actual Loading on Critical Lines
1600
Harding - Chamberlin
Hanna - Juniper
Star - South Canton
Sammis - Star Line Trip
Sammis - Star
Star - South Canton Line Trip
Hanna - Juniper Line Trip
Flows (MW)
1200
East Lake 5 Trip
800
400
Harding - Chamberlin Line Trip
0
12:00
13:00
14:00
Time - EDT
40
15:00
16:00
Actual Voltages Leading to Sammis-Star
370
Harding - Chamberlin
345 kV Line Trip
Hanna - Juniper
345 kV Line Trip
Gaps in Data Records
100% Voltage
350
95% Voltage
Voltage (kV)
330
90% Voltage
310
Star
290
270
Hanna
Star - South Canton
345 kV Line Trip
Sammis - Star
345 kV Line Trip
Beaver
Perry
250
15:00
41
Time - EDT
16:00
Major Path to Cleveland Blocked after Loss of
Sammis-Star 4:05:57.5 PM
Remaining
Paths
42
345 kV Lines Trip Across Ohio to West
ONTARIO
43
Generation Trips 4:09:08 – 4:10:27 PM
ONTARIO
44
345 kV Transmission Cascade Moves North into
Michigan 4:10:36 – 4:10:37 PM
45
Northern Ohio and Eastern Michigan Served Only
from Ontario after 4:10:37.5 – 4:10:38.6 PM
46
Power Transfers Shift at 4:10:38.6 PM
47
Eastern Eastern Michigan (Detroit) Unstable
Voltage and Frequency Collapse and Pole Slipping
Ontario – Michigan Interface Flow and Voltages Beginning 16:10:38
48
Generator Trips to 16:10:38
49
Generator Trips – Next 7 Seconds
50
Overloads on PJM – NY Ties 4:10:39 PM
51
PJM – NY Separating 4:10:44 PM
52
Cleveland – Toledo Island 4:10:39 - 4:10:46 PM
Cleveland Blacks Out
53
Northeast Completes Separation from Eastern
Interconnection 4:10:43 – 4:10:45 PM
North of Lake
Superior
54
Conditions at Niagara Indicate Progressively
Worsening Stability Conditions with Prior Events
55
Island Breaks Up: 4:10:46 – 4:13 PM
56
Frequency in Ontario and New York during Breakup
Niagara Generation Stays with Western NY
57
Generator Trips – After 16:10:44
58
End of the Cascade
Some Local Load
Interrupted
Areas Affected by the Blackout
Service maintained
in some area
59
Blackout Root Cause Group 1
FE Situational Awareness
● FE did not ensure a reliable system after
contingencies occurred because it did not have
an effective contingency analysis capability
● FE did not have effective procedures to ensure
operators were aware of the status of critical
monitoring tools
● FE did not have effective procedures to test
monitoring tools after repairs
● FE did not have additional high level monitoring
tools after alarm system failed
60
Blackout Out Root Cause Group 2
Vegetation Management
● FE did not adequately manage tree growth
in its transmission rights of way
61
Blackout Cause Group 3
Reliability Coordinator Diagnostics
● MISO’s state estimator failed due to a data error.
● MISO’s flowgate monitoring tool didn’t have real-
time line information to detect growing
overloads
● MISO operators couldn’t easily link breaker
status to line status to understand changing
conditions.
● PJM and MISO ineffective procedures and wide
grid visibility to coordinate problems affecting
their common boundaries
62
Near-Term Industry Actions
Responses from Control Areas and Reliability
Coordinators Due December 15
63

Voltage support/reactive supply

Reliability communications

Computer failure response & notifications

Emergency action plans and capabilities

Operator training for emergencies

Vegetation management
Next Steps
● U.S./Canada Power Outage TF hearings
 Public hearings to allow comment on report and input
on recommendations
 December 4
 December 5
 December 8 – Toronto
 Industry technical conference
 December 10 – Philadelphia
● NERC next steps
 NERC executive committees December 11
 NERC committees meet January 13-14
 Continue investigation
 Near term analysis and recommendations in support of U.S.
Canada Task Force
 Long term analysis and recommendations for NERC
64