B
B
O
O
N
N
N
N
E
E
V
V
I
I
L
L
L
L
E
E
P
P
O
O
W
W
E
E
R
R
A
A
D
D
M
M
I
I
N
N
I
I
S
S
T
T
R
R
A
A
T
T
 NERC Monitoring and Situational Awareness Conference 
Bonneville Power Administration
Self-Monitoring: Network and System Operations Center (NSOC)
Title
Sub Title
September 30, 2015
1
I
I
O
O
N
N
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
Today’s Topics
 Agency Overview;
 NSOC: Definition, Purpose and Services:
•
Monitoring system health;
•
Monitoring network and telecommunication health; and
•
Monitoring cyber security.
 Benefits of an NSOC; and
 Challenges of Developing an NSOC.
2
I
O
N
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
I
Agency Overview
 Bonneville Power Administration (BPA) is a federal agency
established in 1937;
 BPA is self-funded and recovers all Agency costs through rates for
wholesale power sales and transmission services; and
 BPA’s service territory spans 15,000 transmission miles in the
Pacific Northwest and consists of Oregon, Washington, Idaho,
and parts of California, Montana, Nevada, and Utah;
 BPA is registered with NERC for the following functions:
• BA, TOP, TO, TP, TSP, PC, PSE, and RP
3
O
N
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
Agency Overview,
M
I
N
I
S
T
R
A
T
I
continued
BPA:
 Assures resource adequacy to meet the Pacific Northwest
region’s firm power requirements;
 Markets, but does not own, wholesale power from federal dams
and acquired from non-federal generation;
 Builds, owns, operates and maintains a high-voltage
transmission system to integrate and deliver power from federal
and non-federal generation to regional customers in the Pacific
Northwest and between the Northwest and the Southwest; and
 Protects and enhances fish and wildlife in the Columbia River
basin.
4
O
N
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
I
BPA Statistics
BPA Statistics
Amount
Balancing Authority (BA) Installed
Generation
32,157 MW
BA Peak Load, Feb. 6, 2014
10,643 MW
Average Load, Jan. 1 May 27, 2015
6,200 MW
Total Exports 2014
83,267,202 MWh
Total Imports 2014
25,221,117 MWh
Interchange Points with 17 BAs
~247
Transmission Customers
~500
Operates and manages the federal transmission system (over 15,000 circuit
miles over ― 11,000 circuit miles 230kV or higher, over 260 substations).
5
O
N
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
I
O
N
Agency Overview
Power
Services
Transmission
Services
6
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
I
O
Agency Overview
BPA markets electric power from 31 federal
Power
Services
dams, the Columbia Generating Station
Nuclear Plant, and several small nonfederal power plants:
 About 80 percent of the power BPA
sells is hydroelectric; and
 BPA accounts for about 30 percent
of the electric power consumed
within the region.
7
N
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
I
O
N
Agency Overview
Power
Services
Transmission
Services
8
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
I
O
N
Agency Overview
Transmission System
Operating voltage Circuit miles
1,000 kV…………………....... 264*
500 kV ………………......... 4,803
345 kV ………………………. 570
287 kV ………………………. 229
230 kV …………………….. 5,327
161 kV ………………………. 119
138 kV ………………………… 53
115 kV …………………….. 3,509
below 115 kV ………………...382
Total
15,156
Transmission
Services
BPA SUBSTATIONS . ………259
*BPA’s portion of the PNW/PSW direct-current intertie. The total length of this
line from The Dalles, Ore., to Los Angeles, Calif., is 846 miles.
BPA’s transmission system
contains more than 15,000
miles of high voltage lines.
About 75 percent of the
high-voltage grid in the
Pacific Northwest.
9
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
I
O
N
Transmission Services – System Operations
System Operations is responsible for the safe, reliable, open access operation and dispatch of the
high voltage transmission system and interconnected generation. This is accomplished through
the operation and management of two independent control centers. These two control centers,
one in Vancouver and one in Spokane, are connected via BPA-owned and operated, fully
redundant, telecommunication systems comprised of both fiber optic and radio systems, with the
same BPA-owned communication systems also used to communicate to most remote sites.
Both centers are staffed and operating on a 24/7 basis.
During normal operations, each control center
has access to all transmission data, most
through independent communications, but
responds to only that portion currently under
their jurisdiction. Either the locally hosted
systems or those of the alternate control
center can be used.
During emergency operations, jurisdictions can
be reassigned between control centers based
on communication connectivity or a single
control center can assume jurisdiction of the
entire BPA footprint.
10
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
Network and System Operation Center
Provides continuous Network and
System monitoring, incident response, IT support,
remedial action, and incident coordination.
11
I
O
N
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
I
O
What Is an NSOC?
 The N-SOC is a combination of a Network Operation Center (NOC)
and a Systems Operation Center (SOC);
 The N-SOC support identifies problems as a result of alerts
processed by monitoring tools, customer complaints, or other
actionable information; and
 The N-SOC support group is staffed by people with broad technical
skills, so they can address complex operational issues with little or
no assistance.
12
N
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
I
O
N
NSOC Services
 Provide single point of contact for work affecting the control center
computers, software, networks, and building systems;
 Perform on-site system response, mitigation, and recovery;
 Provide immediate IT/OT response, support, and maintenance;
 Support the coordination of work in the Control Center’s data center
to ensure no concurrent outages of critical systems;
 Remote OTDR system allows for immediate response;
 Personnel tracking at remote sites;
 Provide damage assessments and begin system recovery
immediately after a critical failure;
continued…
13
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
NSOC Services,
D
M
I
N
I
S
T
R
A
T
I
continued
 Advise dispatch on system availability, path degradation, and
schedule work impacts to the BES;
 Perform detection and reporting of potential cyber instances; and
 Centralized log collection correlation and monitoring.
14
O
N
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
NSOC
15
I
O
N
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
System Health: Monitoring
 Infrastructure status and functionality;
 Functionality alarming and response;
 Stale-data detection;
 Performance utilization; and
 Availability metrics.
16
I
O
N
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
I
― SYSTEM HEALTH ―
Infrastructure Status and Functionality
SCADA alarms for Control Center infrastructure status
Server status and functionality

Infrastructure status:
•
•
HVAC, UPS, Generator, etc., status; and
Rack power availability.
 Infrastructure functionality:
•
Server status and health indications:
 Drive space;
 CPU utilization; and
 Power supply health.
17
O
N
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
I
O
N
― SYSTEM HEALTH ―
Functionality Alarming and Response
EMS System Alarm Screen
 Functionality alarming:
•
•
EMS systems provide alarming related to the function
of the system; and
Alarming provided for metering and curtailment utilities
along with various systems used by our dispatchers and
schedulers.
 Response:
Procedure Storage
•
Document store contains over 900 procedures for immediate
response:
• Documents searchable and tagged or manually
grabbed from store.
18
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
― SYSTEM HEALTH ―
Stale-Data Detection
 SCADA Datamon alarms:
•
•
Watches various indications on data from RTUs and ICCP; and
Alarms with data has not changed in a predetermined time-frame.
SCADA alarm for stale data
ICCP Dataset Status


Displays ICCP Data
status; and
Graphical alert if a
single dataset has
bad data.
19
I
O
N
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
Network and Telecom Health: Monitoring
 Remote communication and environmental monitoring:
•
•
•
•
SCADA RTU status;
Fiber monitoring;
Microwave monitoring;
Telemetry data
 Network availability:
•
•
•
Between Control Centers;
Between primary and secondary systems; and
Between ICCP and points.
 Network utilization; and
 Network status mapping.
20
I
O
N
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
― NETWORK AND TELECOM HEALTH ―
Remote Site Communication and
Environmental Monitoring
 Communication
Alarming:
•
•
•
Telecommunication Alarming
Communication
status;
Fiber status; and
Microwave status.
 Environmental:
•
•
•
Door alarms;
HVAC status; and
Battery and
generator status.
21
I
O
N
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
I
O
― NETWORK AND TELECOM HEALTH ―
Network Availability and Utilization
ICCP Network availability and utilization
 Network availability:
•
Graphical
representation of
control center
networks and
services.
 Network utilization:
•
Graphical path
utilization alerting.
22
N
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
Cyber Security
 Monitoring;
 Response; and
 Security controls testing.
23
I
O
N
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
― CYBER SECURITY ―
Monitoring
 Malware detection;
 Intrusion detection;
 Login failures;
 Account changes and creation of elevated accounts;
 Predictive firewall analytics; and
 Security controls testing on control systems.
24
I
O
N
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
― CYBER SECURITY ―
Response
Cyber Security Response:
 Centralized log collected for systems with-in the control center;
 Tiered Cyber Security approach:
•
•
Functional groups maintain cyber response and visibility; and
Dedicated Cyber Security and Analysis Center helps identify APTs and
performs forensics.
Splunk Alert Manager
25
I
O
N
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
I
― CYBER SECURITY ―
Security Controls Testing
Security controls testing overview:
Security controls testing:
 Approximately 460 security controls locked, enforced, and tested to ensure
security;
 Security controls exemption by exception and justification only, other
mitigations may be enforced; and
 Servers and workstations with-in the Control Centers monitored.
26
O
N
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
NSOC Final Comments
 Benefits; and
 Challenges to Development.
27
I
O
N
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
I
O
NSOC Benefits
 The benefits of the N-SOC include improved reliability, shorter
Mean-Time-To-Recovery (MTTR);
 Centralized tools requires a lower lifetime investment and increases
human resources utilization and operational efficiencies; and
 The N-SOC operates around-the-clock and is physically located
next to our Dispatch customers, making it an unmatched option at
an ideal location.
28
N
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
I
NSOC Challenges to Development
 Implementing the N-SOC required a large investment of time and
resources. A significant investment in equipment, facilities
infrastructure, staff hiring and employee training was required.
Time was required to develop the skill sets needed to manage the
new technologies and tools, cross-train, and develop equipment
maintenance programs.
 To enable the N-SOC functions, new network, hardware and
software infrastructure were needed. Additional communication
lines and video conferencing equipment, were required for full
N-SOC functionality.
29
O
N
B
O
N
N
E
V
I
L
L
E
P
O
W
E
R
A
D
M
I
N
I
S
T
R
A
T
Questions?
NERC Monitoring and Situational Awareness Conference
Self-Monitoring: Network and System Operations Center (NSOC)
30
I
O
N