Utility Communication Fundamentals
w/ Focus on SONET
William Fletcher
WECC, Senior Compliance Auditor
NERC CEA Training
February 21, 2012
Salt Lake City, UT.
General Background
•
•
•
•
•
•
•
•
2
25 Years experience in IT and Telecom
2001 – 2008: Oregon’s largest COU
Utility Telco: SONET, WDM services over fiber
Multiple SONET ring additions substation
automation for Oregon COU
Presentations on WDM at UTC / National
Presentations for GE / Multilin re: COM/CIP
Conceptual utility product development – T1
2008 – present: WECC (with a short consulting
term at ICF International, 2011)
Agenda
• COM-001-0 Emergency /non-routine concepts
and facilities review
• Sidebar: Cell Phone Investigation
• SONET Overview – Concepts and Facilities
• SONET capacity and provision review
• Typical utility services – by SONET payload
• Typical Mapping of SONET to COM-001-0
- By requirements
• Redundancy: SONET Style
• Ring configurations
• Q & A / References
3
Utility and Telecom Interdependency
Critical Infrastructure Key Resource TLV
Source: Previous DHS/ NIPP Plans showing CIKR relationships
www.dhs.gov/xprevprot/programs/editorial_0827.shtm
4
COM-001-0 Review
Applicability: RC, BA, TOP (R1-R5)
R1.
Each Reliability Coordinator, Transmission Operator and Balancing Authority shall provide adequate
and reliable telecommunications facilities for the exchange of Interconnection and operating information:
R1.1. Internally.
R1.2. Between the Reliability Coordinator and its Transmission Operators and Balancing Authorities.
R1.3. With other Reliability Coordinators, Transmission Operators, and Balancing Authorities as necessary to
maintain reliability.
R1.4. Where applicable, these facilities shall be redundant and diversely routed.
R2.
Each Reliability Coordinator, Transmission Operator, and Balancing Authority shall manage, alarm, test
and/or actively monitor vital telecommunications facilities. Special attention shall be given to emergency
telecommunications facilities and equipment not used for routine communications.
R3.
Each Reliability Coordinator, Transmission Operator and Balancing Authority shall provide a means to
coordinate telecommunications among their respective areas. This coordination shall include the ability to investigate
and recommend solutions to telecommunications problems within the area and with other areas.
R4.
Unless agreed to otherwise, each Reliability Coordinator, Transmission Operator, and Balancing
Authority shall use English as the language for all communications between and among operating personnel
responsible for the real-time generation control and operation of the interconnected Bulk Electric System.
Transmission Operators and Balancing Authorities may use an alternate language for internal operations.
R5.
Each Reliability Coordinator, Transmission Operator, and Balancing Authority shall have written
operating instructions and procedures to enable continued operation of the system during the loss of
telecommunications facilities.
5
Source: http://www.nerc.com/files/com-001-0.pdf
COM-001-0 Review
Applicability: RC, BA, TOP (R1-R5)
R1.
Each Reliability Coordinator, Transmission Operator and Balancing Authority shall provide
adequate and reliable telecommunications facilities for the exchange of Interconnection and operating information:
R1.1. Internally.
R1.2. Between the Reliability Coordinator and its Transmission Operators and Balancing Authorities.
R1.3. With other Reliability Coordinators, Transmission Operators, and Balancing Authorities as necessary to
maintain reliability.
R1.4. Where applicable, these facilities shall be redundant and diversely routed.
R2.
Each Reliability Coordinator, Transmission Operator, and Balancing Authority shall manage,
alarm, test and/or actively monitor vital telecommunications facilities. Special attention shall be given to emergency
telecommunications facilities and equipment not used for routine communications.
R3.
Each Reliability Coordinator, Transmission Operator and Balancing Authority shall provide a
means to coordinate telecommunications among their respective areas. This coordination shall include the ability to
investigate and recommend solutions to telecommunications problems within the area and with other areas.
R4.
Unless agreed to otherwise, each Reliability Coordinator, Transmission Operator, and Balancing
Authority shall use English as the language for all communications between and among operating personnel
responsible for the real-time generation control and operation of the interconnected Bulk Electric System.
Transmission Operators and Balancing Authorities may use an alternate language for internal operations.
R5.
Each Reliability Coordinator, Transmission Operator, and Balancing Authority shall have written
operating instructions and procedures to enable continued operation of the system during the loss of
telecommunications facilities.
6
Source: http://www.nerc.com/files/com-001-0.pdf
COM-001-0 Sidebar: February, 2012
DHS / Secret Service Investigation – Cell Phones
•
•
•
•
•
•
Story broke February 1, 2012 with charges filed in NY State
Cloning of accounts for Int’l calling market
Illegal $250,000,000.00 operation
Twelve actors indicted thus far
Accounts stolen: “..tens of thousands..”
Relationship to COM-001-0 R2: / EOP-008-0 R1.1
“..emergency telecommunications facilities..”
Potential Reliability Impact?
Any cloned accounts belonging to
RE’s for backup or reliability COM or
EOP (standby) could have been impacted
if the illegal use resulted in undetected deactivation of
legitimate devices.
Source &
Photo credit: http://blog.dhs.gov/2012/02/secret-service-investigates.html
7
WHY SONET?
Concepts
•
•
•
•
•
•
SONET Docs Frequently seen at audit
Typical utility services transported via SONET
Payloading
Redundancy: SONET Style
Ring configurations
COM-001-0 Emergency /non-routine concepts
and facilities
• Q&A
• References
8
SONET / SDH / TDM
Terminology
Synchronous
Optical
NETworking








9
Synchronous
Digital
Hierarchy
Time
Division
Multiplexing
SONET: First Standards Based Optical Transport Model
Conceptualized in the mid 1980’s
OSI Layer? (1.5)
Standards-based “Protection” offered at the Physical /Transport
Layer
Standards-based governance re: failover velocity
Alternative to point to point “TDM” circuit services
Compatibility with point to point “TDM” services
Ability to “concatenate” payloads to provide higher capacity
SONET Applications:
Transport for a Utility Network
10
RTU / PLC Overview
Source: http://members.iinet.net.au/~ianw/rtu.html
11
Typical RTU
Source: http://store.gedigitalenergy.com
Disclaimer: Use does not constitute endorsement
12
Primer: RTUs and COM Facilities
RTU State Diagram - MODBUS
Examples: RTUs typically use (rely on) a dedicated, TDM
point to point circuit to transfer bit signals between itself
and a destination
13
Primer: RTUs and COM Facilities
State Diagram – MODBUS Slave
Examples: RTUs typically use (rely on) a dedicated, TDM
point to point circuit to transfer bit signals between itself
and a destination
14
Primer: TDM
Time Division Multiplexing
Examples: RTUs typically use a
TDM point to point circuit of up to
56 kbps/sec. Typical phone or
“POTS” line is a common telco
service provision. Sometimes
more than 1 RTU shares a
common line.
Source: US DOT / FHA:
http://ops.fhwa.dot.gov/publications/telecomm_handbook/chapter2_03.htm
15
Primer: TDM and ANSI T1
Circuit Multiplexing
A T1 circuit (sometimes referred to as DS-1) uses TDM to
group and transport 24 POTs circuits, also known as ‘DS-0’s.
16
Source: US DOT / FHA:
http://ops.fhwa.dot.gov/publications/telecomm_handbook/chapter2_03.htm
Primer: TDM and ANSI T1
Time Division Multiplexing
Further, T-1 circuit channels can
include DS-0s independently
assigned for RTUs, Phones, or
Data, either singularly or in
groups.
Source: US DOT / FHA:
http://ops.fhwa.dot.gov/publications/telecomm_handbook/chapter2_03.htm
17
Primer: TDM and ANSI T1
Time Division Multiplexing
Source: US DOT / FHA:
http://ops.fhwa.dot.gov/publications/telecomm_handbook/chapter2_03.htm
18
Primer: TDM and ANSI Hierarchy
Time Division Multiplexing
Point to Point non-redundant provisioning
Further, T-3 circuit channels can
include 28 T-1channels
independently assigned for
Voice or Data Transport
Source: http://www.dcbnet.com/notes/9611t1.html
19
Primer: Typical TDM Pay loading within SONET
Transport is usually protected (N+1) path
Source: http://www.dcbnet.com/notes/9611t1.html
20
Primer: Typical SONET data payloading
Virtual Tributary (VT) concatenation
VT 1.5 ~ 1.5 Mbps/sec (1 Traditional T1 circuit)
Source: Various Public sources
21
Primer: Typical SONET data payloading
Virtual Tributary (VT) to OC Mapping
Source: http://www.tek.com/primer/sonet-telecommunicationsstandard-primer
22
Primer: Typical SONET data payloading
Virtual Tributary (VT) to STS-N Mapping
Source:
http://www.ieee802.org/3/10G_study/public/sept99/nicholl_1_0999.pdf
23
Typical Simplified SCADA Diagram
Source: Google search of non-copyrighted images
24
Generic Utility SCADA /Data Network
If you see this at audit think of what evidence is missing
within that “FAILOVER” oval
Source: Google search of non-copyrighted images
25
Generic Utility Diagram: Teleprotection
Source: Rad.com
Disclaimer: Use does not constitute endorsement
26
Typical Merged utility application of SONET/PSN
Source: Rad.com
Disclaimer: Use does not constitute endorsement
27
Typical Merged utility application of SONET
Source: http://www.gedigitalenergy.com
Disclaimer: Use does not constitute endorsement
28
Synchronous Optical NETwork
Why / Where SONET?
• SONET Typically used to:
- Efficiently aggregate multiple T1 or other services
- Offers redundancy for extended area transport
- Unique standards-based failover at a low layer
in the OSI model
- Can be made IEEE 1613 conformant for Substation
installations
- Easily add additional services or capability
- Extensible and Expandable capability
- Can payload and transport non-native TDM
services such as Ethernet or ATM alongside TDM
• MPLS is replacing SONET, but SONET likely to remain
a solid choice for Utility operations
29
Synchronous Optical NETwork
A Look ahead to MPLS – DHYB1 
Presenter’s note: Transition requires successful “circuit emulation” services paradigm
30
1
Don’t Hold Your Breath
Typical SONET Node or Multiplexer:
Note: Many products now
also include a “blade option”
for gigabit ethernet
(protected) over SONET for
OC-48+ line rates
Source: http://www.gedigitalenergy.com
Disclaimer: Use does not constitute endorsement
31
SONET Multiplexer Protection
Typical utility configuration (Node)
 Typically two power supplies (Either 48V DC or 125V DC)
 Alarming: Typically on circuit or ring failure
 Sometimes on the power supply, but not necessarily the
power (i.e., low voltage alarm)
 Data Center / Control Center often 48V
 Circuits protected 1+1 (blade level + circuit level)
 Independent PS, but often tied back to common station battery
 Ring configurations: UPSR / BLSR
 SONET: Ring protection: 50 ms failover or 3.0 Hz
(1 Hz = 16.67 MS)
32
SONET Multiplexer Protection
Utility Application: Orderwire Circuits
Frequent application for Substation-Substation
communications as alternative or backup to mobile
phone use.
33
SONET APS (UPSR )Automatic
Protection Switching – Normal
Normal: traffic is clockwise, Protection is counterclockwise
UPSR: Unidirectional Path Switch Ring
UPSR is easier to O & M but traffic between any adjacent
nodes consumes traffic on entire ring for protection
(translation: less bandwidth efficiency)
34
Source: http://www.sonet.com
SONET APS (UPSR )Automatic
Protection Switching – Protection
Failure: Fiber cut between Nodes C & B (normal & protect)
Traffic: D-A-C flows DC on protection
Traffic: A-B-C now flows A-D-C on Protection
Traffic: B-C now flows B-A-D on protection
Failover occurs within 50 MS per SONET specification
Source: http://www.sonet.com
35
SONET APS (UPSR, OC-3 )
Example w/ sample Path Protection State Report
Note: While this may seem cryptic it is also not unusual to see
this in compliance evidence. Path has failed but circuits are
active via “protection”. This can evidence “adequate and
reliable” during a failure condition (backhoe, storm, etc.)
36 Source: http://www.sonet.com
SONET APS (BLSR )
Bi-directional Line Switch Ring
Increased bandwidth efficiency, capacity is used only
between where traffic is added and dropped.
Source: http://www.sonet.com
37
SONET APS (BLSR )
Failure condition
Line level signaling indicates failure, traffic is re-routed.
Source: http://www.sonet.com
38
SONET and COM / Audit Evidence
•
•
•
•
•
•
•
SONET Documentation frequently seen at audit
Typically OFIM, but not always!
Used to support BES-applicable SCADA
Also for Protective Relay communication
Used for COM, CIP provisioning and transport
A word about “diversely routed”
Outside Plant considerations – future presentations:
- Building entrances
- Local power supplies
- AHJ issues
- What about Microwave / SONET Hybrid systems
• Typical Mapping of SONET to COM-001-0
39
COM-001-0 Review
Applicability: RC, BA, TOP (R1-R4, R5)
R1.
Each Reliability Coordinator, Transmission Operator and Balancing Authority shall provide
adequate and reliable telecommunications facilities for the exchange of Interconnection and operating information:
R1.1. Internally.
R1.2. Between the Reliability Coordinator and its Transmission Operators and Balancing Authorities.
R1.3. With other Reliability Coordinators, Transmission Operators, and Balancing Authorities as necessary to
maintain reliability.
R1.4. Where applicable, these facilities shall be redundant and diversely routed.
R2.
Each Reliability Coordinator, Transmission Operator, and Balancing Authority shall manage,
alarm, test and/or actively monitor vital telecommunications facilities. Special attention shall be given to emergency
telecommunications facilities and equipment not used for routine communications.
R3.
Each Reliability Coordinator, Transmission Operator and Balancing Authority shall provide a
means to coordinate telecommunications among their respective areas. This coordination shall include the ability to
investigate and recommend solutions to telecommunications problems within the area and with other areas.
R4.
Unless agreed to otherwise, each Reliability Coordinator, Transmission Operator, and Balancing
Authority shall use English as the language for all communications between and among operating personnel
responsible for the real-time generation control and operation of the interconnected Bulk Electric System.
Transmission Operators and Balancing Authorities may use an alternate language for internal operations.
R5.
Each Reliability Coordinator, Transmission Operator, and Balancing Authority shall have written
operating instructions and procedures to enable continued operation of the system during the loss of
telecommunications facilities.
40
Source: http://www.nerc.com/files/com-001-0.pdf
COM-001-0 Review – Futures
Presenter comment:
41
Audits may become more difficult as SONET infrastructure used for COM001 compliance becomes part of increasingly complex merged utility
communications networks.
Questions?
William Fletcher, CISA
Senior Compliance Auditor
Western Electricity Coordinating Council
bfletcher@wecc.biz
360-567-4058