Speaker: Doug McGinnis
Company: Exelon

Smart Grid Communication Architecture
Smart Grid at Exelon
2
Smart Home / Business
Smart Meters (AMI)
Smart Distribution System
Smart Utility
• Real-time usage and pricing
statistics
• A method to enable two-way
information flow
• Real-time reporting of status and
outages
• Home Area Network (HAN)
composed of smart devices
and appliances that know
the price of energy
• System status, customer
outage status, usage and
pricing signals delivered to
and from location
• Automated controls of relays and
reclosers. Efficient field force
management
• More efficient data
collection, processing and
back office functions
• Asset Monitoring
• Effective interconnection of
renewable energy sources
Leveraging integrated communication systems and information processing is critical
Customer End-Use

Advanced Pricing &
Billing

In-home Devices
PowerCo
PowerCo
1234567
1234
Vehicle Electrification

Plug-in Hybrid
Electric Vehicles
Customer Gateway

Advanced Metering
Infrastructure (AMI)
Grid Modernization

Distribution
Automation

Smart Substations

Renewable
Interconnection
1.
2.
3.
4.
5.
6.
7.
8.
Security – Robust end-to-end, aligned with NISTIR 7628
Deterministic – Smart Grid applications will share a logically isolated
deterministic communications infrastructure.
Interoperable – Industry standard protocols will be utilized with a
focus on migrating to IP/Ethernet consistent with industry direction
Privately Owned – Privately owned communications is preferred.
Eliminate telecom circuits, O&M cost savings.
No Unanalyzed Single Points of Failure (Self Healing) – The
communication architecture will be designed with no unanalyzed
single points of failure.
Communications Maintenance Management & Monitoring – Inherent
to the communications Architecture will be Communications
Maintenance Management & Monitoring, i.e. the ability to maintain,
monitor and control network devices.
Tiered Architecture – Unit/Distribution substations are linked to
larger Transmission substations in a hierarchical design through new
wireless technologies
Relay Protection Communications – Highest level of reliability and
availability including diverse backup paths.

Substation Application Portfolio – 6 application groups exist in the
substation and are logically partitioned over common transport
◦ Telemetry – RTU/IED – EMS/DMS communications (encrypted)
◦ CIP Telemetry – NERC CCA sites (encrypted)
◦ DA – Field Distribution Automation traffic aggregated by TGB
(encrypted)
◦ Enterprise – Business applications (email, VoIP, surveillance video)
◦ Security – Card readers
◦ AMI – TGB backhaul handoff to Core PoP

Tier 1 – Substation will be a point-of-presence to the SONET
backbone.
◦ SONET infrastructure provides Ethernet and TDM provisioning
◦ Routing capability to permit application provisioning over Ethernet
(Layer 3)
◦ Quality of service management, data prioritization

Tier 2 – RF bridge between TGB’s and non-PoP substations
◦ WiMax/Ethernet transport
◦ VLAN partitioning per application for separation

Relay Protection Communications
Parallel dual SONET ring configuration (OC3 & OC48)
with PoP’s at each substation
No relay channel failure with loss of a ring
Path diversity – loop topology with fast switching to
protected path
OC3 ring – Primary Relay communications
No other applications will use OC3
Static environment
OC48 ring – Backup Relay, TDM & Ethernet service
GigE provisioned to support Ethernet services
TDM circuits for relay protection, voice & serial
communications as required

Substation LAN
Access switch – VLAN provisioned (Layer 2)
No inter-application routing will be permitted
Telemetry network access/authentication will be through
core SCADA Firewall – NERC CIP compliant
TGB’s and other substation IP devices will be connected to
switch partitioned in their respective VLAN’s

Substation WAN
Router (layer 3) will interface with switch and will provision
Virtual Route Forwarding Tunnels (VRF)
6 VRF tunnels will be created for logical separation
VRF tunnels will be encrypted using Dynamic Multipoint VPN
(DMVPN)
IP addressing schema will be defined for entire substation
population based on application requirements
Substation WAN Communication Architecture
Application
Telemetry
CIP Tele
DA (Field)
Enterprise
Security
AMI
Encrypt
Yes
Yes
Yes
No
Yes
No
Plymouth
`
DFR PC
Substation
SONET
Ring
IP
ECS
DMACS
RS-232
1200/2400 Baud
Substation
Serial RS-232
or DNP/IP
DFR
IP
IED
SONET OC48 JMux
Ring
`
SEL 2020
SI PC
Serial TDM
RFL 9660
RTU
MOB
`
TGB DA
DFR PC
TGB AMI
Tier 2
AMI TGB
ECS
Substation
DA Recloser
DMACS
RS-232
1200/2400 Baud
`
Work Station
Network
Core
VRF
Tunnels
Substation
Telemetry
VoIP
VLAN extended to
switch per Application
Camera
CardReader
Switch
RTU
SCADA
CIP Telemetry
Firewall Router
Gigabit
Ethernet
Field DA
Enterprise
Core
Router
Switch
Security
AMI
Security
DA TGB
Incorporates Layer 3 VRF Tunneling
and Dynamic Multipoint VPN
AMI TGB
Ethernet based devices
Enterprise
Firewall
AMI/RNI

Transport the Smart Grid application portfolio
◦ AMI backhaul – (200kbps/TGB @ 70 Aggregation
pts)
◦ Distribution Automation (100kbps/TGB @
Aggregation pts)
◦ Substation Telemetry (56kbps/substation)
◦ Voice/Video (~1Mbps per video stream)

Application Traffic Considerations
◦
◦
◦
◦
Bandwidth consumption (5-20Mbps)
Latency sensitivity (QoS tagging)
Security (PKI)
Logical provisioning of applications (VLAN tagging)
Spectrum Options – Licensed vs. Unlicensed
Unlicensed (ISM)
Licensed
• Free
• High cost – Spectrum Market
• Limited Power
• Strict FCC deployment rules
• Uncontrolled Noise Floor
• High Power Operation
• Uncontrolled Interference
• Low/controlled Noise Floor
• 900 MHz – Hi Noise
• Interference Remedy
• 2.4 GHz – Hi Noise
•700 MHz Public Safety
• 3.65 GHz (Lite License)
• 900 MHz
• 5.X GHz
• 2.3 GHz

Free-space path loss is proportional to the square
of the distance between the transmitter and
receiver, and proportional to the square of the
frequency of the radio signal.
◦ Higher the frequency – Lower the signal
propagation
◦ What type of coverage are you planning for?
 Blanket umbrella (target lower frequencies with
minimal interference characteristics)
 Surgical microcell (target higher frequencies)
 Urban (target lower frequencies with minimal
interference characteristics)
 Suburban/rural (depends on the type of coverage,
Blanket vs. Surgical)
Frequencies
Requirements
700Mhz
900Mhz
2.3Ghz
3.65GHZ
5.8Ghz
6-11Ghz
Risk
High
High
High
Medium
Low
Low
Cost
Low
Low
High
Low
Low
High
Excellent
Adequat
e
Good
Good
Good
Excellent
Limited
Good
Growing
Growing
Good
Good
√
√
√
No
No
√
Unlicensed
No
√
No
√
√
No
Lightly
No
No
No
√
No
No
√
√
√
√
√
√
No
No
No
No
√
√
Point-to-Multi Point
√
√
√
√
No
No
Overall Ranking
2
6
5
1
3
4
Coverage
Equipment Availability
Licensed
Availability – PECO area
Point-to-Point
Ranking: 1 high - 6 low
UNIT
SUBS
MOB
Tier II
Point-to-Point
Point-to-Multipoint
RTU’s
Corporate
SONET
RING
Plymouth
Stimulus
Substations
Selections
DAX Switching
Substation SONET
Ring x # of
Substations
Berwyn
Skyscraper
AMI Meters
Point-to-Point
TGB Collector to
Tier II Interface
Point-to-Point
Tier II
AMI Collector
Point-to-Point
UNIT
SUB
UNIT
SUB
Tier II Devices and wireless connections
labeled in RED
Two prevailing backhaul Standards
WiMax 802.16
WiFi 802.11
• Long Distance 5-10 Miles
• Shorter range 1-2 Miles
• P2P & P2MP
• Meshing capability
• Time Slotted Access
• Contention Access DSSS
• Mobility (802.16.e)
• 22 MHz Channels
• 5-10 MHz Channels
• 802.1Q (VLAN, QoS)
• 802.1Q (VLAN, QoS)
• Security EAP/TLS/WPA 2
• Security EAP/TLS/AES



Backhaul 100 AMI/DA collector sites
Provide backhaul of non-fiber substations
Requires surgical deployment, not blanket
coverage
◦ Point-to-point links
◦ Microcell canopies

Decided WiMax using 3.65 GHz Lite-License
◦ Minimal noise at this time
◦ Do not require full territory coverage
◦ Point-to-point links where necessary
C
Tier II
Wireless Backhaul Simple Network Design
Security
Telemetry
(undecided)
C
Point-to-Multipoint
Point-to-Point
Unit Substation
(undecided)
AMI
TGB Collection
Points
Tier I
Core
Network
T
CIP RTU
Serial RTU
RTU
(BSTUN)
(Sage 2300)
RTU
Converter
Security Devices
Enterpise Devices
AMI
DA
(Card Readers, Cameras)
CGS 2510 Ethernet Switch
`
Telemetry VLAN
CIP Telemetry VLAN
Field DA VLAN
Security VLAN
Enterprise VLAN
AMI VLAN
Cisco 2010 Grid
Router
SONET
GigE Ring
MOB
Plymouth
Cisco 3945
Cisco 3945
(RTU Termination via BSTUN)
(RTU Termination via BSTUN)
Cisco 3945
Cisco 3945
Switch
Switch
Firewall
Firewall
BSTUN
BSTUN
SCADA VRF
Security VRF
Enterprise VRF
AMI VRF
Enterprise Application Services
Enterprise Application Services
(Voice, Video, email, IM, CRM, AD, DNS)
(Voice, Video, email, IM, CRM, AD, DNS)
Netflow
Sy
slo
CSM
RADIUS
(ACS)
g
SNMP Traps
Syslog
Ne
w
tflo
SDEE, RDEP
Retrieve IPS Alerts