A Lightning Fast Alternative for Teleprotection
October 2011
Presented By:
Motty Anavi – VP Network Architecture
ETX Portfolio , 2011 Slide 1
Transport Technologies ‐ Perceptions
• Sonet
– Static design – Built‐in ring redundancy
– Accurate timing
– Inefficient in delivering packet Traffic
– Carrier grade technology
• Ethernet
– Cheap
– Unreliable
– Efficient in delivering packet technologies
– No reliable redundancy
– Enterprise grade technology
ETX Portfolio , 2011 Slide 2
Ethernet – Changing Capabilities
•
•
•
•
•
•
Cheap
Efficient transport for Packet
Resilient and redundant
Predictable and deterministic
Accurate timing Can transport anything over it
ETX Portfolio , 2011 Slide 3
What’s a “Smart Grid?”
US Department of Energy
ETX Portfolio , 2011 Slide 4
ETX Portfolio , 2011 Slide 5
GAO Report – Released January 2011
http://www.gao.gov/products/GAO‐11‐117
ETX Portfolio , 2011 Slide 6
Carrier Ethernet Defined
Carrier Ethernet for the Business Users:
• The MEF has defined Carrier Ethernet as
• A ubiquitous, standardized, carrier‐
class Service and Network defined by five attributes that distinguish it from familiar LAN based Ethernet ETX Portfolio , 2011 Slide 7
Carrier Ethernet Defined
Carrier Ethernet for Service Providers:
• A set of certified network elements that connect to transport Carrier Ethernet services for all users, locally & worldwide
• Carrier Ethernet services are carried over physical Ethernet networks and other legacy transport technologies
ETX Portfolio , 2011 Slide 8
What is Carrier Ethernet?
• Question:
– “Is it a service, a network, or a technology?”
• Answer for an end‐user
– It’s a Service defined by 5 attributes
• Answer for a service provider – It’s a set of certified network elements that connect to transport the services offered to the customer
– It’s a platform for value added services
– A standardized service for all users
ETX Portfolio , 2011 Slide 9
Carrier Ethernet Terminology
• User to Network Interface (UNI)
– Physical interface/demarcation between service provider/Cable Operator/Carrier/ and subscriber
• Ethernet Virtual Connection (EVC)
– Logical representation of an Ethernet service as defined by the associate between 2 or more UNIs
• Network to Network Interface (NNI)
– Demarcation between carrier Ethernet networks operated by one or more carriers
UNI, EVC and NNI are the Fundamental Constructs of an Ethernet Service
ETX Portfolio , 2011 Slide 10
MEF Carrier Ethernet Terminology
‐ Ethernet Virtual Connection (EVC)
• An Ethernet Service Instantiation
– Most commonly identified via 802.1ad S‐VLAN ID
• Connects two or more subscriber sites (UNIs)
– Can multiplex multiple EVCs on the same UNI
• Three types of EVCs defined by MEF
– Point‐to‐Point, Multipoint‐to‐Multipoint, Rooted Multipoint (Point‐to‐Multipoint)
– Typically distinguished by VLAN tags or Q‐in‐Q
UNI
(port)
e.g. 1GE
EVC1
UNI
EVC2
ETX Portfolio , 2011 Slide 11
Carrier Ethernet: Three Ethernet Service Types
• E‐Line Service Type
– Ethernet Private Lines
– Virtual Private Lines (site‐to‐site Layer 2 VPNs)
– Ethernet Internet Access
• E‐LAN Service Type
– Multi‐site Layer 2 VPNs
– Transparent LAN Service
• E‐Tree Service Type
– Point‐to‐Multipoint Infrastructure
– Triple play backhaul
– Cell sites backhauled to mobile switching center
ETX Portfolio , 2011 Slide 12
Services Using E‐Line Service Type
• Ethernet Private Line (EPL)
– Replaces a TDM Private line
– Port‐based service with single service (EVC) across dedicated UNIs providing site‐to‐site connectivity
– Typically delivered over SDH (Ethernet over SDH)
– Most popular Ethernet service due to its simplicity
Storage Service Provider
UNI
UNI
CE
UNI
CE
Carrier Ethernet Network
ISP
POP
Internet
UNI
Point‐to‐Point EVCs
CE
ETX Portfolio , 2011 Slide 13
Services Using E‐Line Service Type
• Ethernet Virtual Private Line (EVPL)
– Replaces Frame Relay or ATM L2 VPN services
• To deliver higher bandwidth, end‐to‐end services
– Enables multiple services (EVCs) to be delivered over single physical connection (UNI) to customer premises
– Supports “hub and spoke” connectivity via Service Multiplexed UNI at hub site
• Similar to Frame Relay or Private Line hub and spoke deployments
Service Multiplexed Ethernet UNI
UNI
UNI
CE
Carrier Ethernet Network
CE
UNI
Point‐to‐Point EVCs
CE
ETX Portfolio , 2011 Slide 14
EVC Service Attributes
Bandwidth Profiles per EVC (service) and per CoS
– CIR (Committed Information Rate)
• CIR assured via Bandwidth Reservation and Traffic Engineering
– EIR (Excess Information Rate)
• EIR bandwidth is considered ‘excess’
• Traffic dropped at congestion points in the network
– CBS/EBS (Committed/Excess Burst Size)
• Higher burst size results in improved performance
10Mbps UNI
(port)
UNI
EVC1
EVC2
CoS 6
1Mbps CIR for VoIP
CoS 2
6Mbps CIR for VPN data traffic
3Mbps for Internet Access
EVC‐2
EVC‐1
EIR
EVC‐3
BWPs can divide bandwidth per EVC (service) over a single UNI
– Multiple services over same port (UNI)
– CoS markings enable the network to determine the network QoS to provide
CIR defines the assured bandwidth EIR improves the network’s Goodput
ETX Portfolio , 2011 Slide 15
Ethernet Virtual Private Line (EVPL) Example
Internet Service Provider (ISP)
Service Multiplexed UNI
VLAN 178  Blue
VLAN 179  Yellow
VLAN 180  Green
VLAN 2000  Green
ISP
Customer 3
VLAN 2000  Blue
ISP
Customer 1
VLAN 2000  Yellow
ISP
Customer 2
•Efficient use of ISP router ports
•Easy configuration at ISP customer sites
• This port and VLAN 2000 (or even untagged) to ISP
ETX Portfolio , 2011 Slide 16
Drivers for Ethernet OAM
• OAM benchmarks
– Set by TDM and existing WAN technologies
• Operational Efficiency
– Reduce OPEX, avoid truck‐rolls
– Downtime cost
• Management Complexity
– Large Span Networks
– Multiple constituent networks belong to disparate organizations/companies
ETX Portfolio , 2011 Slide 17
Problem Taxonomy
Fault Management
Fault Detection
Fault Verification Fault Isolation
Fault Recovery
Fault Notification
Performance Management
Frame Loss Measurement Delay Measurement Delay Variation Measurement Availability Measurement
Carrier
Ethernet
Services
Configuration Management
Service Provisioning
ETX Portfolio , 2011 Slide 18
Ethernet OAM
• IEEE 802.1ag
– Connectivity Fault Management (CFM)
– Also referred as Service OAM
• IEEE 802.3ah (clause 57)
– Ethernet Link OAM
– Also referred as 802.3 OAM, Link OAM or Ethernet in the First Mile (EFM) OAM
• ITU‐T Y.1731
– OAM functions and mechanisms for Ethernet‐based networks ETX Portfolio , 2011 Slide 19
Standards:
Operations Admin. & Maintenance
A summary of available Ethernet OAM mechanisms
End‐End/Last Segment
Performance Monitoring
Type
Fault Propagation
ETX Implementation
IEEE 802.3‐2005 (formerly 802.3ah)
Single Segment
No
Link
No
Standard
IEEE 802.1ag
End‐to‐End
No
Connectivity
No
Standard
ITU‐T Y.1731
End‐to‐End
Yes
Service
Yes
Standard
ETX Portfolio , 2011 Slide 20
CFM Functionality
• a) CC – Continuity Check
• Multicast unidirectional heartbeat • Use: Fault Detection
• b) Loopback
• Unicast bi‐directional request/response
• Use: Fault detection, verification • c) Traceroute
• Multicast request/unicast responses
• Use: Fault Isolation
ETX Portfolio , 2011 Slide 21
Other OAM Functionality (out of scope for .1ag)
• d) Discovery
– i) Service • e.g. discover all PEs supporting common service instance
– ii) Network
• e.g. discover all devices (PE and P) common to a domain
• e) AIS/RDI
• Use: E<‐>W Fault propagation, Alarm Suppression
• f) Performance Management
ETX Portfolio , 2011 Slide 22
Ethernet SLA Tools Example
Connectivity
Verification
Fault
Detection &
Isolation
• IEEE 802.3ah heartbeat
• IEEE 802.1ag CC
• IEEE 802.1ag LB, MAC Ping
• IEEE 802.1ag LT, MAC Trace-route
• IEEE 802.1ag LB, MAC Ping
• L3 Ping and Trace-route
Fault
Propagation
• Subscriber port shutdown
• ITU-T Y.1731 RDI
• IEEE 802.3ah Dying Gasp, SNMP Trap
Diagnostic
Loopbacks
• L1 loopback
• IEEE 802.3ah loopback
• L2/L3 loopback, MAC Swap, per EVC/VLAN/Src. Add.
Performance
Management
• ITU-T Y.1731/RFC2544: Packet Loss, PD, PDV,
• Per EVC.cos Statistics
ETX Portfolio , 2011 Slide 23
Anything Over Ethernet
ETX Portfolio , 2011 Slide 24
What is Pseudowire (PW)?
• Pseudo = Simulated, Seemingly
• Emulation of a native service over a Packet Switched Network (PSN). • The native services can be ATM, TDM, Frame Relay or ETH, while the PSN can be ETH, IP or MPLS.
• Supports voice, data and video
• Provides a transparent tunnel through the PSN
• Provides clock distribution and synchronization over PSN
ETX Portfolio , 2011 Slide 25
What is Pseudowire (PW)? PSN Network
ATM VCC
PW-GW
PW-GW
ATM VCC
HDLC
HDLC
TDM
TDM
ETX Portfolio , 2011 Slide 26
PW Switching‐ IP/ETH
1. User services are mapped to PW
VCC/E1
PW-Label
2. Adding IP Header
E1-TDM
VCC/E1
PW-Label
Switching is
done according
to the IP
addresses
E1-TDM
IP Header
PW-GW
VCC
PW-GW
ETH
ETH
IP/ETH
Network
VCC
VCC
VCC
3. The IP Header is stripped
VCC/E1
PW-Label
IP Header
4. PW Label mapped back to
user services
VCC/E1
PW-Label
ETX Portfolio , 2011 Slide 27
PW Switching – MPLS/ETH
1. User Services are mapped to PW
VCC/E1
Switching is
done according
to the Tunnel
label
PW-Label
2. Adding Tunnel Label
E1-TDM
VCC/E1
PW-Label
E1-TDM
Tunnel Label
PSN-GW
VCC
PSN-GW
ETH
ETH
VCC
MPLS
VCC
VCC
3. The Tunnel Label is stripped
VCC/E1
PW-Label
Tunnel Label
4. PW Label mapped back to
user services
VCC/E1
PW-Label
ETX Portfolio , 2011 Slide 28
PW Products Offering • Preserves investment in legacy TDM/ATM equipment in migration to PSN
• Lowers OpEx of TDM/ATM service by utilizing packet infrastructure
• Ultimate voice quality (no compression, no Silence Suppression)
TDM
Service
PW GW
Packet
Network
PW GW
TDM
Service
Pseudowire
ETX Portfolio , 2011 Slide 29
Timing Over Ethernet
ETX Portfolio , 2011 Slide 30
IEEE 1588
IEEE-1588 is a standard for a Precision Clock Synchronization
Protocol for Networked Measurement and Control Systems
 Defines a Precision Time Protocol (PTP) designed to
synchronize real-time clocks in a hierarchical
distributed system
 Intended for LAN using multicast communications
 Targeted accuracy of microseconds or submicrosecond (v1)
 v1 approved in September 2002 and published
November 2002
 v2 approved in June 2007
 RAD is an active member starting September 2006
ETX Portfolio , 2011 Slide 31
What is IEEE1588v2?
• IEEE1588v1 originally was designed for time distribution for the measurement and control industry.
• IEEE1588v2 is designed to distribute frequency and time to a higher accuracy and precision, to the scale of nanoseconds and fractional nanoseconds.
• The protocol operates over packet switched networks. The standard is currently defined to run over IEEE 802.3, UDP/IPv4, UDP/IPv6, DeviceNet, ControlNet and PROFINET.
• Designed to operate automatically to establish master slave hierarchy for time distribution. (not for Telecomm industry)
• Introduces “Transparent Clocks” to overcome the network’s delay variation.
ETX Portfolio , 2011 Slide 32
What is IEEE1588v2? (cont.)
• IEEE 1588v2 is designed to answer the timing needs of the measurement, automation, control and telecomm industries as well as home entertainment (802.1as) and military applications.
ETX Portfolio , 2011 Slide 33
Why IEEE1588v2?
• IEEE1588v1 did not overcome the network’s delay variations.
• NTP has accuracy of up to milliseconds, Client‐Server based, which increases the network traffic. Generates a logical clock.
• Synchronous Ethernet is designed to distribute frequency only and requires the entire clock distribution path to be Sync‐E compliant.
• IEEE1588v2 is a Master‐Slave hierarchy, with fractional nanosecond precision, hardware time stamping PTP packets and manipulation of the local clock, enables smooth transition from unsupported networks and removes PDV using Transparent Clocks.
ETX Portfolio , 2011 Slide 34
Protection Over Ethernet
G.8031
ETX Portfolio , 2011 Slide 35
G.8031 Applications – End to End path protection
•
•
•
•
Redundancy on S‐Tags in the network
APS is running over one standby EVC only
Revertive and Non revertive modes
End to end service shell be maintained – TLS , Accesses to L3 VPN • CCM or ETH‐AIS is used to trigger protection event X
Customer Premises
Metro /
VPLS Ethernet
NID
CPE
Customer Premises
L2PE
NID
L2PE
CPE
L2PE
Online EVC
End to End path protection
Redundant EVC
ETX Portfolio , 2011 Slide 36
Slide 36
G.8031 Protection • Protection as per ITU‐T G.8031
– 1:1 Mode
– Unidirectional Only – Using APS messages • Triggers
– Port Signal loss – CCM LOC , ETH‐AIS • Protection time – 10ms protection for one pair of EVC
– As low as 40ms protection 4 pairs of EVCs • Topologies – EVC protection with one fiber (both EVC’s running on the same Fiber) – EVC protection with 2 fiber each path on different fiber (dual link) – EVC protection with dual fiber working with MC‐LACP to dual PE
– EVC protection with Dual NTU (Future development) ETX Portfolio , 2011 Slide 37
G.8031 Applications – Segment Protection Customer Premises
X
NID
CPE
Confidential
L2PE
L3PE
Ethernet
Access Network
Metro /
VPLS L2PE
GE
Secured access segment
•
•
•
•
Redundancy on S‐Tags in the network
Revertive and Non revertive modes One EVC failure can trigger on all other EVC’s
End to end service is maintained CPE
NID
– TLS , Accesses to L3 VPN Online EVC
Redundant EVC
APS MSG.
ETX Portfolio , 2011 Slide 38
Challenges: Next Generation Migration Business Aspect
Manage Migration Economically
• Avoid CapEx hikes related to increase in number of network devices:
– Continue using legacy installed‐base while introducing IP/Ethernet devices
– Ensure service quality for mission critical apps (e.g., Teleprotection)
• Avoid over‐burdening network operations and management due to TDM/PSN transport co‐existence
• Reconcile required investment in SDH/SONET equipment with decommissioning plans
• Maintain smooth operation of current networks, despite discontinued vendor products
ETX Portfolio , 2011 Slide 39
Challenges: Next Generation Migration Technology Aspect
Smart Communications over Packet
• Service assurance for mission critical apps in PSN environment:
– Low end‐to‐end delay – High Availability
– SDH/SONET‐level Resiliency
• Differentiated quality of service for SCADA, video surveillance, voice, Teleprotection, radio and data traffic • Remote operations, administration and maintenance (OAM) for fault management and lower OpEx
• Efficient connectivity for IEC 61850 intelligent electronic devices (IEDs)
ETX Portfolio , 2011 Slide 40
Substation Connectivity
• Legacy services such as TDM PBX, Analog voice, legacy RTU and SCADA devices.
• Modern Ethernet based SCADA and Industrial Ethernet with advanced processing and security features
• Ethernet connectivity over SDH/SONET and or Ethernet
• Ethernet extension over fiber, copper and wireless
• Easy migration from TDM to PSN
SCADA
Control Center
Teleprotection
SCADA
MUX
ETH
MUX
Voice
SCADA
PSN
SDH/SONET
PBX
Voice
MUX
MUX
SDH/SONET/GBE
Wireless
SCADA
Fiber
Wireless
MUX
SHDSL.Bis
PBX
SCADA
DSL MODEM
OP MUX
ETX Portfolio , 2011 Slide 41
Teleprotection
• Deliver Teleprotection signals with mission‐critical accuracy over dedicated fiber, TDM or IP
• C37.94‐compliant Teleprotection communication channels allow reliable transmission by minimizing data errors due to EM and RF interference, or ground potential rise (GPR)
•Ultra‐low end‐to‐end propagation delay supports immediate delivery of Transfer Trip commands from protective relay/contact transfer to remote‐end substations
•Maintain performance levels when migrating to packet networks with hard QoS, as well as robust latency and jitter protection
ETX Portfolio , 2011 Slide 42
Migration To Ethernet Infrastructure
Required Services
SDH/SONET
Legacy to SDH/SONET
Legacy to PSN
PSN
Aggregation Network
Data
ETH to PSN
ETH to SDH/SONT
VoIP
NMS
Access Aggregation
Access
ETX Portfolio , 2011 Slide 43
Summary
• PSN is becoming an increasingly reliable network
• Other legacy features are developed at the same time
• Carrier Ethernet OAM standards now allow Ethernet to provide as reliable an infrastructure as SONET networks did in the past.
• Ethernet OAM enables the quick fault isolation and verification of network infrastructure
• Today’s Ethernet OAM technologies detect faults in under 10ms and trigger switchover
• The most complex and critical application in utility networks is Teleprotection
• Ethernet OAM allows utilities today to make use of these networks for both smart grid applications as well as for future migration of all their existing applications
ETX Portfolio , 2011 Slide 44
Thank You For Your Attention
Motty Anavi
VP Business Development
Email: motty@radusa.com
Phone: (201) 378‐0213
www.rad.com
ETX Portfolio , 2011 Slide 45