Teleprotection over an IP/MPLS
Network
Technical validation
ir. Lieven LEVRAU
Alcatel-Lucent IP Division
April 4th, 2011
•
•
•
•
IP/MPLS-based Utility Networks
Challenges for IP/MPLS in teleprotection
Other design considerations
Conclusion
© Alcatel-Lucent 2011 All Rights Reserved
1
IP/MPLS-based Utility Networks
All Rights Reserved © Alcatel-Lucent © 2011
IP/MPLS-Based Utility Network
Power
Generation
7705 SAR
Corporate HQ / NOC
OS6855
LAN
Internet Omni PCX
5620 SAM
SCADA
IED
7750 SR
7750 SR
IP/MPLS
Network
7750 SR
Substation
7705 SAR
Substation
7750 SR
Servers Collaboration
Tool
NOC
Transmission
7705 SAR
7705 SAR
IED
7705 SAR
RTU LAN
OS6855
Video
Surveillance
TPR
LAN
TDM
Video
Surveillance
TPR
4 | MPLS Network for Teleprotection | Sep 2009
All Rights Reserved © Alcatel-Lucent © 2011
How IP/MPLS Address the challenges of the Utility Networks?
Reliability
High Availability, Fast Reroute, end-to-end,…
Traffic Isolation
IP/MPLS based - L2-VPNs, VLL, L3-VPNs, PWE
Multiple Legacy technologies
FR, ATM, TDM, Serial, analog voice, synchronous, …
Security
Traffic isolation, Rtng protocols authentication, L2/L3
Encryption + dynamic secure Key distribution, DoS
Unified management
End-to-end secure management of network and
services
Quality of Service
Guaranteed bandwidth, low latency and jitter
`
Enhanced QoS on different network layers
All Rights Reserved © Alcatel-Lucent © 2011
2
Challenges for IP/MPLS in
teleprotection
All Rights Reserved © Alcatel-Lucent © 2011
Using MPLS Network for Teleprotection
Substation
Substation
kV
G.703
MUX
E1/T1
∆t
TPR
TPR
E&M
RS-232
Ethernet
E&M
RS-232
Ethernet
IP/MPLS
7705 SAR
7710 SR
7750 SR
G.703
MUX
E1/T1
7705 SAR
Teleprotection relay (TPR) signals must be transferred reliably and fast with low latency
End to end delay = telecom network latency + teleprotection equipment delay
TDM over MPLS for legacy support (requires integration of legacy interface in MPLS node
to limit and control end to end delay)
VPLS for IEC 61850 (requires high reliable Layer 2 transport)
7 | MPLS Network for Teleprotection | Sep 2009
All Rights Reserved © Alcatel-Lucent © 2011
Main Challenges for Teleprotection
End to End Delay
Depending on vendor equipment
May include packetisation/depacketisation
Jitter
Variation of delay in certain circumstances such as
traffic burst on network
Asymmetry
Delay variations between transmit and receive
Resiliency
The impact of a failure in the network shouldn’t
be noticed by application
Denial of Services
How DoS/DDoS can affect applications
Synchronization
TDM applications need
end to end synchronization
Quality of Service
Guaranteed bandwidth, low latency and jitter
Enhanced QoS on different network layers
All Rights Reserved © Alcatel-Lucent © 2011
End-to-End TDM Transport Model (only left-to-right
direction shown)
TDM Packets moving in this direction
DS1
Access
Circuit
DS1
LIU
Data
Packetization
Si
g
Packetization
• As TDM traffic from the
Access Circuit (AC) is
received, it is packetized
and transmitted into the
PSN
• Two modes of operation:
• CESoPSN (RFC5086) for
structured nxDS0/64k
channels
• SAToP (RFC4553) for
unstructured T1
9 | MPLS Network for Teleprotection | Sep 2009
GigE
Packet
Switched
Network
(PSN)
GigE
Network
• Fixed delay
• Packet transfer delay based
on link speeds and distances
from end to end
• Variable delay
• the number of and type of
switches
• queuing point in the
switches
• QoS is key to ensure
effective service delivery
All Rights Reserved © Alcatel-Lucent © 2011
Jitter Data
Buffer
Si
g
DS1
LIU
DS1
Access
Circuit
Playout
• TDM PW packets are
received from the PSN
and stored into its
associated configurable
jitter buffer
• Play-out of the TDM data
back into the AC when it’s
at least 50% full
Latency
Using MPLS Network for Teleprotection
kV
Substation
Substation
G.703
MUX
E1/T1
∆t
TPR
TPR
E&M
RS-232
Ethernet
E&M
RS-232
Ethernet
IP/MPLS
7705 SAR
7750 SR
TPR relay signals must be transferred in < ∆t
∆T = 1 cycle at 50Hz or 60Hz = 20ms or 16ms
End to end delay = telecom network latency
7750 SR
G.703
MUX
E1/T1
7705 SAR
The total end-to-end latency is calculated
by summing the packetization delay (PD),
network delay (ND) and jitter buffer delay
(JBD) as shown here:
Total Latency = PD + ND + JBD
+ teleprotection equipment time
latency = packetization delay + network
delay + jitter buffer delay
10 | Teleprotection over IP/MPLS - Validation - March2011
– e.g. PD of 2 ms (16 T1 frames/packet), ND of 3 ms, JBD
of 4 ms
– Total Latency = 2 + 3 + 4 = 9 ms
All Rights Reserved © Alcatel-Lucent © 2011
Using MPLS Network for Teleprotection
Latency
kV
Substation
Substation
G.703
MUX
E1/T1
∆t
TPR
TPR
E&M
RS-232
Ethernet
E&M
RS-232
Ethernet
IP/MPLS
7705 SAR
7750 SR
7750 SR
G.703
MUX
E1/T1
7705 SAR
TPR relay signals must be transferred in < ∆t
∆T = 1 cycle at 50Hz or 60Hz = 20ms or 16ms
End to end delay = telecom network latency + teleprotection equipment time
Telecom network latency = packetization delay + network delay + jitter buffer delay
11 | Teleprotection over IP/MPLS - Validation - March2011
All Rights Reserved © Alcatel-Lucent © 2011
Calculation of Latency for Teleprotection
Latency
Latency is mainly at the edge where low speeds are present (Serial / E1 /
100 FX)
Latency in the core depends on number of nodes but mainly transmission
delays
Each node adds a maximum of :
150µs (eqpt latency)
10µs (transmission of 1500 Bytes over a Gigabit link)
3µs / km : speed of light transmission over fiber
Example of a connexion between 2 TPRs :
Over 1000km – traversing 10 nodes : 4ms
12 | Teleprotection over IP/MPLS - Validation - March2011
All Rights Reserved © Alcatel-Lucent © 2011
Jitter
Jitter in normal operations may come from the equipments themselves
This is minimal (less than 1% of router latency) and compensated by jitter
buffer.
Jitter happens in non normal conditions such as congestion.
This can be solved by applying the correct QoS parameters to the node.
The implementation should allow total control of the bandwidth required per
application, minimizing jitter.`
Rate Limit TPR to 100Kbps
TPR gets to EF class
Rate Limit CCTV to 6Mbps
CCTV to BE Class
TPR
Per SAP policing
Per interface queuing
Ingress
Egress
Make Sure that no application can go higher than
expected bandwidth. Via Rate limit per SAP
Ensure that TPR application always gets the
priority
H-QoS
13 | Teleprotection over IP/MPLS - Validation - March2011
All Rights Reserved © Alcatel-Lucent © 2011
Impact of failure
Today’s teleprotection applications were developed towards SDH/PDH 50ms
failover time.
The impact of a failure (node or link) can have huge impact in case the failure
exceeds the 50ms.
MPLS FRR technology allows 50ms failover time in any failure scenario.
These 50ms apply only in the core of MPLS network, but not in the case of the
Access switch failing.
14 | Teleprotection over IP/MPLS - Validation - March2011
All Rights Reserved © Alcatel-Lucent © 2011
Conclusion on network failure impact
Failover in the backbone can be limited to 50ms with FRR
FRR has to be implemented in the first node to minimize failure risks.
All ALU Service routers support FRR.
Integrating TDM in MPLS (as in 7705 SAR) brings the FRR to the application and
limits the failure risks (compared to a 2 box solution)
Providing extended rapid failover scenarios may be key in many applications.
Support very rapid convergence time even in case of non direct connectivity
(BFD), for example with Microwave links
Implementing MC-LAG, G.8032 and BGP-MH for IEC 61850 greatly enhances
total availability of the applications.
All Rights Reserved © Alcatel-Lucent © 2011
3
Other design considerations for
Teleprotection
16 | Teleprotection over IP/MPLS - Validation - March2011
All Rights Reserved © Alcatel-Lucent © 2011
Packet flow asymmetry
LSPs are unidirectional
Topology changes or mis-configuration may result in different path being
used in both directions.
Only trouble shooting tools may discover that.
5620 CPAM allows this detection and raises alarms.
Bypass
Green and
actual
Red LSP
path
follow
different
Path
Active
Bypass
Path
Logical
Link
Tracking of LSP status and history
17 | Teleprotection over IP/MPLS - Validation - March2011
Tracking of LSP Path for a given service
All Rights Reserved © Alcatel-Lucent © 2011
Synchronization
Information need to be delivered with time
precision from an application standpoint.
Many TDM applications require clock synchronization
(as SDH)
Some applications require Time of Day type of
synchronization
Packet networks can deliver clocking through
different techniques :
18 | Teleprotection over IP/MPLS - Validation - March2011
All Rights Reserved © Alcatel-Lucent © 2011
4
Summary
All Rights Reserved © Alcatel-Lucent © 2011
Conclusion
Teleprotection is the most stringent application that
can be transported over networks due to :
Low delays requirements, very low jitter requirements
Impact of a failure in the application
Alcatel-Lucent demonstrates that its IP/MPLS can be used as
the next generation network for such applications with :
Native TDM integration (incl. interfaces, synchronization, …)
Very high resistance to potential failures in network
Several management tools to anticipate, control and trouble
shoot network
IP/MPLS is the foundation to prepare the Smart Grid data explosion, and manage
the transition of existing mission critical applications
20 | Teleprotection over IP/MPLS - Validation - March2011
All Rights Reserved © Alcatel-Lucent © 2011
www.alcatel-lucent.com