The POWERful Choice –
Carrier Ethernet or MPLS
For Power Utilities
Yaakov (J) Stein
CTO
SONET/SDH is being phased out
SONET technology is widely deployed, but
•
SONET technology is aging
•
SONET equipment is becoming obsolete and hard to find
•
SONET is hard to maintain (parts hard to obtain and expensive)
•
finding staff with SONET expertise is becoming ever more difficult
•
no new rates/functionality/standards/applications are being developed for SONET
Modern packet-based networks (based on Ethernet, MPLS, and IP)
•
are the present and future
•
are broadband and becoming even more so
•
are less expensive (both CAPEX and OPEX) and more flexible
•
are being actively extended (e.g., migration to 61850)
But there are open questions
•
can all the relevant services be migrated to packet (e.g., teleprotection, synchrophasors)?
•
which packet-based network to choose ?
The POWERful Choice - Carrier Ethernet or MPLS
2
The options
• Carrier Ethernet
–
–
–
–
–
Based on most popular technology in the world
Look and feel similar to SONET/SDH networks
Mature carrier-grade technology
Support for synchronization
Network security mechanisms available
• MPLS
–
–
–
–
Core network technology
Inherits rich IP control plane
Deterministic paths available (MPLS-TE)
Has no inherent network security
• MPLS-TP
– Based on MPLS, but adds mechanisms patterned after Carrier Ethernet
• OAM and protection switching (including rings)
– Look and feel similar to SONET/SDH networks
– Does not require IP forwarding or control plane
– Has no inherent network security
The POWERful Choice - Carrier Ethernet or MPLS
3
What is Carrier Ethernet ? (1)
Blue means Ethernet
Ethernet started out as a LAN technology
LAN networks are small
and operated by consumer
and hence are easily managed
When Ethernet left the LAN environment
new mechanisms were needed, e.g.
–
–
–
–
–
–
Metcalf’s original sketch of Ethernet
scalability (to reach 100s of thousands of end-points)
OAM (Fault Management, Performance Monitoring)
deterministic (Connection-Oriented) connections
support for various topologies (e.g., point-point, rings, trees)
resilience mechanisms (e.g., Automatic Protection Switching)
support for synchronization
Carrier Ethernet (CE) adds carrier-grade features to Ethernet
so that it can replace SONET/SDH as a transport network
The POWERful Choice - Carrier Ethernet or MPLS
4
What is Carrier Ethernet ? (2)
• Mature Technology
– widely deployed by service providers
– promoted and maintained by Metro Ethernet Forum (MEF)
• Deterministic and Connection Oriented (unlike connectionless IP)
– provisioning through management system (not routing)
– support for point-point, multipoint-multipoint, ring, tree, … topologies
• Support for Quality of Service (up to 8 Classes of Service)
– enforcement of bandwidth profiles (dual token bucket shaping/policing)
– color (conformance) marking
• Carrier-grade operations mechanisms:
–
–
–
–
–
service activation testing (Y.1564)
Fault Management (802.1ag, Y.1731)
Performance Monitoring (Y.1731)
Automatic Protection Switching (G.8031, G.8032)
Synchronization <timing distribution> (SyncE, 1588)
• Network security mechanisms:
– access authorization (802.1X)
– source authentication, integrity and optional encryption (MACSec)
The POWERful Choice - Carrier Ethernet or MPLS
5
What is MPLS ? (1)
Red means MPLS
MPLS started out as a technology to accelerate IP forwarding
by setting up tunnels to transport IP
other traffic can be transported via pseudowires
MPLS defined by the IETF, and inherits the rich IP protocol suite
like all IETF protocols, MPLS does not define layer 2 or below
MPLS is a mature technology for core IP networks
full Traffic Engineering is available, but not traffic conditioning (policing/shaping)
supports mesh topologies
uses local Fast ReRoute (not protection switching) for resilience
no network security mechanisms (since core elements are trusted)
A new MPLS version (MPLS-TP)
takes MPLS out of the core network into the transport domain
WARNING: there are two non-interoperable versions (from IETF and ITU-T)
The POWERful Choice - Carrier Ethernet or MPLS
6
What is MPLS ? (2)
We can now distinguish four distinct flavors of MPLS:
1.
best effort MPLS (usually with LDP, perhaps with RSVP-TE for FRR)
not true CO – pinned to route not to Network Elements
used in Internet core
2.
MPLS for L3VPN services (RFC 4364 <ex-2547> using BGP)
used to deliver VPN services to business users
3.
traffic engineered MPLS-TE (currently with RSVP-TE)
true CO with resource reservation
used when strict SLA guarantees must be given (banks, government, …)
4.
transport profile - MPLS-TP (with management or RSVP-TE)
–
–
–
–
–
–
does not assume the existence of IP forwarding plane
does not require the IP control plane (can work with management systems)
implements OAM and APS functionality (based on Carrier Ethernet)
supports ring topologies
still in initial phases of deployment (little interop testing has been performed)
does not add network security features (still susceptible to attack)
The POWERful Choice - Carrier Ethernet or MPLS
7
The battlefront
ETHERNET
first mile
core
network
TRANSPORT NETWORK
local
network
MPLS
last mile
• Ethernet started in the local network (LAN)
and for many years has moved into transport networks
• MPLS started in the core network (WAN)
and is now trying to conquer transport networks with MPLS-TP
The POWERful Choice - Carrier Ethernet or MPLS
8
Technical
Comparison
Features in common
Both Ethernet and MPLS (all flavors) :
• can natively transport IP traffic
– Ethernet can natively transport other traffic types (EtherType)
– MPLS can transport other traffic types via pseudowire technology
• can be transported over SONET/SDH and OTN
• are being actively developed (by multiple standards organizations)
– Ethernet by the IEEE, MEF, ITU, …
– MPLS by the IETF, ITU-T, …
• may exhibit very high or very low transit delays (and everything in-between)
(unlike SONET/SDH which has constant switching latency)
– very high delay when packets need to wait in a queue
– very low delay (much lower than SONET/SDH) for prioritorized traffic
Both CE and MPLS-TP :
• typically use network management systems for configuration
• define FM/PM OAM and diagnostic tests
• support rings and define APS
The POWERful Choice - Carrier Ethernet or MPLS
10
1st reason for differences – format
Ethernet packet headers are self-describing
DA(6B)
•
•
•
•
•
SA(6B)
VT(2B)
VLAN(2B)
T/L(2B)
a globally unique source address
a globally unique destination address
an optional connection identifier (VLAN)
optional Class of Service and Drop Eligibility Indicator
a payload protocol type identifier (EtherType)
MPLS packet headers are only locally meaningful
Label (20b)
•
•
•
•
TC(3b) S(1b)
TTL (8b)
no unique addresses
a locally meaningful label (stack)
a TTL field (to avoid packet looping)
optionally a Traffic Class (TC) field
The POWERful Choice - Carrier Ethernet or MPLS
11
2nd reason for differences – control
• Ethernet was zero-touch in broadcast domain LANs
• CE uses network management to support large networks
• Ethernet does define L2 control protocols (STP, LACP, LLDP, …)
but does not define a routing protocol (neglecting TRILL, E-VPN, etc.)
• Best effort MPLS tunnels according to topology found by IP routing protocols
• So best effort MPLS:
– does not require sophisticated management system
– does requires the full logistics of an IP network
• MPLS-TE requires both IP routing and a sophisticated management system
• MPLS-TP is the only flavor of MPLS that does not require IP routing
but when routing is not used, configuration management is required
(basically equivalent to Carrier Ethernet)
The POWERful Choice - Carrier Ethernet or MPLS
12
Additional differences
• Ethernet defines physical (L1) layers (but may run over MPLS as a PW)
MPLS requires a server layer to transport it (which is usually Ethernet)
• Ethernet can not tolerate forwarding loops
Carrier Ethernet supports rings with G.8032
and Industrial Ethernet supports them with High-availability Seamless Redundancy
MPLS can (since it contains a TTL field)
• Carrier Ethernet supports bandwidth profiles (bucketing)
• Ethernet supports IEEE 1588 timing distribution over packet
and defines a physical layer to support Synchronous Ethernet
MPLS may obtain support for 1588 (work ongoing in IETF) but since MPLS does
not a physical layer it can not provide physical layer synchronization support
• Ethernet has network security mechanisms (MACsec, 802.1X, SNMPv3)
MPLS does not define any standardized network security mechanisms
and since MPLS has no source address
it can not provide source authentication
The POWERful Choice - Carrier Ethernet or MPLS
13
The new trend – SDN
Distributed routing protocols are limited to
•
•
finding simple connectivity
minimizing number of hops
but can not perform more sophisticated operations
• optimizing paths under constraints (e.g., delay, security)
•
•
setting up backup paths
integrating networking functionalities (e.g., NAT, firewall) into paths
Lately, a new paradigm has arisen – Software Derived Networking, which:
•
•
•
removes control protocols from network elements
replaces distributed routing with centralized path computation
configures the forwarding actions of the switches from a central site
SDN sees the IP/MPLS control plane as a disadvantage
and adopts the Carrier Ethernet / MPLS-TP approach
New SDN tools can optimally manage operational networks
•
•
SDN services can be added and modified at the speed of software
SDN should lead to significant OPEX reductions
The POWERful Choice - Carrier Ethernet or MPLS
14
Why not use both ? (1)
We have seen that MPLS is missing several critical features
in particular, synchronization and network security
So, why not use both Ethernet and MPLS taking the best features of each ?
In fact, MPLS does not define its own physical layer
and the most common physical layer supporting MPLS is Ethernet
although MPLS can be transported over other physical layers, e.g., SDH or OTN
So the real question is whether to maintain an Ethernet network
or an MPLS network in addition to an Ethernet network !
MPLS
ETHERNET
The POWERful Choice - Carrier Ethernet or MPLS
15
Why not use both ? (2)
How many networks are there ?
Ethernet defines its own physical layer
although Ethernet can be transported over other physical layers
When transporting IP over Ethernet there are actually 2 or 3 networks
3 IP
2 Ethernet
1 Ethernet or optionally SONET/SDH or OTN
MPLS does not define its own physical layer
When transporting IP over MPLS there are actually 3 or 4 networks
3 IP
2.5MPLS
2 Ethernet
1 Ethernet or optionally SONET/SDH or OTN
Do we care how many networks there are ?
The POWERful Choice - Carrier Ethernet or MPLS
16
Why not use both ? (3)
Yes, because maintaining networks is never trivial or expense-free!
• Attempts to design a network to use Ethernet as a dumb pipe under MPLS
usually end up using a large number of Ethernet mechanisms
• For example, when running MPLS over Ethernet, one usually needs :
– staff trained in Ethernet technologies and staff trained in IP/MPLS technologies
– to be able to run Ethernet OAM and MPLS diagnostic tools
– to maintain an Ethernet NMS and MPLS management screens
• Network management is the core business of a network service provider
and for them it may be reasonable to maintain
duplicate staff, tools, operations centers, etc.
Network maintenance is not the core business of a power utility and the
duplication and added complexity is usually not justifiable
The POWERful Choice - Carrier Ethernet or MPLS
17
Operational
Comparison
Utilities network requirements
• Traffic types (not an exhaustive list)
–
–
–
–
–
SCADA operational traffic
teleprotection traffic
synchrophasor traffic
surveillance video
general TCP/IP
and there is a growing demand for bandwidth
• Determinism (CO behavior)
– best effort / nondeterministic (Internet-like) behavior is not acceptable
• Resilience (critical infrastructures must be highly reliable)
• Low (and constant) end-end delay (for SCADA and teleprotection applications)
• Management
– networks presently employ centralized management
– end-to-end provisioning and maintenance are musts
• Synchronization
• Network security (merits discussion in a separate section)
– cyber security is a growing concern
– regulatory requirements are appearing
The POWERful Choice - Carrier Ethernet or MPLS
19
Traffic types
• SONET/SDH was designed to transport certain traffic types and rates
– mapping new traffic types is difficult and complex
– transport of most traffic rates is inefficient
– no higher rates are being defined for SONET/SDH
• Ethernet was designed to transport arbitrary traffic types and rates
– EtherType mechanism to indicate payload types
– pseudowire technology may also be used
– no rate constraints
– higher rates being defined (presently 100Gbps)
• MPLS was designed to transport IP traffic
– pseudowire technology enables transport of arbitrary traffic types
– MPLS imposes no rate constraints or limitations
So, regarding traffic, SONET/SDH is reaching End-of-Life
while Ethernet and MPLS are future proof!
The POWERful Choice - Carrier Ethernet or MPLS
20
Determinism
Networks are deterministic
when traffic consistently flows through the network in the same way
With nondeterministic networks (e.g., IP and best effort MPLS)
each packet may take a different route through the network, thus
•
•
•
enabling intermittent faults (only when the packets happen to go there)
complicating troubleshooting (where did the packets go?)
excluding the reservation of resources or specific processing
at particular network elements (you can’t be sure the packets will go where you want …)
SONET/SDH networks are Circuit Switched, and thus completely deterministic
CE and some types of MPLS (TE, TP) are Connection Oriented
and thus relatively deterministic
traffic consistently takes the same path through the network
but does not always take precisely the same time to traverse
So, due to lack of determinism,
best effort MPLS is not a reasonable candidate
for a power utility operational network
The POWERful Choice - Carrier Ethernet or MPLS
21
Resilience
• SONET/SDH is well-known for its Automatic Protection Switching
– gold standard 1:1 APS supports < 50 millisecond protection switching time
– 1+1 APS can provide hitless switching (at the cost of increased bandwidth)
• Best effort MPLS relies on slow rerouting for recovery
• MPLS with Fast ReRoute performs local detours around failures
– at the expense of loss of determinism
• CE and MPLS-TP support several types of APS
– CE’s G.8031 and G.8032 and MPLS-TP’s RFC 6378, 6974, ITU-T G.8131
– 1+1 pseudowire redundancy achieves hitless switching
at the cost of increased bandwidth consumption
So, from the point of view of resilience
CE and MPLS-TP are as good as SONET/SDH !
The POWERful Choice - Carrier Ethernet or MPLS
22
End-end delay and delay consistency
Some operational traffic require low and consistent delay
For example, teleprotection’s end-end delay budget may be 6 milliseconds
• SONET/SDH latency is typically sufficiently low (e.g., under 2 msec.)
– is constant
– is independent of SONET/SDH rate (whether OC3 or OC192)
• Carrier Ethernet and MPLS may have much lower transit latencies prioritorized
packets only wait for the packet already exiting the switch for the worst case (1500B
packet that just started) this latency is:
– 123 msec at 100 Mbps (about the same as a SONET/SDH frame)
– 12.3 msec at 1 Gbps
– 1.23 msec at 10 Gbps
• TDM pseudowire traffic requires a jitter buffer
– eliminates delay variation
– adds additional latency (under 1 msec for prioritorized, low PDV, traffic)
So, delay considerations actually favor CE and MPLS over SONET/SDH !
The POWERful Choice - Carrier Ethernet or MPLS
23
What about delay asymmetry?
For some bi-directional applications
the delay must be symmetric
(the same in both directions)
• SONET/SDH
– ADM rings have constant delay asymmetry
(without “spatial reuse” management)
– teleprotection mechanisms compensate for this
SONET/SDH–
Delay asymmetry
• CE and MPLS
– CE is always co-routed and thus symmetric
– best effort MPLS may not be co-routed
– but MPLS-TE and MPLS-TP can be
• TDM pseudowire
– may introduce buffer asymmetry
– correct implementation keeps this very low
CE or MPLS
Symmetric delay
So, delay asymmetry considerations
actually favor CE and MPLS-TP over SONET/SDH !
The POWERful Choice - Carrier Ethernet or MPLS
24
Management
• SONET/SDH networks typically are typically supported
by sophisticated management platforms
(Operation Support Systems, Network Management Systems)
developed by vendors or users over decades
• Carrier Ethernet was developed
to replace SONET/SDH in service provider networks
and thus borrowed heavily from existing SONET/SDH management
architecture, terminology, and look-and-feel
• MPLS-TP was developed
to be functionally equivalent to previously developed CE
and thus borrowed heavily from existing SONET/SDH management
architecture, terminology, and look-and-feel
So, from the point of view of management
SONET/SDH, CE and MPLS-TP are exceptionally similar
while best-effort MPLS is completely different
The POWERful Choice - Carrier Ethernet or MPLS
25
Synchronization
• Synchronization (AKA timing)
the ability to transfer highly accurate frequency or time
over a network (obviating reliance on GPS)
While timing may not be a requirement in present-day utilities networks
it is crucial to support some imminent applications
such as new teleprotection mechanisms and synchrophasors
• SONET/SDH has native support for frequency transfer
as it requires highly accurate frequency for its own operation
but does not support time transfer
• Ethernet fully supports both time and frequency transfer
by use of Synchronous Ethernet (ITU-T G.8261/2/4) for physical layer support
and support for IEEE 1588 Precision Time Protocol for packet layer distribution
• MPLS does not currently support timing at all
work in IETF-TICTOC is progressing to provide some support for IEEE 1588
having no physical layer, MPLS will never support physical layer frequency distribution
So, regarding synchronization CE is the best alternative
followed by SONET/SDH (and MPLS has no support)
The POWERful Choice - Carrier Ethernet or MPLS
26
Summary (so far)
So far we have compared CE, MPLS, and MPLS-TP to SONET/SDH, and found
Traffic types and growing demand for bandwidth
• Determinism
– SONET/SDH, CE and MPLS-TP are all acceptable
– best effort MPLS is unacceptable for critical operational networks
• Resilience
– CE and MPLS-TP (but not non-TP MPLS) are as good as SONET/SDH
• Delay (including consistency and asymmetry)
– favors CE and MPLS (for asymmetry only MPLS-TP) over SONET/SDH
• Management
– CE and MPLS-TP (but not non-TP MPLS) are equivalent to SONET/SDH
• Synchronization
– CE has full support, SONET/SDH supports frequency, MPLS is deficient
In the final section we will discuss Network Security
and discover further differences between Carrier Ethernet and MPLS
The POWERful Choice - Carrier Ethernet or MPLS
27
Network Security
for
Power Utilities
Security highlights
• MPLS was invented for core networks
where network elements are in secure locations, and therefore trusted
and was thus designed without any security mechanisms
• In particular, the MPLS forwarding plane
– can not be source authenticated (no source address!)
– has no standardized integrity mechanism
and the MPLS control plane uses soft-state protocols
• Ethernet was designed for untrusted network elements
• CE does not suffer from most of these ailments since Ethernet ports can be:
– Authorized (by 802.1X)
and Ethernet packets can be
– Source authenticated (by MACsec)
– Integrity (and replay) tested (by MACsec)
and CE uses a security-enabled management plane (instead of a control plane)
Let’s see why this is important !
The POWERful Choice - Carrier Ethernet or MPLS
29
MPLS data plane DoS (injection) attack
CE can block this attack
using 802.1X authorization
Central Site
DMS/EMS
PE
Substation
RTU
Data Center
MPLS
Core
PE
TPR
PE
LSP
LAN
LSP
PE
Connect to any free
MPLS port
•
•
•
Once a packet is inside an MPLS network it can not be blocked (no authentication)
If an attacker gains physical access to an MPLS network node (e.g., by using a free port)
he/she can inject fake MPLS packets (guessing until a valid label is found)
At high rates this injection can overwhelm forwarding resources
The POWERful Choice - Carrier Ethernet or MPLS
30
MPLS man in the middle attack
Central Site
CE can block this attack
using MACSec’s integrity check
DMS/EMS
Substation
RTU
Data Center
MPLS
Core
TPR
PE
LAN
LSP
•
•
•
•
Tampering means falsifying SCADA RTU/IED <-> control station data
Can be implemented by owning the switch or by inserting an evil SFP into a port
MPLS has no integrity mechanisms to detect tampering
Result can be power disruption and/or physical damage to equipment
The POWERful Choice - Carrier Ethernet or MPLS
31
MPLS LSP swap attack
Central Site
DMS/EMS
CE can block this attack
using MACSec’s source authentication
Substation A
RTU
Data Center
TPR
PE
LAN
MPLS
Core
Substation B
RTU
TPR
PE
LAN
•
•
•
•
The attacker exchanges the internal labels belonging to 2 substations
Implemented by owning the switch or via an Evil SFP
MPLS has no source authentication mechanisms
The Central Site control systems now believe that indications from substation A
belong to substation B (and vice versa)
The POWERful Choice - Carrier Ethernet or MPLS
32
MPLS control plane attack
Not relevant for MPLS-TP w/o control plane
Central Site
DMS/EMS
Attack is not applicable to CE
which doesn’t use a Control Plane
Substation
RTU
Data Center
MPLS
Core
•
•
•
TPR
PE
LAN
MPLS control protocols (e.g., LDP and RSVP-TE) are soft-state
(when contact with a peer is lost, LSPs are withdrawn)
Intermittently deleting consecutive few heartbeat packets
causes massive denial of service
A more complex attack can poison the Label Information Base
The POWERful Choice - Carrier Ethernet or MPLS
33
Summary (final this time)
In our previous summary we saw that
• Carrier Ethernet and MPLS-TP (but not MPLS)
were as good as, or even better than, SONET/SDH on most accounts
and had the further advantage of being future proof
• Best effort MPLS is nondeterministic
and should not be considered for operational networks
• Concerning synchronization (crucial for up-and-coming applications)
Carrier Ethernet has full support
while MPLS has none (thus diminishing its status as being future proof)
Now we have seen that
• Regarding Network Security
MPLS is highly vulnerable
while Carrier Ethernet possesses mechanisms to fight off attacks
These facts should be taken into account
when planning future transport networks
The POWERful Choice - Carrier Ethernet or MPLS
34
Yaakov (J) Stein
CTO
yaakov_s@rad.com