WAN Architectures and Design
Principles
Stephen Lynn
Consulting Systems Architect
BRKRST-2041
Agenda
§
WAN Technologies & Solutions
•
•
•
•
§
WAN Architecture Design Considerations
•
•
•
§
WAN Transport Technologies
WAN Overlay Technologies
WAN Optimisation
Wide Area Network Quality of Service
WAN Design and Best Practices
Secure WAN Communication
WAN Extension into the Cloud
Summary
WAN Transport Technologies
The Architectural Continuum
Early-Mid 1990s
Early Networking
Mid 1990s-Mid 2000s
Global Scale
Flat/Bridged
Multiprotocol
Large Scale
Experimental Networks
Business Enabling
Mission Critical
Architectural
Lessons
Architectural
Lessons
Route First,
Bridge only if Must
Architectural
Lessons
Redundancy
Protocols required for
Scale & Restoration
1960
ARPAnet
1960
Internet
Protocol
1970
TCP/IP
X.25
1980
RIP (BSD)
OSPF,
ISDN,
ATM
1990
IP Ubiquity
Business Survival
Build to Scale
Frame-Relay
BGP
Today
IPv6
GRE
Planning
?
DMVPN
2000
Future
2010
Metro-Ethernet
Tag
SDN
Switching
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
5
The Challenge
•
Build a network that can adapt to a quickly changing business and technical
environment
•
Realise rapid strategic advantage from new technologies
•
•
•
•
•
•
IPv6: global reachability
Cloud: flexible diversified resources
Internet of Things
Fast-IT
What’s next?
Adapt to business changes rapidly and smoothly
Mergers & divestures
• Changes in the regulatory & security requirements
• Changes in public perception of services
•
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
6
Network Design Modularity
East Theater
Tier 1
West Theater
Tier 2
Global
IP/MPLS Core
In-Theater
IP/MPLS Core
West Region
East Region
Tier 3
Internet
Cloud
Public Voice/Video Mobility
Private
IP
Service
Metro
Service
Public
IP
Service
BRKRST-2041
Metro
Service
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
7
Hierarchical Network Principle
•
Use hierarchy to manage network scalability and complexity
while reducing routing algorithm overhead
•
Hierarchical design used to be…
Three routed layers
• Core, distribution, access
• Only one hierarchical structure end-to-end
•
•
Hierarchical design has become any design that…
Splits the network up into “places,” or “regions”
• Separates these “regions” by hiding information
• Organises these “regions” around a network core
• “hub and spoke” at a macro level
•
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
8
MPLS L3VPN Topology
Definition
Spoke
Site 1
Spoke
Site Y
Spoke
Site 2
SP-Managed
MPLS IP WAN
Spoke
Site X
Hub Site
(The Network)
Equivalent to
Spoke
Site N
Spoke
Site 1
Spoke
Site 2
Spoke
Site X
§
MPLS WAN is provided by a service provider
§
As seen by the enterprise network, every site is one IP “hop” away
§
Equivalent to a full mesh, or to a “hubless” hub-and-spoke
BRKRST-2041
Spoke
Site Y
Spoke
Site N
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
9
Virtual Routing and Forwarding Instance (VRF)
Provides Network Virtualisation and Path Isolation
VRF
VRF
VRF
§ Virtualisation at Layer 3 forwarding
§ Associates to Layer 3 interfaces on router/switch
§ Each VRF has its own
Forwarding table (CEF)
Routing process (RIP, OSPF, BGP)
§ VRF-Lite
Hop-by-hop
§ MPLS VPN
Multi-hop
VRF
VRF
VRF
! PE Router – Multiple VRFs
ip vrf blue
rd 65100:10
route-target import 65100:10
route-target export 65100:10
ip vrf yellow
rd 65100:20
route-target import 65100:20
route-target export 65100:20
!
interface GigabitEthernet0/1.10
ip vrf forwarding blue
interface GigabitEthernet0/1.20
ip vrf forwarding yellow
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
10
Wide Area Network Design Trends
Dual Providers Design
Single Provider Design
§
§
§
§
Enterprise will home all sites
into a single carrier to
provide L3 MPLS VPN
connectivity.
Pro: Simpler design with
consistent features
Con: Bound to single carrier
for feature velocity
Overlay Network Design
§
Enterprise will single or dual home
sites into one or both carriers to
provide L3 MPLS VPN connectivity.
§
Overlay tunnelling technologies
with encryption for provider
transport agnostic design
§
Pro: Protects against MPLS service
failure with Single Provider
§
§
Pro: Potential business leverage for
better competitive pricing
Pro: Can use commodity
broadband services for lower
cost higher bandwidth service
§
Pro: Flexible overlay network
topology that couples from the
physical connectivity
§
Con: Increased design
complexity
§
Con: Does not protect
against MPLS cloud failure
with Single Provider
§
Con: Increased design complexity
due to service implementation
differences (e.g. QoS, BGP AS
Topology)
Con: Feature differences between
§ Con: Additional technology
providers could force customer to use needed for SLA over commodity
least common denominator features.
transport services
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
11
Single Carrier Site Types (Non-Transit)
§ Dual Homed Non Transit
AS 64517
CE3
Only advertise local prefixes (^$)
Typically with Dual CE routers
CE4
CE5
BGP design:
eBGP to carrier
AS 200
iBGP between CEs
Redistribute cloud learned routes
into site IGP
CE2
CE1
Site IGP
C2
C1
§ Single Homed Non Transit
Advertise local prefixes and
optionally use default route.
AS 64512
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
12
Dual Carrier: Transit vs. Non Transit
§ To guarantee single homed site
reachability to a dual homed site
experiencing a failure, transit sites
had to be elected.
§ Transit sites would act as a BGP
bridge transiting routes between the
two provider clouds.
§ To minimise latency costs of transits,
transits need to be selected with
geographic diversity (e.g. from the
East, West and Central US.)
CE3
CE4
CE5
Transit
AS 64545
AS 100
CE1
Site
IGP
AS 200
CE2
C2
C1
Prefix X
Prefix Y
Prefix Z
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
13
Single vs. Dual Carriers
Single Provider
Pro: Common QoS support
model
Dual Providers
Pro: More fault domains
Pro: Only one carrier to “tune”
Pro: More product offerings to
business
Pro: Reduced head end circuits
Pro: Ability to leverage vendors
for better pricing
Pro: Overall simpler design
Pro: Nice to have a second
vendor option
Con: Carrier failure could be
catastrophic
Con: Increased Bandwidth
“Paying for bandwidth twice”
Con: Do not have another
carrier “in your pocket”
Con: Increased overall design
complexity
Con: May be reduced to
“common denominator” between
carriers
Resiliency Drivers vs. Simplicity
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
14
Metro Ethernet Service (L2VPN)
§
§
§
§
E-Line (Point-to-Point)
Replaces TDM private line and
Frame-Relay or ATM L2VPN
Point-to-point EVCs offer predictable
performance for applications
One or more EVCs allowed per
single physical interface (UNI)
Supports “hub & spoke” topology
§
§
§
§
E-LAN (Point-to-Multipoint)
Offers point to multipoint connectivity
Transparent to VLANs and Layer 2
control protocols
4 or 6 classes of QoS support
Supports service multiplexing (Ex.
Internet access and corporate VPN
via one UNI)
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
15
MPLS (L3VPN) vs. Metro Ethernet (L2VPN)
MPLS Layer 3 Service
§
Routing protocol dependent on the
carrier
§
Layer 3 capability depends on
carrier offering
QoS (4 classes/6 classes)
• IPv6 capability
•
§
Transport IP protocol only
§
Highly scalable and ideal for large
network
MetroE Layer 2 Service
§
Flexibility of routing protocol and
network topology independent of
the carrier
§
Customer manages layer 3 QoS
§
Capable of transport IP and
non-IP traffic.
§
Routing protocol determines
scalability in point-to-multipoint
topology
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
16
WAN Overlay Technologies
Types of Overlay Service
Layer 2 Overlays
Layer 3 Overlays
• Layer 2 Tunnelling Protocol—Version 3
(L2TPv3)
• IPSec—Encapsulating Security Payload
(ESP)
– Layer 2 payloads (Ethernet, Serial,…)
– Pseudowire capable
• Other L2 overlay technologies –
OTV, VxLAN
– Strong encryption
– IP Unicast only
• Generic Routing Encapsulation (GRE)
– IP Unicast, Multicast, Broadcast
– Multiprotocol support
• Other L3 overlay technologies –
MPLSomGRE, LISP, OTP
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
18
Tunnelling
GRE and IPSec Transport and Tunnel Modes
IP HDR
IP Payload
GRE packet with new IP header: Protocol 47 (forwarded using new IP dst)
IP HDR
GRE
20 bytes
4 bytes
IP HDR
IP Payload
2 bytes
IPSec Transport mode
IP HDR
20 bytes
ESP HDR
30 bytes
IP Payload
Encrypted
Authenticated
20 bytes
ESP HDR
54 bytes
ESP
Trailer Auth
2 bytes
IPSec Tunnel mode
IP HDR
ESP
IP HDR
IP Payload
Encrypted
Authenticated
BRKRST-2041
ESP
ESP
Trailer Auth
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
19
Locator/Identifier Separation Protocol (LISP)
Dynamic Tunnelling Analogous to a DNS but for Network Infrastructure
§ DNS resolves IP addresses for URLs
[ who is www.cisco.com] ?
host
DNS
Server
DNS
URL Resolution
[153.16.5.29, 2610:D0:110C:1::3 ]
§ LISP resolves locators for queried identities
[ where is 2610:D0:110C:1::3] ?
LISP
router
[ location is 128.107.81.169 ]
LISP
Mapping
System
LISP
Identity-to-location
Map Resolution
This Topic Is Covered in Detail in BRKRST-3045
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
20
Locator / ID Separation Protocol (LISP)
LISP “Mapping System” is analogous to a DNS lookup
‒ DNS resolves IP addresses for queried URL
Answers the “WHO IS” question
[ Who is www.cisco.com ] ?
DNS
Server
Host
DNS
Name -to- IP
URL Resolution
[ Address is 153.16.5.29, 2610:D0:110C:1::3 ]
‒ LISP resolves locators for queried identities
Answers the “WHERE IS” question
[ Where is 2610:D0:110C:1::3 ] ?
LISP
Router
LISP Map
System
[ Locator is 128.107.81.169, 128.107.81.170 ]
LISP
ID -to- Locator
Map Resolution
This Topic Is Covered in Detail in BRKRST-3045
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
21
LISP §Overview
Terminologies
EID (Endpoint Identifier) is the IP address of a host – just as it is today
§ RLOC (Routing Locator) is the IP address of the LISP router for the host
§ EID-to-RLOC mapping is the distributed architecture that maps EIDs to RLOCs
ITR – Ingress Tunnel Router
• Receives packets from site-facing interfaces
• Encap to remote LISP sites, or native-fwd to
non-LISP sites
Provider A
10.0.0.0/8
xTR-1
ETR – Egress Tunnel Router
• Receives packets from core-facing interfaces
• De-cap, deliver packets to local EIDs at site
Provider X
12.0.0.0/8
ETR
ITR
ITR
packet flow
S
xTR-1
ETR
packet flow
ETR
ETR
ITR
xTR-2
ITR
Provider B
11.0.0.0/8
Provider Y
13.0.0.0/8
LISP Site 1
D
xTR-2
LISP Site 2
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
22
LISP Operation Example
LISP Data Plane - Unicast Packet Forwarding
Map-Cache Entry
3
EID-prefix: 2001:db8:2::/48
Locator-set:
12.0.0.2, priority: 1, weight: 50 (D1)
13.0.0.2, priority: 1, weight: 50 (D2)
PI EID-prefix
2001:db8:1::/48
11.0.0.2 -> 12.0.0.2
LISP Site 1
LISP Site 2
7
xTR-1
5
10.0.0.2
PI EID-prefix
2001:db8:2::/48
2001:db8:1::1 -> 2001:db8:2::1
Provider X
12.0.0.0/8
Provider A
10.0.0.0/8
2001:db8:1::1 -> 2001:db8:2::1
ETR
ITR
6
2001:db8:1::1 -> 2001:db8:2::1
xTR-1
2
This policy controlled
by the destination site
ETR
4
12.0.0.2
11.0.0.2 -> 12.0.0.2
ITR
2001:db8:1::1 -> 2001:db8:2::1
S
ETR
ITR
xTR-2
1
13.0.0.2
11.0.0.2
DNS entry:
D.abc.com AAAA
Provider B
11.0.0.0/8
Provider Y
13.0.0.0/8
ETR
ITR
D
xTR-2
2001:db8:2::1
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
23
LISP Use Cases
Efficient Multi-Homing
IPv6 Transition
v6
PxTR
v4 v6
IPv4 Core
v6 service
xTR
v6
§
§
Internet
IPv6
Internet
IPv4
Internet
LISP
Site
IPv6-over-IPv4, IPv6-over-IPv6
IPv4-over-IPv6, IPv4-over-IPv4
§
§
Legacy Site
LISP Site
IP Portability
Ingress Traffic Engineering Without BGP
Data Centre/ VM Mobility
Virtualisation/Multi-tenancy
Legacy Site
LISP
routers
Legacy Site
Data
Centre 1
Data
Centre 2
Internet
PxTR
IP Network
LISP
routers
Mapping
DB
LISP
routers
VM move
West
DC
§
East
DC
Large Scale Segmentation
§
VM
VM
a.b.c.1
a.b.c.1
Cloud / Layer 3 VM Move
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
24
Cisco Site to Site VPN Technologies Comparison
Features
Infrastructure Network
Network Style
Failover Redundancy
Scalability
IP Multicast
QoS
Policy Control
Technology
DMVPN
§
§
§
§
Public or Private Transport
Overlay Routing
IPv4/IPv6 dual Stack
Large Scale Hub and Spoke
with dynamic Any-to-Any
FlexVPN
GET VPN
§ Public or Private Transport
§ Overlay Routing
§ Converged Site to Site and
Remote Access
§ Private IP Transport
§ Flat/Non-Overlay IP
Routing
§ Any-to-Any;
(Site-to-Site)
§ Active/Active based on
Dynamic Routing
§ Dynamic Routing or IKEv2 Route
Distribution
§ Server Clustering
§ Transport Routing
§ COOP Based on GDOI
§ Unlimited
§ 3000+ Client/Srv
§ Unlimited
§ 3000+ Client/Srv
§ 8000 GM total
§ 4000 GM/KS
§ Multicast replication at hub
§ Multicast replication at hub
§ Multicast replication in
IP WAN network
§ Per Tunnel QoS, Hub to Spoke
§ Per SA QoS, Hub to Spoke
§ Per SA QoS, Spoke to Spoke
§ Transport QoS
§ Locally Managed
§ Centralised Policy Management
§ Locally Managed
§ Tunneled VPN
§ Multi-Point GRE Tunnel
§ IKEv1 & IKEv2
§ Tunneled VPN
§ Point to Point Tunnels
§ IKEv2 Only
§ Tunnel-less VPN
§ Group Protection
§ IKEv1
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
25
Dynamic Multipoint VPN (DMVPN)
•
Branch spoke sites establish an IPsec tunnel to and
register with the hub site
•
IP routing exchanges prefix information for each site
•
BGP or EIGRP are typically used for scalability
•
With WAN interface IP address as the tunnel
source address, provider network does not
need to route customer internal IP prefixes
•
Data traffic flows over the DMVPN tunnels
•
When traffic flows between spoke sites, the hub
assists the spokes to establish a site-to-site tunnel
•
SECURE ON-DEMAND TUNNELS
ASR 1000
Hub
Branch n
IPsec
VPN
ISR
ISR
Branch 1
ISR
Branch 2
Traditional Static Tunnels
Per-tunnel QOS is applied to prevent hub site
oversubscription to spoke sites
DMVPN On-Demand Tunnels
Static Known IP Addresses
Dynamic Unknown IP Addresses
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
26
DMVPN – How it Works
•
•
•
•
•
•
•
•
•
Spokes build a dynamic permanent GRE/IPsec tunnel to the
hub, but not to other spokes. They register as clients of the
NHRP server (hub) and register their NBMA address
Active-Active redundancy model—two or more hubs per spoke
All configured hubs are active and are routing neighbours
with spokes
Routing protocol routes are used to determine traffic forwarding
A spoke will initially send a packet to a destination (private)
subnet behind another spoke via the hub, and the hub will send
it an NHRP redirect.
The redirect triggers the spoke to send an NHRP query for the
data packet destination address behind the destination spoke
The destination spoke initiates a dynamic GRE/IPsec tunnel to
the source spoke (it now knows its NBMA address) and sends
the NHRP reply.
The dynamic spoke-to-spoke tunnel is built over the mGRE
interface
When traffic ceases then the spoke-to-spoke tunnel is removed
Dual DMVPN Design
Single mGRE tunnel on Hub,
two mGRE tunnels on Spokes
192.168.0.0/24
Physical: 172.17.0.5
Tunnel0:
10.0.1.1
Physical: 172.17.0.1
Tunnel0:
10.0.0.1
Physical: (dynamic)
Tunnel0: 10.0.0.12
Tunnel1: 10.0.1.12
.1
192.168.3.0/24
Physical: (dynamic)
Tunnel0: 10.0.0.11
Tunnel1: 10.0.1.11
.1
.1
192.168.1.0 /24
BRKRST-2041
192.168.2.0 /24
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
27
DMVPN Network Designs
Spoke-to-hub tunnels
Spoke-to-spoke tunnels
2547oDMVPN tunnels
Server Load Balancing
VRF-lite
Spoke-to-spoke
Hierarchical
Increase in Scale
Hub and spoke
2547oDMVPN
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
28
Any-to-Any Encryption
Before and After GETVPN
Public/Private WAN
Private WAN
Before: IPSec P2P Tunnels
After: Tunnel-Less VPN
WAN
Multicast
§ Scalability—an issue (N^2 problem)
§ Overlay routing
§ Any-to-any instant connectivity can’t
§
§
be done to scale
Limited QoS
Inefficient Multicast replication
§ Scalable architecture for any-to-any
§
§
§
§
connectivity and encryption
No overlays—native routing
Any-to-any instant connectivity
Enhanced QoS
Efficient Multicast replication
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
29
Group Security Functions
Key Server
§ Validate Group Members
§ Manage Security Policy
§ Create Group Keys
§ Distribute Policy/Keys
Key Server
Routing Member
§ Forwarding
§ Replication
§ Routing
Group
Member
Routing
Members
Group
Member
Group Member
§ Encryption Devices
§ Route Between Secure/
Unsecure Regions
§ Multicast Participation
Group
Member
Group
Member
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
30
Group Security Elements
Key Servers
Group Policy
KS Cooperative
Protocol
Key Encryption Key
(KEK)
Traffic Encryption
Key (TEK)
Group
Member
Routing
Members
Group
Member
Group
Member
RFC3547àRFC6407:
Group Domain of
Interpretation (GDOI)
Group
Member
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
31
GETVPN - Group Key Technology
Operation Example
GM3
GM4
GM2
§
Step 1: Group Members (GM)
“register” via GDOI (IKE) with the
Key Server (KS)
•
•
GM5
GM1
GM6
GM9
KS authenticates and authorises the GM
KS returns a set of IPsec SAs
for the GM to use
KS
GM8
GM3
GM7
GM4
GM2
§
Step 2: Data Plane Encryption
•
•
§
GM exchange encrypted traffic using the
group keys
The traffic uses IPSec Tunnel Mode with
“address preservation”
GM5
GM1
GM6
GM9
GM3
Step 3: Periodic Rekey of Keys
•
KS pushes out replacement IPsec
keys before current IPsec keys expire;
This is called a “rekey”
KS
GM8
GM7
GM4
GM2
GM5
GM1
GM6
GM9
BRKRST-2041 GM8
KS
GM7
© 2017 Cisco
and/or its affiliates. All rights reserved. Cisco Public
32
GETVPN Virtualisation Deployment Model
GETVPN Segmented WAN
CE
PE
PE
CE
MPLS VPN
LISP with GETVPN
CE
PE
PE
CE
GET Encrypted LISP
LISP over GETVPN
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
33
WAN Optimisation
The WAN Is the Barrier to Branch
Application Performance
§
Applications are
designed to work
well on LAN’s
•
•
•
§
High bandwidth
Low latency
Reliability
WANs have
opposite
characteristics
•
•
•
Round Trip Time ~ 0ms
Low bandwidth
High latency
Packet loss
Client
LAN Switch
Server
Round Trip Time ~ Many milliseconds
Client
LAN
Switch
WAN
LAN Switch
Server
WAN Packet Loss and Latency =
Slow Application Performance =
Keep and manage servers in branch offices ($$$)
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
35
TCP Behaviour
Return to maximum
throughput could take a
very long time!
Window
Size
Packet loss
Slow start
Packet loss
Packet loss
Congestion avoidance
Packet loss
TCP
Time (RTT)
RFC1323 - TCP Extensions for High Performance
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
36
WAAS—TCP Performance Improvement
§ Transport Flow Optimisation (TFO) overcomes TCP and WAN bottlenecks
§ Shields nodes connections from WAN conditions
– Clients experience fast acknowledgement
– Minimise perceived packet loss
– Eliminate need to use inefficient congestion handling
WAN
LAN TCP
Behaviour
Window Scaling
Large Initial Windows
Congestion Mgmt
Improved Retransmit
BRKRST-2041
LAN TCP
Behaviour
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
37
WAAS Advanced Compression
DRE and LZ Manage Bandwidth Utilisation
Data Redundancy Elimination
(DRE)
Persistent LZ Compression
LZ
•Application-agnostic compression
•Up to 100:1 compression
•WAAS 4.4: Context Aware DRE
Benefits
•Session-based
compression
• Application-agnostic
compression
•Up
compression
• Uptoto10:1
100:1
compression
•Works
cold
DREDRE
cache
• WAASeven
4.4: during
Context
Aware
WAN
LZ
DRE
DRE
Synchronised
Compression
History
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
38
Comparing TCP and Transport
Flow Optimisation
Cisco TFO Provides Significant Throughput
Improvements over Standard TCP Implementations
Window
Size
TFO
TCP
Slow start
Congestion avoidance
Time (RTT)
TFO is using RFC2018, RFC1323, RFC3390 and BIC-TCP
http://netsrv.csc.ncsu.edu/export/bitcp.pdf
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
39
Cisco WAAS: WAN Optimisation Deployment
Virtual Private Cloud
WAAS
Express
Branch Office
vWAAS
WAE
Server
VMs
Nexus 1000v
vPATH
VMware ESXi Server
Branch Office
WAAS
Service
Module/ UCSe
Nexus 1000v VSM
WAN
UCS /x86 Server
FC SAN
Branch Office
WAAS
Appliance
Regional Office
WAAS
Appliance
Regional Office
WAAS-XE
on 4451
Internet
Data Centre or
Private Cloud
WAAS
Appliances
AppNav +
WAAS
VMware ESXi
BRKRST-2041
vWAAS
Appliances
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Server VMs
40
WAN Quality of Service
Quality of Service Operations
How Does It Work and Essential Elements
Classification and
Marking
§
The first element to a QoS policy is to classify/identify the traffic that is to be treated differently.
Following classification, marking tools can set an attribute of a frame or packet to a specific value.
Policing:
•
§
Post-Queuing
Operations
Classification and Marking:
•
§
Queuing and
Dropping
Determine whether packets are conforming to administratively-defined traffic rates and take action
accordingly. Such action could include marking, remarking or dropping a packet.
Scheduling (including Queuing and Dropping):
•
Scheduling tools determine how a frame/packet exits a device. Queuing algorithms are activated only
when a device is experiencing congestion and are
deactivated
when the congestion clears.
42
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Enabling QoS in the WAN
Traffic Profiles and SLA Requirements
Voice
§
§
§
§
§
Smooth
Benign
Drop sensitive
Delay sensitive
UDP priority
Bandwidth per Call
Depends on Codec,
Sampling-Rate,
and Layer 2 Media
Latency ≤ 150 ms
§ Jitter ≤ 30 ms
§ Loss ≤ 1%
§ Bandwidth (30-128Kbps)
One-Way Requirements
§
Telepresence
SD Video Conf
§
§
§
§
§
Bursty
Greedy
Drop sensitive
Delay sensitive
UDP priority
§
§
§
§
§
Data
Bursty
Drop sensitive
Delay sensitive
Jitter sensitive
UDP priority
§
§
§
§
§
SD/VC has the Same
Requirements as
VoIP, but Has
Radically Different
Traffic Patterns
(BW Varies Greatly)
HD/VC has Tighter
Requirements than
VoIP in terms of jitter,
and BW varies based
on the resolutions
§ Latency ≤ 150 ms
§ Jitter ≤ 30 ms
§ Loss ≤ 0.05%
§ Bandwidth (1Mbps)
One-Way Requirements
§ Latency ≤ 200 ms
§ Jitter ≤ 20 ms
§ Loss ≤ 0.10%
§ Bandwidth (5.5-16Mbps)
One-Way Requirements
BRKRST-2041
Smooth/bursty
Benign/greedy
Drop insensitive
Delay insensitive
TCP retransmits
Traffic patterns for
Data Vary Among
Applications
§
§
§
§
§
Data Classes:
Mission-Critical Apps
Transactional/Interactive Apps
Bulk Data Apps
Best Effort Apps (Default)
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
43
policy-map WAN
class VOICE
priority percent 10
class VIDEO
priority percent 23
class CRITICAL-DATA
bandwidth percent 15
random-detect dscp-based
class DATA
bandwidth percent 19
random-detect dscp-based
class SCAVENGER
bandwidth percent 5
class NETWORK-CRITICAL
bandwidth percent 3
service-policy MARK-BGP
class class-default
Packets
bandwidth percent
25
random-detect
Out
Scheduling Tools
LLQ/CBWFQ Subsystems
IOS Interface Buffers
1 Mbps
VoIP
Policer
5 Mbps
LLQ
RT-Interactive
Policer
Packets
In
CBWFQ
Scheduler
Tx-Ring
CBWFQ
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
44
Traffic Shaping
Line
Rate
Without Traffic Shaping
With Traffic Shaping
Shaped
Rate
Traffic Shaping Limits the Transmit Rate to a Value Lower Than Line Rate
§
Policers typically drop traffic
§
Shapers typically delay excess traffic, smoothing bursts
and preventing unnecessary drops
§
Very common with Ethernet WAN, as well as NonBroadcast Multiple-Access (NBMA) network topologies
such as Frame-Relay and ATM
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
45
Hierarchical QoS For Subrate Service
H-QoS Policy on WAN Interface, Shaper = CIR
Two Levels MQC
Policy-map PARENT
class class-default
shape average 150000000
service-policy output CHILD
Interface gigabitethernet 0/1
service-policy output PARENT
Policy-map CHILD
class VOICE
priority percent 10
class VIDEO
priority percent 23
class CRITICAL-DATA
bandwidth percent 15
random-detect dscp-based
class DATA
bandwidth percent 19
random-detect dscp-based
class SCAVENGER
bandwidth percent 5
class NETWORK-CRITICAL
bandwidth percent 3
service-policy MARK-BGP
class class-default
bandwidth percent 25
random-detect
Gig 0/1
Service Level
Best Effort
Scavenger
150 Mbps
Video
Critical Data
Voice
BRKRST-2041
Network
Critical
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
46
MPLS VPN QoS Considerations
MPLS VPN Port QoS Roles
Campus VPN
Block
Branch 1
E
F
F
MPLS VPN
E
F
F
E
E
Branch 2
CE Routers
PE Routers
CE Routers
Enterprise Subscriber (Unmanaged CE Routers)
E Outbound Policies:
Inbound Policies:
HQoS Shaper (if required)
+ LLQ for VoIP (EF)
Trust DSCP
+ LLQ or CBWFQ for RT-Interactive (CS4)
+ Remark RTI (if necessary)
+ Restore RT-Interactive to CS4 (if necessary)
+ CBWFQ for Signalling (CS3)
+ Remark Signalling (if necessary) + Restore Signalling to CS3 (if necessary)
≤ 33%
of BW
Service Provider:
Outbound Policies:
F
+ LLQ for Real-Time
+ CBWFQ for Critical Data
Inbound Policies:
Trust DSCP
Police on a per-Class Basis © 2017
Cisco and/or its affiliates. All rights reserved. Cisco Public
47
GRE/IPSec QoS Consideration
ToS Byte Preservation
ToS
ToS byte is copied to
the new IP Header
IP HDR
IP Payloaad
GRE
HDR
IP HDR
ToS
ToS
GRE Tunnel
IP HDR
IP Payload
IP HDR
ESP HDR
ToS
ToS
IPSec Tunnel mode
IP HDR
IP Payload
BRKRST-2041
ESP
Trailer
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
ESP
Auth
48
GRE and IPsec Network QoS Design
Direction of Packet Flow
DSCP CS5
Packet initially marked to
DSCP AF41according to
RFC-4594
DSCP CS5
DSCP AF41
DSCP CS5
DSCP CS5
By default, ToS values are
copied to IPsec header
DSCP CS5
Topmost ToS value is
rewritten on egress to match
service provider classes
§ Remark DSCP on egress to align with each SP’s
SLA class of service requirements
§ H-QOS with shaping to offered rate on egress
§ Hub per tunnel QOS to minimise spoke
oversubscription
Re-marks the DSCP value on the
encrypted and encapsulated header on
the egress interface
BRKRST-2041
Packet decapsulated to
reveal the original ToS byte
policy-map WAN-SP-CLASS-OUTPUT
class VOICE
priority percent 10
class VIDEO-INTERACTIVE
priority percent 23
set dscp af41
class NETWORK-MGMT
bandwidth percent 5
service-policy MARK-BGP
class class-default
bandwidth percent 25
random-detect
!
policy-map Int-Gig-Agg-HE
class class-default
shape average 1000000000
service-policy WAN-Out
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
49
Per Site Traffic Shaping to Avoid Overruns
DMVPN Per-Tunnel QoS
§ User NHRP group to dynamically provision HQoS
policy on a DMVPN hub per-spoke basis
Data Centre/HQ
Spoke: Configure NHRP group name
Hub: NHRP group name mapped to QoS template policy
CE
Multiple spokes with same NHRP group mapped to
individual instances of same QoS template policy
§ GRE ,IPsec &L2 header are included in
calculations for shaping and bandwidth.
150 Mbps
50 Mbps
§ Queuing and shaping is performed at the outbound
physical interface
20 Mbps
10 Mbps
10 Mbps
§ Can be used with DMVPN with or without IPSec.
Remote
Branches
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
50
Per-tunnel QoS Configurations
class-map match-all typeA_voice
match access-group 100
class-map match-all typeB_voice
match access-group 100
class-map match-all typeA_Routing
match ip precedence 6
class-map match-all typeB_Routing
match ip precedence 6
policy-map typeA
class typeA_voice
priority 1000
class typeA_Routing
bandwidth percent 20
policy-map typeB
class typeB_voice
priority percent 20
class typeB_Routing
bandwidth percent 10
policy-map typeA_parent
class class-default
shape average 3000000
service-policy typeA
policy-map typeB_parent
class class-default
shape average 2000000
service-policy typeB
Hub
For Your
Reference
interface Tunnel0
ip address 10.0.0.1 255.255.255.0
…
ip nhrp map group typeA service-policy output typeA_parent
ip nhrp map group typeB service-policy output typeB_parent
…
ip nhrp redirect
no ip split-horizon eigrp 100
ip summary-address eigrp 100 192.168.0.0 255.255.192.0 5
…
Hub (cont)
interface Tunnel0
ip address 10.0.0.11 255.255.255.0
…
ip nhrp group typeA
ip nhrp map multicast 172.17.0.1
ip nhrp map 10.0.0.1 172.17.0.1
ip nhrp nhs 10.0.0.1
…
Spoke1
interface Tunnel0
ip address 10.0.0.12 255.255.255.0
…
ip nhrp group typeB
ip nhrp map multicast 172.17.0.1
ip nhrp map 10.0.0.1 172.17.0.1
ip nhrp nhs 10.0.0.1
…
Spoke2
interface Tunnel0
ip address 10.0.0.13 255.255.255.0
…
ip nhrp group typeA
ip nhrp map multicast 172.17.0.1
ip nhrp map 10.0.0.1 172.17.0.1
ip nhrp nhs 10.0.0.1
…
Spoke3
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
51
WAN Architecture Design and Best
Practices
Cisco Validate Design
MPLS WAN Technology Design Guide
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
53
WAN Aggregation Reference Design
Data
Centre/
Campus
Campus/
Data Centre
WAAS Service
WAN
Key
Servers
Services/
Distribution
VPN Termination
WAN Edge
MPLS A
MPLS B
Internet
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
54
Remote Branch
Transport & Redundancy Options
Non-Redundant
MPLS
Redundant-Links
& Routers
Redundant-Links
MPLS
MPLS
MPLS
Internet
MPLS
MPLS
MPLS WAN
MPLS +
Internet WAN
Internet
Internet
MPLS
Internet
Internet
Internet WAN
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
55
Routing Topology at WAN Aggregation
Core Layer
Campus/
Data Centre
WAN Distribution
Layer
EIGRP AS 100
Summaries+
Default
DMVPN Hub
Routers
EIGRP AS = 100
BGP AS = 65511
iBGP
EIGRP AS = 100
MPLS CE
Routers
BGP AS = 65511
eBGP
MPLS A
MPLS B
EIGRP AS = 100
Internet Edge
EIGRP AS = 200
Layer 2 WAN
CE Router
Layer 2
WAN
DMVPN 1 DMVPN 2
BRKRST-2041
Internet
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
56
WAN Edge
Connection Methods Compared
Recommended
Multi-Chassis
EtherChannel
VSS/3850
Stacks
Si
Shared
Si
Si
Layer 3
Si
P-to-P Link
LAN
WAN
WAN
WAN
§
§
No Static Routes
No First Hop Redundancy Protocols
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
57
Optimise Convergence and Redundancy
Multi-chassis EtherChannel
VSS/3850
Stacks
Si
Layer 3
Si
P-to-P Link
IGP recalc
Channel
Member
Removed
§ Link redundancy achieved through
redundant L3 paths
§ Provide Link Redundancy and reduce
peering complexity
§ Flow based load-balancing through CEF
forwarding across
§ Tune L3/L4 load-balancing
hash to achieve maximum utilisation
§ Routing protocol reconvergence when uplink § No L3 reconvergence required when
member link failed
failed
§ Convergence time may depends on routing § No individual flow can go faster than
the speed of an individual member of
protocol used and the size of routing entries
the link
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
58
Link Recovery Comparison
ECMP vs. Multichassis EtherChannel
§
ECMP convergence is dependent on the number of
routes
Si
Layer 3
Si
P-to-P Link
MEC convergence is consistent, independent of the
number of routes
2.5
ECMP
MEC Max
2
sec of lost voice
§
VSS/3850
Stacks
1.5
1
0.5
0
1000
3000
6000
9000
12000
Number of Routes – Sup2T
Number of Routes - Sup720C
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
59
Redundancy vs. Convergence Time
More Is Not Always Better
§ In principle, redundancy is easy
§ Any system with more parallel paths
through the system will fail less
often
§ Increasing parallel paths increases
routing complexity, therefore
increasing convergence times
2.5
Seconds
§ The problem is a network isn’t really
a single system but a group of
interacting systems
0
BRKRST-2041
Routes
10000
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
60
Best Practice —
Summarise at Service Distribution
§
It is important to force summarisation
at the distribution towards WAN Edge
and towards campus & data centre
§
Summarisation provides topology
change isolation.
§
Summarisation reduce routing table
size.
Campus/
Data Centre
Summary
10.5.0.0/16
Summaries +
Default
10.4.0.0/16
0.0.0.0/0.0.0.0
interface Port-channel1
description Interface to MPLS-A-CE
no switchport
ip address 10.4.128.1 255.255.255.252
ip pim sparse-mode
ip summary-address eigrp 100 10.5.0.0 255.255.0.0
MPLS A
BRKRST-2041
MPLS B
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
61
Best Practice –
Preventing Routing Loops with Route Tag and Filter
§ Mutual route redistribution between protocols can
cause routing loops without preventative measures
IGP Domain
(EIGRP/OSPF)
§ Use route-map to set tags and then redistribute
based on the tags
Campus
§ Routes are implicitly tagged when distributed from
eBGP to EIGRP/OSPF with carrier AS
§ Use route-map to block re-learning of WAN routes
via the distribution layer (already known via iBGP)
router eigrp 100
distribute-list route-map BLOCK-TAGGED-ROUTES in
default-metric [BW] 100 255 1 1500
redistribute bgp 65500
route-map BLOCK-TAGGED-ROUTES deny 10
match tag 65401 65402
MPLS WAN
BGP Domain
route-map BLOCK-TAGGED-ROUTES permit 20
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
62
Dual Carriers with BGP as CE-PE Protocol
Use iBGP for Path Selection
§
Run iBGP between the CE routers to
exchange prefixes associated with each
carrier
§
CE routers will use only BGP path selection
information to select both the primary and
secondary preferences for any destinations
announced by the IGP and BGP
§
Use IGP (OSPF/EIGRP) for prefix readvertisement will result in equal-cost paths
at remote-site
bn-br200-3945-1# sh ip bgp 10.5.128.0/21
BGP routing table entry for 10.5.128.0/21, version 71
Paths: (2 available, best #2, table default, RIB-failure(17))
Not advertised to any peer
65401 65402, (aggregated by 65511 10.5.128.254)
10.4.142.26 from 10.4.142.26 (192.168.100.3)
Origin IGP, localpref 100, valid, external, atomicaggregate
65402, (aggregated by 65511 10.5.128.254)
10.4.143.26 (metric 51456) from 10.5.0.10 (10.5.0.253)
Origin IGP, metric 0, localpref 100, valid, internal,
atomic-aggregate, best
Campus
10.5.128.0/21
iBGP
MPLS B
MPLS A
B
A
iBGP
10.5.128.0/21
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
63
Best Practice - Implement AS-Path Filter
Prevent Branch Site Becoming Transit Network
§
Dual carrier sites can unintentionally become
transit network during network failure event and
causing network congestion due to transit traffic
§
Design the network so that transit path between
two carriers only occurs at sites with enough
bandwidth
§
Implement AS-Path filter to allow only locally
originated routes to be advertised on the
outbound updates for branches that should not
be transit
router bgp 65511
neighbor 10.4.142.26 route-map NO-TRANSIT-AS out
!
ip as-path access-list 10 permit ^$
!
route-map NO-TRANSIT-AS permit 10
match as-path 10
Campus
MPLS B
MPLS A
B
A
iBGP
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
64
Golden Rules
For Your
Reference
Route Preference for EIGRP & OSPF
EIGRP
OSPF
§
Internal EIGRP – Admin Dist. 90
§
Admin Dist. 110
§
External EIGRP – Admin Dist. 170
§
Route Preference
§
Metric Calculation
metric = bandwidth + delay
1.
2.
3.
Bandwidth (in kb/s)
• Delay (in microseconds)
•
4.
§
Intra-Area
Inter-Area
External E1 (Internal + External Cost)
External E2 (External Cost)
Cost Calculation
Cost= Reference BW / Interface BW
Default Reference BW = 100Mbps
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
65
MPLS + Internet WAN
Prefer the MPLS Path over Internet
Campus
EIGRP
AS100
§ eBGP routes are redistributed into EIGRP 100 as
external routes with default Admin Distance 170
§ Running same EIGRP AS for both campus and
DMVPN network would result in Internet path preferred
over MPLS path
10.4.128.2
eBGP
Internet
MPLS A
EIGRP
AS100
10.5.48.0/21
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
66
MPLS + Internet WAN
Use Autonomous System for IGP Path Differentiation
Campus
D EX
EIGRP
AS100
10.5.48.0/21 [170/28416] via 10.4.128.2
§
eBGP routes are redistributed into EIGRP 100 as
external routes with default Admin Distance 170
§
Running same EIGRP AS for both campus and
DMVPN network would result in Internet path preferred
over MPLS path
§
Multiple EIGRP AS processes can be used to provide
control of the routing
10.4.128.2
eBGP
§
Internet
MPLS A
EIGRP
AS200
§
§
EIGRP 100 is used in campus location
EIGRP 200 over DMVPN tunnels
Routes from EIGRP 200 redistributed into EIGRP 100 appear as
external route (distance = 170)
Routes from both WAN sources are equal-cost paths.
To prefer MPLS path over DMVPN use eigrp delay to
modify path preference
MPLS CE router#
10.5.48.0/21
router eigrp 100
default-metric 1000000 10 255 1 1500
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
67
MPLS VPN BGP Path with IGP Backdoor Path
§ eBGP as the PE-CE Routing Protocol
Campus
§ MPLS VPN as preferred path learned via
eBGP
R2
MPLS A
Internet
IGP Backup Link
§ Default configuration the failover to backup
path works as expected
R1
eBGP
§ Secondary path via backdoor IGP link
(EIGRP or OSPF) over tunneled connection
(DMVPN over Internet)
EIGRP
AS100
10.4.160.0/24
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
68
MPLS VPN BGP Path with IGP Backdoor Path
Campus
§ After link restore, MPLS CE router receives
BGP advertisement for remote-site route.
§ Does BGP route get (re)installed in the route
table?
R1
R2
MPLS A
Internet
R1# show ip route
B
10.4.144.0/24 [20/0] via 10.4.142.2, 01:30:06
B
10.4.145.0/24 [20/0] via 10.4.142.2, 01:30:06
D EX 10.4.160.0/24 [170/3584] via 10.4.128.9, 00:30:06
B
10.4.160.0/24 [20/0]....
IGP Backup Link
eBGP
D EX 10.4.160.0/24 [170/3584]....
EIGRP
AS100
10.4.160.0/24
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
69
For Your
Reference
BGP Route Selection Algorithm
BGP Prefers Path with:
1.
Highest Weight
2.
Highest Local Preference
3.
Locally originated (via network or aggregate BGP)
4.
Shortest AS_PATH
5.
Lowest Origin type
IGP>EGP>INCOMPLETE (redistributed into BGP)
6.
Lowest Multi-Exit Discriminator (MED)
7.
Prefer Externals (eBGP over iBGP paths)
8.
Lowest IGP metric to BGP next hop (exit point)
9.
Lowest Router ID for exit point
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
70
BGP Prefers Path with Highest Weight
§
Routes redistributed into BGP are considered locally originated and get a default
weight of 32768
§
The eBGP learned prefix has default weight of 0
§
Path with highest weight is selected
ASR1004-1#show ip bgp 10.4.160.0 255.255.255.0
BGP routing table entry for 10.4.160.0/24, version 22
Paths: (3 available, best #3, table default)
Advertised to update-groups:
4
5
65401 65401
10.4.142.2 from 10.4.142.2 (192.168.100.3)
Origin IGP, localpref 200, valid, external
Local
10.4.128.1 from 0.0.0.0 (10.4.142.1)
Origin incomplete, metric 26883072, localpref 100, weight 32768, valid, sourced, best
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
71
Prefer the eBGP Path over IGP
Set the eBGP weight > 32768
§
To resolve this issue set the weights on route learned via eBGP peer higher
than 32768
neighbor 10.4.142.2 weight 35000
ASR1004-1#show ip bgp 10.4.160.0 255.255.255.0
BGP routing table entry for 10.4.160.0/24, version 22
Paths: (1 available, best #1, table default)
Not advertised to any peer
65401 65401
10.4.142.2 from 10.4.142.2 (192.168.100.3)
Origin IGP, metric 0, localpref 100, weight 35000, valid, external, best
ASR1004-1#show ip route
....
B
10.4.160.0/24 [20/0] via 10.4.142.2, 05:00:06
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
72
Secure WAN Communication
Dual Carrier GETVPN Topology
COOP Key Server
Key Servers
GM
GM
MPLS A
MPLS B B
MPLS
MPLS A
GM
GM
GM
GM
GM
BRKRST-2041
GM
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
74
Best Practice - High Availability with Cooperative Key Servers
§
Two or more KSs known as COOP KSs manage a common set of keys and security policies for
GETVPN group members
§
Group members can register to any one of the available KSs
§
Cooperative KSs periodically exchange and synchronise group’s database, policy and keys
§
Primary KS is responsible to generate and distribute group keys
Cooperative KS1
Cooperative KS2
Subnet 1
Subnet 2
GM 1
GM 2
IP Network
Subnet 3
Subnet 4
GM 4
GM 3
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
75
Best Practice - Key Server Recommendations
•
For Your
Reference
Maintain reliable KS communication:
Insure multiple routing paths exist between all KS
• Use loopback interface for KS registration and Cooperative KS protocol Use IKE keepalive for KS-KS communication
•
•
Use only globally applicable policies in KS proxy identifiers:
Site specific policies should be applied at the GM
• Goal is to create symmetric policies on KS
• Exception policy development should be done on GM, not KS
•
•
Use sufficiently long key lifetimes to minimise key transitions:
Traffic Encryption Key (TEK) > 3600 sec
• Key Encryption Key (KEK) > 86400 sec
•
•
Insure rekey interval extends longer than routing convergence time
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
76
Transition from Clear-text to GETVPN
SA Receive-Only Method
§
§
§
Goal
•Incrementally deploy infrastructure
without encryption
•Immediate transition to encryption
controlled by KS
Method
•Deploy KS with Receive-only SA’s
(don’t encrypt, allow decryption)
•Deploy GM throughout infrastructure
and monitor rekey processes
•Transition KS to Normal SA (encrypt,
decrypt)
Assessment
•Pro: Simple transition to network-wide
encryption
•Con: Correct policies imperative
•Con: Deferred encryption until all CE
are capable of GM functions
permit ip 10.1.4.0 0.0.1.255 10.1.4.0 0.0.1.255
10.1.4.0/24
KS
10.1.6.0/24
GM
GET
GM
GM
GM
10.1.5.0/24
10.1.7.0/24
permit ip 10.1.4.0 0.0.3.255 10.1.4.0 0.0.3.255
10.1.4.0/24
KS
10.1.6.0/24
GM
GET
GM
GM
GM
10.1.5.0/24
BRKRST-2041
10.1.7.0/24
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
77
Group Member
For Your
Reference
Secured Group Member Interface
interface Serial0/0
ip address 192.168.1.14 255.255.255.252
crypto map svn
access-group pack-filter out
<- WAN ENCRYPTION
<- ALLOW IPsec and Control
Packet filter (after encryption)
ip access-list extended pack-filter
permit esp any any
permit ip host 192.168.1.14 host 192.168.1.13
permit tcp host 192.168.1.14 eq ssh any
<- ALLOW IPsec
<- ALLOW ROUTE ADJACENCY
<- ALLOW SECURE SHELL
Crypto Map Association to Group Security
crypto map svn 10 gdoi<- GROUP CRYPTO MAP ENTRY
set group secure-wan
<- GROUP MEMBERSHIP
match address control_plane
<- LOCAL POLICY (EXCLUDE)
Group Member Policy Exceptions
ip access-list extended control_plane
<- CONTROL PLANE PROTOCOLS
deny ip host 192.168.1.14 host 192.168.1.13 <- PE-CE LINK (BGP, ICMP)
deny tcp host 192.168.1.14 eq ssh any
<- MANAGEMENT SECURE SHELL
Group Member Association
crypto gdoi group secure-wan
identity number 3333
server address ipv4 <ks1_address>
server address ipv4 <ks2_address>
<<<<-
GROUP ENCRYPTION
MEMBER’S GROUP IDENTITY
KS ADDRESS TO REGISTER
ALTERNATE KS REGISTRATION
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
78
Key Server
For Your
Reference
crypto gdoi group secure-wan
identity number 3333
<- GROUP ID
server local
<- KEY SERVER
rekey retransmit 40 number 3
<- REKEY RETRANSMITS
rekey authentication mypubkey rsa my_rsa <- KS MSG AUTHENTICATION
rekey transport unicast
<- Unicast Rekey
saipsec 10
<- SECURITY ASSOCIATION
profile GETVPN-GDOI-PROFILE
<- CRYPTO ATTRIBUTES SELECTION
match address ipv4ipsec-policy
<- ENCRYPTION POLICY
no replay
<- NO ANTI-REPLAY
address ipv4 <ks_address>
<- KS ADDRESS
Crypto Attributes
crypto ipsec profile GETVPN-GDOI-PROFILE
set security-association lifetime seconds 7200
set transform-set AES256/SHA
<- AES256 for Encryption and SHA for Hash
Encryption IPsec Proxy ID’s (mandatory)
ip access-list extended ipv4ipsec-policy
deny udp any eq 848 any eq 848
permit ip 10.0.0.0 0.255.255.255 10.0.0.0 0.255.255.255
permit ip 10.0.0.0 0.255.255.255 232.0.0.0 0.255.255.255
<<<<-
BRKRST-2041
ENCRYPTION POLICY
ALLOW GDOI
UNICAST
MULTICAST
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
79
DMVPN Deployment over Internet
Multiple Default Routes for VPN Headend
§
§
VPN Headend has a default route to ASA
firewall’s VPN-DMZ interface to reach
Internet
Remote site policy requires centralised
Internet access
default
INSIDE
default
Internet Edge
Block
default
§
Enable EIGRP between VPN headend &
Campus core to propagate default to remote
§
Static default (admin dist=0) remains active,
§
VPN-DMZ is wrong firewall interface for user
traffic
§
Adjust admin distance so EIGRP route
installed (to core)
§
VPN tunnel drops
VPN-DMZ
OUTSIDE
default
Internet
Internet
Internet
default
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
80
DMVPN Deployment over Internet
§
The RED-VRF contains the default
route to VPN-DMZ Interface needed
for Tunnel Establishment
default
EIGRP
§
Enable FVRF with DMVPN to
separate out the two default routes
default
INSIDE
Internet Edge
Block
default
§
§
§
2nd
A
default route exist on the Global
Routing Table used by the user data
traffic to reach Internet
VPN-DMZ
OUTSIDE
To prevent split tunnelling the default
route is advertised to spokes via
Tunnel
default
default
Internet
Internet
Spoke’s tunnel drops due to 2nd
default route conflict with the one
learned from ISP
default
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
81
Best Practice – VRF-aware DMVPN
Keeping the Default Routes in Separate VRFs
Same Technique
Enable FVRF DMVPN on the Spokes
§
Allow the ISP learned Default Route in the
RED-VRF and used for tunnel
establishment
§
§
EIGRP
§
default
default
INSIDE
Internet
Edge Block
default
Global VRF contains Default Route
learned via tunnel. User data traffic follow
Tunnel to INSIDE interface on firewall
VPN-DMZ
OUTSIDE
default
default
Allow for consistency for implementing
corporate security policy for all users
Internet
Internet
default
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
82
DMVPN and FVRF
Configuration Example
DMVPN Tunnel
Interface
Global
Routing Table
Global
Branch LAN Traffic
VRFs have independent
routing and forwarding planes
F-VRF
ip vrf RED
rd 65512:1
!
crypto keyring DMVPN-KEYRING vrf RED
pre-shared-key address 0.0.0.0 0.0.0.0 key cisco123
!
!
crypto isakmp policy 10
encr aes 256
authentication pre-share
group 2
!
crypto isakmp keepalive 30 5
!
crypto isakmp profile FVRF-ISAKMP-RED
keyring DMVPN-KEYRING
match identity address 0.0.0.0 RED
!
Front Side
Provider VRF
Internet
interface GigabitEthernet0/1
ip vrf forwarding RED
ip address dhcp
!
interface Tunnel10
ip address 10.4.132.201 255.255.254.0
….
tunnel mode gre multipoint
tunnel vrf RED
tunnel protection ipsec profile DMVPN-PROFILE
!
router eigrp 200
network 10.4.132.0 0.0.0.255
network 10.4.163.0 0.0.0.127
eigrp router-id 10.4.132.201
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
83
Best Practices — Enable Dead Peer Detection (DPD)
Improve DMVPN Network Convergence
§
Dead Peer Detection (DPD) is a mechanism for
detecting unreachable IKE peers
§
Each peer’s DPD state is independent of the others
§
Without DPD spoke routers will continue to encrypt
traffic using old SPI which would be dropped at the
hub. May take up to 60 minutes for spokes to
reconverge
§
Use ISAKMP keepalives on spokes
tun10
Internet
crypto isakmp keepalives <initial> <retry>
•
•
§
ISAKMP invalid-SPI-recovery is not useful with DMVPN
ISAKMP keepalive timeout should be greater than
routing protocol hellos
Not recommended for Hub routers – may cause an
increase of CPU overhead with large number of peers
Informational RFC 3706
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
84
DMVPN Internet Deployment
Dynamic IP Address Assignment on the Spokes
§
Spokes are receiving dynamic address assignment from
the ISP
§
Spoke reboots and receive a new IP address from the
ISP, VPN session is established but no traffic passes
§
Following error message appears on the spoke
"%NHRP-3-PAKREPLY: Receive Registration
Reply packet with error - unique address
registered already(14)"
§
Hub router (NHS) reject registration attempts for the
same private address that uses a different
NBMA address
§
To resolve this issue, configure following command on
spoke routers –
Internet
ip nhrp registration no-unique
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
85
Best Practices — Enable PIM NBMA-Mode
Multicast over DMVPN
§
By default router uses OIL to correlate multicast
group join to interface
§
This causes problem when hub is connected to
multiple spokes over NBMA network
§
Any spoke that leaves a multicast group would case
all the spokes to be pruned off the multicast group
§
Enable PIM NBMA mode under tunnel interface on
hubs and spokes
§
§
Multicast
PIM
Prune
towards
RP
Internet
ip pim nbma-mode
§
Allows the router to track multicast joins based on IP
address instead of interface
§
Applies only to PIM sparse-mode
Router treats NBMA network as a collection of pointto-point circuits, allowing remote sites to be pruned
off traffic flows
PIM
Prune
IGMP
Leave
Receiver
BRKRST-2041
Receiver
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
86
Best Practices —
Avoid Fragmentation with IPSec VPN
GRE+IPsec
MTU 1500
MTU 1400
MTU 1500
Tunnel Setting (AES256+SHA)
Minimum MTU
Recommended MTU
GRE/IPSec (Tunnel Mode)
1414 bytes
1400 bytes
GRE/IPSec (Transport Mode)
1434 bytes
1400 bytes
IP fragmentation will cause CPU and memory overhead and resulting in
lowering throughput performance
§ When one fragment of a datagram is dropped, the entire original IP
datagram will have to be resent
§ Use ‘mode transport’ on transform-set
§
•
§
NHRP needs for NAT support and saves 20 bytes
Avoid MTU issues with the following best practices
•
•
ip mtu 1400
ip tcp adjust-mss 1360
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
87
MACsec Layer 2 Encryption for WAN
§
Ethernet is not only in the campus and data centre any longer
§
Ethernet is growing rapidly as a WAN & Metro “transport” service
§
WAN/Metro SP offerings are replacing existing T1, ATM/FR, and SONET OC-x
options
§
Ethernet services apply to many areas of the WAN/MAN:
WAN links for core, edge, remote branch
• PE-CE links (leveraging L3 VPN services)
• Metro-E service hand-offs (P2P, P2MP)
•
§
Cisco’s goal is to integrate MACsec as part of new Ethernet interface/LC
development moving forward on routers leveraged in WAN, MAN, Branch
Market Transition – Ethernet WAN Transport
BRKRST-2041
ASR 1001-X XE3.14
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
88
WAN MACSec - Line Rate Encryption L2 Solution
TrustSec Frame Format
Authenticated
Encrypted
DMAC
SMAC
802.1Q
CMD
802.1AE Header
ETYPE
PAYLOAD
ICV
CRC
0x88e5
MISEec EtherType
TCI/AN
SL
Packet Number
SCI (optional)
Simple
•
•
Easy to configure. Simple configuration on interface level only. No GRE Tunnel establishment.
Reduced interoperability issues with other L3 Features
Secure
•
Leverage Advanced Encryption Standard Galios/Counter Mode (AES-GCM-128)
Line Rate Encryption
•
•
Leverages “line rate” Ethernet performance of the port (PHY). Speeds 1/10G, 40G, 100G
Ethernet WAN deployments driving increasing need for higher crypto bandwidths.
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
89
MACsec Deployment Model
Ring Topology
IPv4/v6
IPv4/v6
IPv4/v6
P-t-P E-LINE
P-t-MP E-LAN
CE2
Routers peer per
VLAN subinterface per PW
Point-to-point dark fibre
IPv4/v6
IPv4/v6
IPv4/v6
CE4IPv4/v6
Ethernet Sub-interface with
802.1q support
CE3
IPv4/v6
Clear-Tag Feature: Leverage 802.1Q tag support
for full/partial-mesh over Metro-Ethernet
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
90
WAN Extension into the Cloud
Connectivity Options into AWS Cloud
Corporate DC
AWS Managed VPN
Internet
Cisco
ISR/ASR
VGW
Private
Network
Corporate DC
CSR 1000V
AWS Direct
Connect
Colocation Facility
CSR 1000V
BRKRST-2041
Cisco
ISR/ASR
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
92
AWS Direct Connect Overview
§
Dedicated private connection separated from Internet
§
Over of 40 Direct Connect locations worldwide
§
Each Direct Connect circuit attaches to one AWS Region
§
Uses 802.1q VLANS to partition into multiple virtual interfaces
§
Reduces network costs into/out of AWS for bandwidth-heavy workloads
§
Provides consistent network performance
§
Offers 50Mbps – 500Mbps, 1Gbps, and 10Gbps connectivity options
§
Uses eBGP peering for route exchange (<100 routes)
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
93
AWS Virtual Private Cloud (VPC) Overview
§
§
§
Represents logically isolated network
q
Separate IP range, Subnet, Routes, and
Security
q
Overlap IP address allowed
NO Support for Multicast/Broadcast traffic,
VLANs, and Transit Routing
Internet GW
Subnet
Router
Network Controls Available to you:
q
Route Tables, aka VRFs
q
Internet Gateway (IGW), aka an Internet Router
q
Security Groups and Network ACLs
q
VPNaaS
§
Internet gateway forward traffic to
outsides and in between VPCs
§
Subnet Router forward traffic within
the VPC
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
94
CSR1000v placement in the AWS network
NAT at the Internet GW
§
§
§
§
§
Will break services that do not work
over NAT, such as GET-VPN
Tunnel source will be a private
address
Tunnel destination from the
perspective of VPN peers will be a
public address
Assign EC2 elastic IP address so that
address does not change if the
CSR1K is shutdown
Other VPCs see Elastic IP address
unless using VPC peering
Maps to AWS
Elastic IP
10.2.2.10
Gi2
10.2.1.10
Gi1 10.2.1.11
10.1.2.10
Gi2
10.1.1.10
Gi1
10.1.1.11
Internet IP
54.x.x.x
•
CSR should be the default gateway
for the application VMs
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
95
AWS Transit Routing Limitations
§
You have a VPC peering connection
between VPC A and VPC B
§
And between VPC A and VPC_C
§
But you cannot route packets
directly from VPC B to VPC C
through VPC
B
X
C
A
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
96
Inter-VPN Design using Transit VPCs with CSR1000v
§
End to end routings between
VPCs in all regions and other
none AWS network
§
Used for hierarchical designs
•
•
Scaling beyond VPC peering
limits
Security/Monitoring Services in
Transit VPC
§
Redundant CSRs for HA and
fast convergence
§
Can use VRFs to resolve
recursive routing issues
App VPC
3rd Party VPC
Shared Services VPC
VGW
VPC
peering
VPC
peering
Availability
Zone 1
Availability
Zone 2
Transit VPC
Direct
Connect
Corporate Network/DC
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
97
Transit VPC to Interconnect AWS Regions
AZ 1
AZ 2
Transit VPC
US-West-2
AZ 1
AZ 2
Transit VPC
US-East-2
Corporate Network/DC
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
98
CSR 1000v Advantages over…
Virtual Private Gateway:
VPC Peering:
•
Scalability
•
Overlapping CIDR blocks
•
Continuity of Operations
•
Peering between regions
•
Spoke-to-spoke routing
•
Transitive peering relationships
•
Richer routing features
•
Multiple peerings per VPC
•
Security/Application Visibility
•
Unicast Reverse Path Forwarding
•
Spoke-to-spoke routing
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
99
CSR 1000V Licensing Structure
Example:
Pick one option from each column…
Technology Package
(See next slide for details)
IP Base
Throughput
10 Mbps
50 Mbps
100 Mbps
SEC
License Type
IP Base
250 Mbps
1-Year
Subscription
(1-year, 3-year or perpetual)
250 Mbps
500 Mbps
AppX
1 Gbps
2.5 Gbps
Service Provider License
Agreement (SPLA)
5 Gbps
AX
10 Gbps
* CSR add-on license options not shown above
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
100
CSR 1000V Features Per Technology Package
Technology
Package
IOS-XE Features
§
§
§
§
§
§
IPBase
(formerly Standard)
SEC
(formerly Advanced)
Basic Networking: BGP, OSPF, EIGRP, RIP, ISIS, IPv6, GRE, VRF-LITE, NTP, QoS
Multicast: IGMP, PIM
High Availability: HSRP, VRRP, GLBP
Addressing: 802.1Q VLAN, EVC, NAT, DHCP, DNS
Basic Security: ACL, AAA, RADIUS, TACACS+
Management: IOS-XE CLI, SSH, Flexible NetFlow, SNMP, EEM, NETCONF
IPBase Plus…
§
Advanced Security: Zone Based Firewall, IPSec VPN, EZVPN, DMVPN, FlexVPN,
SSLVPN, GETVPN (express route/direct connect only)
IPBase Plus…
§
Advanced Networking: L2TPv3, BFD, MPLS (tunneled), VRF, VXLAN
§
Application Experience: WCCPv2 (no clustering), AppXNAV, NBAR2, AVC, IP SLA
§
Hybrid Cloud Connectivity: LISP (L3 only), OTV, VPLS, EoMPLS
§
Subscriber Management: PTA, LNS, ISG
AppX
AX
(formerly Premium)
§
ALL FEATURES
Features in Red will not work in Amazon – infrastructure issues (lack of L2 support and Multicast support)
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
101
CSR1000v on AWS performance
CSR1000v Amazon AMI
Throughput Tests
PV - Instance types emulate I/O
devices, low to medium performance,
cheaper instance type
Build: 15.5(3)S2 (XE 03.16.02.S)
HVM - Required to take advantage of
hardware virtualisation acceleration
(e.g. SR-IOV) Best performance,
pricier instance types
All numbers in Mbps
CEF 64
CEF 1400
CEF IMIX
IPSec 1400
IPSec IMIX
PV m3.medium
PV m3.large
PV m3.xlarge
PV c3.large
PV c3.xlarge
PV c3.2xlarge
PV c3.4xlarge
35
67
76
79
73
80
70
313
696
682
534
760
1,031
1,069
108
215
297
255
281
303
306
237
454
782
514
742
1,005
1,023
97
230
250
302
313
336
346
HVM c3.large
HVM c3.xlarge
HVM c3.2xlarge
HVM c3.4xlarge
80
101
203
362
526
758
1,032
2,067
306
382
766
1,254
515
741
1,009
2,020
348
436
874
1,318
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
102
Summary
Modern Hierarchical Global WAN Design
East Theater
Tier 1
West Theater
Tier 2
Global
IP/MPLS Core
In-Theater
IP/MPLS Core
West Region
East Region
Tier 3
Internet
Cloud
Public Voice/Video Mobility
Private
IP
Service
Metro
Service
Public
IP
Service
BRKRST-2041
Metro
Service
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
104
Key Takeaways
§
Understand how WAN characteristics can affect your applications.
•
Bandwidth, latency, loss
§
A modular hierarchical network infrastructure is the foundation for a solid WAN
architecture. Good WAN design have many well-utilised component
§
Encryption is a foundation component of all WAN designs and can be deployed
transparently.
§
Understand how to build wide area network leveraging Internet transport with
Intelligent WAN.
§
Design a network with consistent behaviour that provides predictable
performance.
§
More is not always better. Keep it simple!
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
105
Recommended Reading
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
106
Suggested Sessions
•
BRKRST-2042 Highly Available WAN Design
•
BRKRST-2309 Introduction to WAN MACsec
•
BRKRST-2514 Application Optimisation and Provisioning the IWAN
•
BRKRST-3045 LISP A Next Generation Networking Architecture
•
BRKCRS-2000 Intelligent WAN 2.0 Update
•
BRKCRS-2040 WAN and Remote-Site Deployment using Cisco Validated
Designs
•
BRKSEC-4054 Advanced Concepts of DMVPN
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
107
Continue Your Education
•
Demos in the Cisco campus
•
Walk-in Self-Paced Labs
•
Lunch & Learn
•
Meet the Engineer 1:1 meetings
•
Related sessions
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
108
Q&A
Complete Your Online Session Evaluation
Give us your feedback and receive a
Cisco Live 2017 Cap by completing the
overall event evaluation and 5 session
evaluations.
All evaluations can be completed via the
Cisco Live Mobile App.
Caps can be collected Friday 10 March
at Registration.
Learn online with Cisco Live!
Visit us online after the conference
for full access to session videos and
presentations.
www.CiscoLiveAPAC.com
BRKRST-2041
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
110
Thank you