WAN Architectures and Design Principles Stephen Lynn Consulting Systems Architect BRKRST-2041 Agenda § WAN Technologies & Solutions • • • • § WAN Architecture Design Considerations • • • § WAN Transport Technologies WAN Overlay Technologies WAN Optimisation Wide Area Network Quality of Service WAN Design and Best Practices Secure WAN Communication WAN Extension into the Cloud Summary WAN Transport Technologies The Architectural Continuum Early-Mid 1990s Early Networking Mid 1990s-Mid 2000s Global Scale Flat/Bridged Multiprotocol Large Scale Experimental Networks Business Enabling Mission Critical Architectural Lessons Architectural Lessons Route First, Bridge only if Must Architectural Lessons Redundancy Protocols required for Scale & Restoration 1960 ARPAnet 1960 Internet Protocol 1970 TCP/IP X.25 1980 RIP (BSD) OSPF, ISDN, ATM 1990 IP Ubiquity Business Survival Build to Scale Frame-Relay BGP Today IPv6 GRE Planning ? DMVPN 2000 Future 2010 Metro-Ethernet Tag SDN Switching BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 5 The Challenge • Build a network that can adapt to a quickly changing business and technical environment • Realise rapid strategic advantage from new technologies • • • • • • IPv6: global reachability Cloud: flexible diversified resources Internet of Things Fast-IT What’s next? Adapt to business changes rapidly and smoothly Mergers & divestures • Changes in the regulatory & security requirements • Changes in public perception of services • BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 6 Network Design Modularity East Theater Tier 1 West Theater Tier 2 Global IP/MPLS Core In-Theater IP/MPLS Core West Region East Region Tier 3 Internet Cloud Public Voice/Video Mobility Private IP Service Metro Service Public IP Service BRKRST-2041 Metro Service © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 7 Hierarchical Network Principle • Use hierarchy to manage network scalability and complexity while reducing routing algorithm overhead • Hierarchical design used to be… Three routed layers • Core, distribution, access • Only one hierarchical structure end-to-end • • Hierarchical design has become any design that… Splits the network up into “places,” or “regions” • Separates these “regions” by hiding information • Organises these “regions” around a network core • “hub and spoke” at a macro level • BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 8 MPLS L3VPN Topology Definition Spoke Site 1 Spoke Site Y Spoke Site 2 SP-Managed MPLS IP WAN Spoke Site X Hub Site (The Network) Equivalent to Spoke Site N Spoke Site 1 Spoke Site 2 Spoke Site X § MPLS WAN is provided by a service provider § As seen by the enterprise network, every site is one IP “hop” away § Equivalent to a full mesh, or to a “hubless” hub-and-spoke BRKRST-2041 Spoke Site Y Spoke Site N © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 9 Virtual Routing and Forwarding Instance (VRF) Provides Network Virtualisation and Path Isolation VRF VRF VRF § Virtualisation at Layer 3 forwarding § Associates to Layer 3 interfaces on router/switch § Each VRF has its own Forwarding table (CEF) Routing process (RIP, OSPF, BGP) § VRF-Lite Hop-by-hop § MPLS VPN Multi-hop VRF VRF VRF ! PE Router – Multiple VRFs ip vrf blue rd 65100:10 route-target import 65100:10 route-target export 65100:10 ip vrf yellow rd 65100:20 route-target import 65100:20 route-target export 65100:20 ! interface GigabitEthernet0/1.10 ip vrf forwarding blue interface GigabitEthernet0/1.20 ip vrf forwarding yellow BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 10 Wide Area Network Design Trends Dual Providers Design Single Provider Design § § § § Enterprise will home all sites into a single carrier to provide L3 MPLS VPN connectivity. Pro: Simpler design with consistent features Con: Bound to single carrier for feature velocity Overlay Network Design § Enterprise will single or dual home sites into one or both carriers to provide L3 MPLS VPN connectivity. § Overlay tunnelling technologies with encryption for provider transport agnostic design § Pro: Protects against MPLS service failure with Single Provider § § Pro: Potential business leverage for better competitive pricing Pro: Can use commodity broadband services for lower cost higher bandwidth service § Pro: Flexible overlay network topology that couples from the physical connectivity § Con: Increased design complexity § Con: Does not protect against MPLS cloud failure with Single Provider § Con: Increased design complexity due to service implementation differences (e.g. QoS, BGP AS Topology) Con: Feature differences between § Con: Additional technology providers could force customer to use needed for SLA over commodity least common denominator features. transport services BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 11 Single Carrier Site Types (Non-Transit) § Dual Homed Non Transit AS 64517 CE3 Only advertise local prefixes (^$) Typically with Dual CE routers CE4 CE5 BGP design: eBGP to carrier AS 200 iBGP between CEs Redistribute cloud learned routes into site IGP CE2 CE1 Site IGP C2 C1 § Single Homed Non Transit Advertise local prefixes and optionally use default route. AS 64512 BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 12 Dual Carrier: Transit vs. Non Transit § To guarantee single homed site reachability to a dual homed site experiencing a failure, transit sites had to be elected. § Transit sites would act as a BGP bridge transiting routes between the two provider clouds. § To minimise latency costs of transits, transits need to be selected with geographic diversity (e.g. from the East, West and Central US.) CE3 CE4 CE5 Transit AS 64545 AS 100 CE1 Site IGP AS 200 CE2 C2 C1 Prefix X Prefix Y Prefix Z BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 13 Single vs. Dual Carriers Single Provider Pro: Common QoS support model Dual Providers Pro: More fault domains Pro: Only one carrier to “tune” Pro: More product offerings to business Pro: Reduced head end circuits Pro: Ability to leverage vendors for better pricing Pro: Overall simpler design Pro: Nice to have a second vendor option Con: Carrier failure could be catastrophic Con: Increased Bandwidth “Paying for bandwidth twice” Con: Do not have another carrier “in your pocket” Con: Increased overall design complexity Con: May be reduced to “common denominator” between carriers Resiliency Drivers vs. Simplicity BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 14 Metro Ethernet Service (L2VPN) § § § § E-Line (Point-to-Point) Replaces TDM private line and Frame-Relay or ATM L2VPN Point-to-point EVCs offer predictable performance for applications One or more EVCs allowed per single physical interface (UNI) Supports “hub & spoke” topology § § § § E-LAN (Point-to-Multipoint) Offers point to multipoint connectivity Transparent to VLANs and Layer 2 control protocols 4 or 6 classes of QoS support Supports service multiplexing (Ex. Internet access and corporate VPN via one UNI) BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 15 MPLS (L3VPN) vs. Metro Ethernet (L2VPN) MPLS Layer 3 Service § Routing protocol dependent on the carrier § Layer 3 capability depends on carrier offering QoS (4 classes/6 classes) • IPv6 capability • § Transport IP protocol only § Highly scalable and ideal for large network MetroE Layer 2 Service § Flexibility of routing protocol and network topology independent of the carrier § Customer manages layer 3 QoS § Capable of transport IP and non-IP traffic. § Routing protocol determines scalability in point-to-multipoint topology BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 16 WAN Overlay Technologies Types of Overlay Service Layer 2 Overlays Layer 3 Overlays • Layer 2 Tunnelling Protocol—Version 3 (L2TPv3) • IPSec—Encapsulating Security Payload (ESP) – Layer 2 payloads (Ethernet, Serial,…) – Pseudowire capable • Other L2 overlay technologies – OTV, VxLAN – Strong encryption – IP Unicast only • Generic Routing Encapsulation (GRE) – IP Unicast, Multicast, Broadcast – Multiprotocol support • Other L3 overlay technologies – MPLSomGRE, LISP, OTP BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 18 Tunnelling GRE and IPSec Transport and Tunnel Modes IP HDR IP Payload GRE packet with new IP header: Protocol 47 (forwarded using new IP dst) IP HDR GRE 20 bytes 4 bytes IP HDR IP Payload 2 bytes IPSec Transport mode IP HDR 20 bytes ESP HDR 30 bytes IP Payload Encrypted Authenticated 20 bytes ESP HDR 54 bytes ESP Trailer Auth 2 bytes IPSec Tunnel mode IP HDR ESP IP HDR IP Payload Encrypted Authenticated BRKRST-2041 ESP ESP Trailer Auth © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 19 Locator/Identifier Separation Protocol (LISP) Dynamic Tunnelling Analogous to a DNS but for Network Infrastructure § DNS resolves IP addresses for URLs [ who is www.cisco.com] ? host DNS Server DNS URL Resolution [153.16.5.29, 2610:D0:110C:1::3 ] § LISP resolves locators for queried identities [ where is 2610:D0:110C:1::3] ? LISP router [ location is 128.107.81.169 ] LISP Mapping System LISP Identity-to-location Map Resolution This Topic Is Covered in Detail in BRKRST-3045 BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 20 Locator / ID Separation Protocol (LISP) LISP “Mapping System” is analogous to a DNS lookup ‒ DNS resolves IP addresses for queried URL Answers the “WHO IS” question [ Who is www.cisco.com ] ? DNS Server Host DNS Name -to- IP URL Resolution [ Address is 153.16.5.29, 2610:D0:110C:1::3 ] ‒ LISP resolves locators for queried identities Answers the “WHERE IS” question [ Where is 2610:D0:110C:1::3 ] ? LISP Router LISP Map System [ Locator is 128.107.81.169, 128.107.81.170 ] LISP ID -to- Locator Map Resolution This Topic Is Covered in Detail in BRKRST-3045 BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 21 LISP §Overview Terminologies EID (Endpoint Identifier) is the IP address of a host – just as it is today § RLOC (Routing Locator) is the IP address of the LISP router for the host § EID-to-RLOC mapping is the distributed architecture that maps EIDs to RLOCs ITR – Ingress Tunnel Router • Receives packets from site-facing interfaces • Encap to remote LISP sites, or native-fwd to non-LISP sites Provider A 10.0.0.0/8 xTR-1 ETR – Egress Tunnel Router • Receives packets from core-facing interfaces • De-cap, deliver packets to local EIDs at site Provider X 12.0.0.0/8 ETR ITR ITR packet flow S xTR-1 ETR packet flow ETR ETR ITR xTR-2 ITR Provider B 11.0.0.0/8 Provider Y 13.0.0.0/8 LISP Site 1 D xTR-2 LISP Site 2 BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 22 LISP Operation Example LISP Data Plane - Unicast Packet Forwarding Map-Cache Entry 3 EID-prefix: 2001:db8:2::/48 Locator-set: 12.0.0.2, priority: 1, weight: 50 (D1) 13.0.0.2, priority: 1, weight: 50 (D2) PI EID-prefix 2001:db8:1::/48 11.0.0.2 -> 12.0.0.2 LISP Site 1 LISP Site 2 7 xTR-1 5 10.0.0.2 PI EID-prefix 2001:db8:2::/48 2001:db8:1::1 -> 2001:db8:2::1 Provider X 12.0.0.0/8 Provider A 10.0.0.0/8 2001:db8:1::1 -> 2001:db8:2::1 ETR ITR 6 2001:db8:1::1 -> 2001:db8:2::1 xTR-1 2 This policy controlled by the destination site ETR 4 12.0.0.2 11.0.0.2 -> 12.0.0.2 ITR 2001:db8:1::1 -> 2001:db8:2::1 S ETR ITR xTR-2 1 13.0.0.2 11.0.0.2 DNS entry: D.abc.com AAAA Provider B 11.0.0.0/8 Provider Y 13.0.0.0/8 ETR ITR D xTR-2 2001:db8:2::1 BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 23 LISP Use Cases Efficient Multi-Homing IPv6 Transition v6 PxTR v4 v6 IPv4 Core v6 service xTR v6 § § Internet IPv6 Internet IPv4 Internet LISP Site IPv6-over-IPv4, IPv6-over-IPv6 IPv4-over-IPv6, IPv4-over-IPv4 § § Legacy Site LISP Site IP Portability Ingress Traffic Engineering Without BGP Data Centre/ VM Mobility Virtualisation/Multi-tenancy Legacy Site LISP routers Legacy Site Data Centre 1 Data Centre 2 Internet PxTR IP Network LISP routers Mapping DB LISP routers VM move West DC § East DC Large Scale Segmentation § VM VM a.b.c.1 a.b.c.1 Cloud / Layer 3 VM Move BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 24 Cisco Site to Site VPN Technologies Comparison Features Infrastructure Network Network Style Failover Redundancy Scalability IP Multicast QoS Policy Control Technology DMVPN § § § § Public or Private Transport Overlay Routing IPv4/IPv6 dual Stack Large Scale Hub and Spoke with dynamic Any-to-Any FlexVPN GET VPN § Public or Private Transport § Overlay Routing § Converged Site to Site and Remote Access § Private IP Transport § Flat/Non-Overlay IP Routing § Any-to-Any; (Site-to-Site) § Active/Active based on Dynamic Routing § Dynamic Routing or IKEv2 Route Distribution § Server Clustering § Transport Routing § COOP Based on GDOI § Unlimited § 3000+ Client/Srv § Unlimited § 3000+ Client/Srv § 8000 GM total § 4000 GM/KS § Multicast replication at hub § Multicast replication at hub § Multicast replication in IP WAN network § Per Tunnel QoS, Hub to Spoke § Per SA QoS, Hub to Spoke § Per SA QoS, Spoke to Spoke § Transport QoS § Locally Managed § Centralised Policy Management § Locally Managed § Tunneled VPN § Multi-Point GRE Tunnel § IKEv1 & IKEv2 § Tunneled VPN § Point to Point Tunnels § IKEv2 Only § Tunnel-less VPN § Group Protection § IKEv1 BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 25 Dynamic Multipoint VPN (DMVPN) • Branch spoke sites establish an IPsec tunnel to and register with the hub site • IP routing exchanges prefix information for each site • BGP or EIGRP are typically used for scalability • With WAN interface IP address as the tunnel source address, provider network does not need to route customer internal IP prefixes • Data traffic flows over the DMVPN tunnels • When traffic flows between spoke sites, the hub assists the spokes to establish a site-to-site tunnel • SECURE ON-DEMAND TUNNELS ASR 1000 Hub Branch n IPsec VPN ISR ISR Branch 1 ISR Branch 2 Traditional Static Tunnels Per-tunnel QOS is applied to prevent hub site oversubscription to spoke sites DMVPN On-Demand Tunnels Static Known IP Addresses Dynamic Unknown IP Addresses BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 26 DMVPN – How it Works • • • • • • • • • Spokes build a dynamic permanent GRE/IPsec tunnel to the hub, but not to other spokes. They register as clients of the NHRP server (hub) and register their NBMA address Active-Active redundancy model—two or more hubs per spoke All configured hubs are active and are routing neighbours with spokes Routing protocol routes are used to determine traffic forwarding A spoke will initially send a packet to a destination (private) subnet behind another spoke via the hub, and the hub will send it an NHRP redirect. The redirect triggers the spoke to send an NHRP query for the data packet destination address behind the destination spoke The destination spoke initiates a dynamic GRE/IPsec tunnel to the source spoke (it now knows its NBMA address) and sends the NHRP reply. The dynamic spoke-to-spoke tunnel is built over the mGRE interface When traffic ceases then the spoke-to-spoke tunnel is removed Dual DMVPN Design Single mGRE tunnel on Hub, two mGRE tunnels on Spokes 192.168.0.0/24 Physical: 172.17.0.5 Tunnel0: 10.0.1.1 Physical: 172.17.0.1 Tunnel0: 10.0.0.1 Physical: (dynamic) Tunnel0: 10.0.0.12 Tunnel1: 10.0.1.12 .1 192.168.3.0/24 Physical: (dynamic) Tunnel0: 10.0.0.11 Tunnel1: 10.0.1.11 .1 .1 192.168.1.0 /24 BRKRST-2041 192.168.2.0 /24 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 27 DMVPN Network Designs Spoke-to-hub tunnels Spoke-to-spoke tunnels 2547oDMVPN tunnels Server Load Balancing VRF-lite Spoke-to-spoke Hierarchical Increase in Scale Hub and spoke 2547oDMVPN BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 28 Any-to-Any Encryption Before and After GETVPN Public/Private WAN Private WAN Before: IPSec P2P Tunnels After: Tunnel-Less VPN WAN Multicast § Scalability—an issue (N^2 problem) § Overlay routing § Any-to-any instant connectivity can’t § § be done to scale Limited QoS Inefficient Multicast replication § Scalable architecture for any-to-any § § § § connectivity and encryption No overlays—native routing Any-to-any instant connectivity Enhanced QoS Efficient Multicast replication BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 29 Group Security Functions Key Server § Validate Group Members § Manage Security Policy § Create Group Keys § Distribute Policy/Keys Key Server Routing Member § Forwarding § Replication § Routing Group Member Routing Members Group Member Group Member § Encryption Devices § Route Between Secure/ Unsecure Regions § Multicast Participation Group Member Group Member BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 30 Group Security Elements Key Servers Group Policy KS Cooperative Protocol Key Encryption Key (KEK) Traffic Encryption Key (TEK) Group Member Routing Members Group Member Group Member RFC3547àRFC6407: Group Domain of Interpretation (GDOI) Group Member BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 31 GETVPN - Group Key Technology Operation Example GM3 GM4 GM2 § Step 1: Group Members (GM) “register” via GDOI (IKE) with the Key Server (KS) • • GM5 GM1 GM6 GM9 KS authenticates and authorises the GM KS returns a set of IPsec SAs for the GM to use KS GM8 GM3 GM7 GM4 GM2 § Step 2: Data Plane Encryption • • § GM exchange encrypted traffic using the group keys The traffic uses IPSec Tunnel Mode with “address preservation” GM5 GM1 GM6 GM9 GM3 Step 3: Periodic Rekey of Keys • KS pushes out replacement IPsec keys before current IPsec keys expire; This is called a “rekey” KS GM8 GM7 GM4 GM2 GM5 GM1 GM6 GM9 BRKRST-2041 GM8 KS GM7 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 32 GETVPN Virtualisation Deployment Model GETVPN Segmented WAN CE PE PE CE MPLS VPN LISP with GETVPN CE PE PE CE GET Encrypted LISP LISP over GETVPN BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 33 WAN Optimisation The WAN Is the Barrier to Branch Application Performance § Applications are designed to work well on LAN’s • • • § High bandwidth Low latency Reliability WANs have opposite characteristics • • • Round Trip Time ~ 0ms Low bandwidth High latency Packet loss Client LAN Switch Server Round Trip Time ~ Many milliseconds Client LAN Switch WAN LAN Switch Server WAN Packet Loss and Latency = Slow Application Performance = Keep and manage servers in branch offices ($$$) BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 35 TCP Behaviour Return to maximum throughput could take a very long time! Window Size Packet loss Slow start Packet loss Packet loss Congestion avoidance Packet loss TCP Time (RTT) RFC1323 - TCP Extensions for High Performance BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 36 WAAS—TCP Performance Improvement § Transport Flow Optimisation (TFO) overcomes TCP and WAN bottlenecks § Shields nodes connections from WAN conditions – Clients experience fast acknowledgement – Minimise perceived packet loss – Eliminate need to use inefficient congestion handling WAN LAN TCP Behaviour Window Scaling Large Initial Windows Congestion Mgmt Improved Retransmit BRKRST-2041 LAN TCP Behaviour © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 37 WAAS Advanced Compression DRE and LZ Manage Bandwidth Utilisation Data Redundancy Elimination (DRE) Persistent LZ Compression LZ •Application-agnostic compression •Up to 100:1 compression •WAAS 4.4: Context Aware DRE Benefits •Session-based compression • Application-agnostic compression •Up compression • Uptoto10:1 100:1 compression •Works cold DREDRE cache • WAASeven 4.4: during Context Aware WAN LZ DRE DRE Synchronised Compression History BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 38 Comparing TCP and Transport Flow Optimisation Cisco TFO Provides Significant Throughput Improvements over Standard TCP Implementations Window Size TFO TCP Slow start Congestion avoidance Time (RTT) TFO is using RFC2018, RFC1323, RFC3390 and BIC-TCP http://netsrv.csc.ncsu.edu/export/bitcp.pdf BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 39 Cisco WAAS: WAN Optimisation Deployment Virtual Private Cloud WAAS Express Branch Office vWAAS WAE Server VMs Nexus 1000v vPATH VMware ESXi Server Branch Office WAAS Service Module/ UCSe Nexus 1000v VSM WAN UCS /x86 Server FC SAN Branch Office WAAS Appliance Regional Office WAAS Appliance Regional Office WAAS-XE on 4451 Internet Data Centre or Private Cloud WAAS Appliances AppNav + WAAS VMware ESXi BRKRST-2041 vWAAS Appliances © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Server VMs 40 WAN Quality of Service Quality of Service Operations How Does It Work and Essential Elements Classification and Marking § The first element to a QoS policy is to classify/identify the traffic that is to be treated differently. Following classification, marking tools can set an attribute of a frame or packet to a specific value. Policing: • § Post-Queuing Operations Classification and Marking: • § Queuing and Dropping Determine whether packets are conforming to administratively-defined traffic rates and take action accordingly. Such action could include marking, remarking or dropping a packet. Scheduling (including Queuing and Dropping): • Scheduling tools determine how a frame/packet exits a device. Queuing algorithms are activated only when a device is experiencing congestion and are deactivated when the congestion clears. 42 BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Enabling QoS in the WAN Traffic Profiles and SLA Requirements Voice § § § § § Smooth Benign Drop sensitive Delay sensitive UDP priority Bandwidth per Call Depends on Codec, Sampling-Rate, and Layer 2 Media Latency ≤ 150 ms § Jitter ≤ 30 ms § Loss ≤ 1% § Bandwidth (30-128Kbps) One-Way Requirements § Telepresence SD Video Conf § § § § § Bursty Greedy Drop sensitive Delay sensitive UDP priority § § § § § Data Bursty Drop sensitive Delay sensitive Jitter sensitive UDP priority § § § § § SD/VC has the Same Requirements as VoIP, but Has Radically Different Traffic Patterns (BW Varies Greatly) HD/VC has Tighter Requirements than VoIP in terms of jitter, and BW varies based on the resolutions § Latency ≤ 150 ms § Jitter ≤ 30 ms § Loss ≤ 0.05% § Bandwidth (1Mbps) One-Way Requirements § Latency ≤ 200 ms § Jitter ≤ 20 ms § Loss ≤ 0.10% § Bandwidth (5.5-16Mbps) One-Way Requirements BRKRST-2041 Smooth/bursty Benign/greedy Drop insensitive Delay insensitive TCP retransmits Traffic patterns for Data Vary Among Applications § § § § § Data Classes: Mission-Critical Apps Transactional/Interactive Apps Bulk Data Apps Best Effort Apps (Default) © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 43 policy-map WAN class VOICE priority percent 10 class VIDEO priority percent 23 class CRITICAL-DATA bandwidth percent 15 random-detect dscp-based class DATA bandwidth percent 19 random-detect dscp-based class SCAVENGER bandwidth percent 5 class NETWORK-CRITICAL bandwidth percent 3 service-policy MARK-BGP class class-default Packets bandwidth percent 25 random-detect Out Scheduling Tools LLQ/CBWFQ Subsystems IOS Interface Buffers 1 Mbps VoIP Policer 5 Mbps LLQ RT-Interactive Policer Packets In CBWFQ Scheduler Tx-Ring CBWFQ BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 44 Traffic Shaping Line Rate Without Traffic Shaping With Traffic Shaping Shaped Rate Traffic Shaping Limits the Transmit Rate to a Value Lower Than Line Rate § Policers typically drop traffic § Shapers typically delay excess traffic, smoothing bursts and preventing unnecessary drops § Very common with Ethernet WAN, as well as NonBroadcast Multiple-Access (NBMA) network topologies such as Frame-Relay and ATM BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 45 Hierarchical QoS For Subrate Service H-QoS Policy on WAN Interface, Shaper = CIR Two Levels MQC Policy-map PARENT class class-default shape average 150000000 service-policy output CHILD Interface gigabitethernet 0/1 service-policy output PARENT Policy-map CHILD class VOICE priority percent 10 class VIDEO priority percent 23 class CRITICAL-DATA bandwidth percent 15 random-detect dscp-based class DATA bandwidth percent 19 random-detect dscp-based class SCAVENGER bandwidth percent 5 class NETWORK-CRITICAL bandwidth percent 3 service-policy MARK-BGP class class-default bandwidth percent 25 random-detect Gig 0/1 Service Level Best Effort Scavenger 150 Mbps Video Critical Data Voice BRKRST-2041 Network Critical © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 46 MPLS VPN QoS Considerations MPLS VPN Port QoS Roles Campus VPN Block Branch 1 E F F MPLS VPN E F F E E Branch 2 CE Routers PE Routers CE Routers Enterprise Subscriber (Unmanaged CE Routers) E Outbound Policies: Inbound Policies: HQoS Shaper (if required) + LLQ for VoIP (EF) Trust DSCP + LLQ or CBWFQ for RT-Interactive (CS4) + Remark RTI (if necessary) + Restore RT-Interactive to CS4 (if necessary) + CBWFQ for Signalling (CS3) + Remark Signalling (if necessary) + Restore Signalling to CS3 (if necessary) ≤ 33% of BW Service Provider: Outbound Policies: F + LLQ for Real-Time + CBWFQ for Critical Data Inbound Policies: Trust DSCP Police on a per-Class Basis © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 47 GRE/IPSec QoS Consideration ToS Byte Preservation ToS ToS byte is copied to the new IP Header IP HDR IP Payloaad GRE HDR IP HDR ToS ToS GRE Tunnel IP HDR IP Payload IP HDR ESP HDR ToS ToS IPSec Tunnel mode IP HDR IP Payload BRKRST-2041 ESP Trailer © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public ESP Auth 48 GRE and IPsec Network QoS Design Direction of Packet Flow DSCP CS5 Packet initially marked to DSCP AF41according to RFC-4594 DSCP CS5 DSCP AF41 DSCP CS5 DSCP CS5 By default, ToS values are copied to IPsec header DSCP CS5 Topmost ToS value is rewritten on egress to match service provider classes § Remark DSCP on egress to align with each SP’s SLA class of service requirements § H-QOS with shaping to offered rate on egress § Hub per tunnel QOS to minimise spoke oversubscription Re-marks the DSCP value on the encrypted and encapsulated header on the egress interface BRKRST-2041 Packet decapsulated to reveal the original ToS byte policy-map WAN-SP-CLASS-OUTPUT class VOICE priority percent 10 class VIDEO-INTERACTIVE priority percent 23 set dscp af41 class NETWORK-MGMT bandwidth percent 5 service-policy MARK-BGP class class-default bandwidth percent 25 random-detect ! policy-map Int-Gig-Agg-HE class class-default shape average 1000000000 service-policy WAN-Out © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 49 Per Site Traffic Shaping to Avoid Overruns DMVPN Per-Tunnel QoS § User NHRP group to dynamically provision HQoS policy on a DMVPN hub per-spoke basis Data Centre/HQ Spoke: Configure NHRP group name Hub: NHRP group name mapped to QoS template policy CE Multiple spokes with same NHRP group mapped to individual instances of same QoS template policy § GRE ,IPsec &L2 header are included in calculations for shaping and bandwidth. 150 Mbps 50 Mbps § Queuing and shaping is performed at the outbound physical interface 20 Mbps 10 Mbps 10 Mbps § Can be used with DMVPN with or without IPSec. Remote Branches BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 50 Per-tunnel QoS Configurations class-map match-all typeA_voice match access-group 100 class-map match-all typeB_voice match access-group 100 class-map match-all typeA_Routing match ip precedence 6 class-map match-all typeB_Routing match ip precedence 6 policy-map typeA class typeA_voice priority 1000 class typeA_Routing bandwidth percent 20 policy-map typeB class typeB_voice priority percent 20 class typeB_Routing bandwidth percent 10 policy-map typeA_parent class class-default shape average 3000000 service-policy typeA policy-map typeB_parent class class-default shape average 2000000 service-policy typeB Hub For Your Reference interface Tunnel0 ip address 10.0.0.1 255.255.255.0 … ip nhrp map group typeA service-policy output typeA_parent ip nhrp map group typeB service-policy output typeB_parent … ip nhrp redirect no ip split-horizon eigrp 100 ip summary-address eigrp 100 192.168.0.0 255.255.192.0 5 … Hub (cont) interface Tunnel0 ip address 10.0.0.11 255.255.255.0 … ip nhrp group typeA ip nhrp map multicast 172.17.0.1 ip nhrp map 10.0.0.1 172.17.0.1 ip nhrp nhs 10.0.0.1 … Spoke1 interface Tunnel0 ip address 10.0.0.12 255.255.255.0 … ip nhrp group typeB ip nhrp map multicast 172.17.0.1 ip nhrp map 10.0.0.1 172.17.0.1 ip nhrp nhs 10.0.0.1 … Spoke2 interface Tunnel0 ip address 10.0.0.13 255.255.255.0 … ip nhrp group typeA ip nhrp map multicast 172.17.0.1 ip nhrp map 10.0.0.1 172.17.0.1 ip nhrp nhs 10.0.0.1 … Spoke3 BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 51 WAN Architecture Design and Best Practices Cisco Validate Design MPLS WAN Technology Design Guide © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 53 WAN Aggregation Reference Design Data Centre/ Campus Campus/ Data Centre WAAS Service WAN Key Servers Services/ Distribution VPN Termination WAN Edge MPLS A MPLS B Internet BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 54 Remote Branch Transport & Redundancy Options Non-Redundant MPLS Redundant-Links & Routers Redundant-Links MPLS MPLS MPLS Internet MPLS MPLS MPLS WAN MPLS + Internet WAN Internet Internet MPLS Internet Internet Internet WAN BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 55 Routing Topology at WAN Aggregation Core Layer Campus/ Data Centre WAN Distribution Layer EIGRP AS 100 Summaries+ Default DMVPN Hub Routers EIGRP AS = 100 BGP AS = 65511 iBGP EIGRP AS = 100 MPLS CE Routers BGP AS = 65511 eBGP MPLS A MPLS B EIGRP AS = 100 Internet Edge EIGRP AS = 200 Layer 2 WAN CE Router Layer 2 WAN DMVPN 1 DMVPN 2 BRKRST-2041 Internet © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 56 WAN Edge Connection Methods Compared Recommended Multi-Chassis EtherChannel VSS/3850 Stacks Si Shared Si Si Layer 3 Si P-to-P Link LAN WAN WAN WAN § § No Static Routes No First Hop Redundancy Protocols BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 57 Optimise Convergence and Redundancy Multi-chassis EtherChannel VSS/3850 Stacks Si Layer 3 Si P-to-P Link IGP recalc Channel Member Removed § Link redundancy achieved through redundant L3 paths § Provide Link Redundancy and reduce peering complexity § Flow based load-balancing through CEF forwarding across § Tune L3/L4 load-balancing hash to achieve maximum utilisation § Routing protocol reconvergence when uplink § No L3 reconvergence required when member link failed failed § Convergence time may depends on routing § No individual flow can go faster than the speed of an individual member of protocol used and the size of routing entries the link BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 58 Link Recovery Comparison ECMP vs. Multichassis EtherChannel § ECMP convergence is dependent on the number of routes Si Layer 3 Si P-to-P Link MEC convergence is consistent, independent of the number of routes 2.5 ECMP MEC Max 2 sec of lost voice § VSS/3850 Stacks 1.5 1 0.5 0 1000 3000 6000 9000 12000 Number of Routes – Sup2T Number of Routes - Sup720C BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 59 Redundancy vs. Convergence Time More Is Not Always Better § In principle, redundancy is easy § Any system with more parallel paths through the system will fail less often § Increasing parallel paths increases routing complexity, therefore increasing convergence times 2.5 Seconds § The problem is a network isn’t really a single system but a group of interacting systems 0 BRKRST-2041 Routes 10000 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 60 Best Practice — Summarise at Service Distribution § It is important to force summarisation at the distribution towards WAN Edge and towards campus & data centre § Summarisation provides topology change isolation. § Summarisation reduce routing table size. Campus/ Data Centre Summary 10.5.0.0/16 Summaries + Default 10.4.0.0/16 0.0.0.0/0.0.0.0 interface Port-channel1 description Interface to MPLS-A-CE no switchport ip address 10.4.128.1 255.255.255.252 ip pim sparse-mode ip summary-address eigrp 100 10.5.0.0 255.255.0.0 MPLS A BRKRST-2041 MPLS B © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 61 Best Practice – Preventing Routing Loops with Route Tag and Filter § Mutual route redistribution between protocols can cause routing loops without preventative measures IGP Domain (EIGRP/OSPF) § Use route-map to set tags and then redistribute based on the tags Campus § Routes are implicitly tagged when distributed from eBGP to EIGRP/OSPF with carrier AS § Use route-map to block re-learning of WAN routes via the distribution layer (already known via iBGP) router eigrp 100 distribute-list route-map BLOCK-TAGGED-ROUTES in default-metric [BW] 100 255 1 1500 redistribute bgp 65500 route-map BLOCK-TAGGED-ROUTES deny 10 match tag 65401 65402 MPLS WAN BGP Domain route-map BLOCK-TAGGED-ROUTES permit 20 BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 62 Dual Carriers with BGP as CE-PE Protocol Use iBGP for Path Selection § Run iBGP between the CE routers to exchange prefixes associated with each carrier § CE routers will use only BGP path selection information to select both the primary and secondary preferences for any destinations announced by the IGP and BGP § Use IGP (OSPF/EIGRP) for prefix readvertisement will result in equal-cost paths at remote-site bn-br200-3945-1# sh ip bgp 10.5.128.0/21 BGP routing table entry for 10.5.128.0/21, version 71 Paths: (2 available, best #2, table default, RIB-failure(17)) Not advertised to any peer 65401 65402, (aggregated by 65511 10.5.128.254) 10.4.142.26 from 10.4.142.26 (192.168.100.3) Origin IGP, localpref 100, valid, external, atomicaggregate 65402, (aggregated by 65511 10.5.128.254) 10.4.143.26 (metric 51456) from 10.5.0.10 (10.5.0.253) Origin IGP, metric 0, localpref 100, valid, internal, atomic-aggregate, best Campus 10.5.128.0/21 iBGP MPLS B MPLS A B A iBGP 10.5.128.0/21 BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 63 Best Practice - Implement AS-Path Filter Prevent Branch Site Becoming Transit Network § Dual carrier sites can unintentionally become transit network during network failure event and causing network congestion due to transit traffic § Design the network so that transit path between two carriers only occurs at sites with enough bandwidth § Implement AS-Path filter to allow only locally originated routes to be advertised on the outbound updates for branches that should not be transit router bgp 65511 neighbor 10.4.142.26 route-map NO-TRANSIT-AS out ! ip as-path access-list 10 permit ^$ ! route-map NO-TRANSIT-AS permit 10 match as-path 10 Campus MPLS B MPLS A B A iBGP BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 64 Golden Rules For Your Reference Route Preference for EIGRP & OSPF EIGRP OSPF § Internal EIGRP – Admin Dist. 90 § Admin Dist. 110 § External EIGRP – Admin Dist. 170 § Route Preference § Metric Calculation metric = bandwidth + delay 1. 2. 3. Bandwidth (in kb/s) • Delay (in microseconds) • 4. § Intra-Area Inter-Area External E1 (Internal + External Cost) External E2 (External Cost) Cost Calculation Cost= Reference BW / Interface BW Default Reference BW = 100Mbps BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 65 MPLS + Internet WAN Prefer the MPLS Path over Internet Campus EIGRP AS100 § eBGP routes are redistributed into EIGRP 100 as external routes with default Admin Distance 170 § Running same EIGRP AS for both campus and DMVPN network would result in Internet path preferred over MPLS path 10.4.128.2 eBGP Internet MPLS A EIGRP AS100 10.5.48.0/21 BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 66 MPLS + Internet WAN Use Autonomous System for IGP Path Differentiation Campus D EX EIGRP AS100 10.5.48.0/21 [170/28416] via 10.4.128.2 § eBGP routes are redistributed into EIGRP 100 as external routes with default Admin Distance 170 § Running same EIGRP AS for both campus and DMVPN network would result in Internet path preferred over MPLS path § Multiple EIGRP AS processes can be used to provide control of the routing 10.4.128.2 eBGP § Internet MPLS A EIGRP AS200 § § EIGRP 100 is used in campus location EIGRP 200 over DMVPN tunnels Routes from EIGRP 200 redistributed into EIGRP 100 appear as external route (distance = 170) Routes from both WAN sources are equal-cost paths. To prefer MPLS path over DMVPN use eigrp delay to modify path preference MPLS CE router# 10.5.48.0/21 router eigrp 100 default-metric 1000000 10 255 1 1500 BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 67 MPLS VPN BGP Path with IGP Backdoor Path § eBGP as the PE-CE Routing Protocol Campus § MPLS VPN as preferred path learned via eBGP R2 MPLS A Internet IGP Backup Link § Default configuration the failover to backup path works as expected R1 eBGP § Secondary path via backdoor IGP link (EIGRP or OSPF) over tunneled connection (DMVPN over Internet) EIGRP AS100 10.4.160.0/24 BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 68 MPLS VPN BGP Path with IGP Backdoor Path Campus § After link restore, MPLS CE router receives BGP advertisement for remote-site route. § Does BGP route get (re)installed in the route table? R1 R2 MPLS A Internet R1# show ip route B 10.4.144.0/24 [20/0] via 10.4.142.2, 01:30:06 B 10.4.145.0/24 [20/0] via 10.4.142.2, 01:30:06 D EX 10.4.160.0/24 [170/3584] via 10.4.128.9, 00:30:06 B 10.4.160.0/24 [20/0].... IGP Backup Link eBGP D EX 10.4.160.0/24 [170/3584].... EIGRP AS100 10.4.160.0/24 BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 69 For Your Reference BGP Route Selection Algorithm BGP Prefers Path with: 1. Highest Weight 2. Highest Local Preference 3. Locally originated (via network or aggregate BGP) 4. Shortest AS_PATH 5. Lowest Origin type IGP>EGP>INCOMPLETE (redistributed into BGP) 6. Lowest Multi-Exit Discriminator (MED) 7. Prefer Externals (eBGP over iBGP paths) 8. Lowest IGP metric to BGP next hop (exit point) 9. Lowest Router ID for exit point BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 70 BGP Prefers Path with Highest Weight § Routes redistributed into BGP are considered locally originated and get a default weight of 32768 § The eBGP learned prefix has default weight of 0 § Path with highest weight is selected ASR1004-1#show ip bgp 10.4.160.0 255.255.255.0 BGP routing table entry for 10.4.160.0/24, version 22 Paths: (3 available, best #3, table default) Advertised to update-groups: 4 5 65401 65401 10.4.142.2 from 10.4.142.2 (192.168.100.3) Origin IGP, localpref 200, valid, external Local 10.4.128.1 from 0.0.0.0 (10.4.142.1) Origin incomplete, metric 26883072, localpref 100, weight 32768, valid, sourced, best BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 71 Prefer the eBGP Path over IGP Set the eBGP weight > 32768 § To resolve this issue set the weights on route learned via eBGP peer higher than 32768 neighbor 10.4.142.2 weight 35000 ASR1004-1#show ip bgp 10.4.160.0 255.255.255.0 BGP routing table entry for 10.4.160.0/24, version 22 Paths: (1 available, best #1, table default) Not advertised to any peer 65401 65401 10.4.142.2 from 10.4.142.2 (192.168.100.3) Origin IGP, metric 0, localpref 100, weight 35000, valid, external, best ASR1004-1#show ip route .... B 10.4.160.0/24 [20/0] via 10.4.142.2, 05:00:06 BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 72 Secure WAN Communication Dual Carrier GETVPN Topology COOP Key Server Key Servers GM GM MPLS A MPLS B B MPLS MPLS A GM GM GM GM GM BRKRST-2041 GM © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 74 Best Practice - High Availability with Cooperative Key Servers § Two or more KSs known as COOP KSs manage a common set of keys and security policies for GETVPN group members § Group members can register to any one of the available KSs § Cooperative KSs periodically exchange and synchronise group’s database, policy and keys § Primary KS is responsible to generate and distribute group keys Cooperative KS1 Cooperative KS2 Subnet 1 Subnet 2 GM 1 GM 2 IP Network Subnet 3 Subnet 4 GM 4 GM 3 BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 75 Best Practice - Key Server Recommendations • For Your Reference Maintain reliable KS communication: Insure multiple routing paths exist between all KS • Use loopback interface for KS registration and Cooperative KS protocol Use IKE keepalive for KS-KS communication • • Use only globally applicable policies in KS proxy identifiers: Site specific policies should be applied at the GM • Goal is to create symmetric policies on KS • Exception policy development should be done on GM, not KS • • Use sufficiently long key lifetimes to minimise key transitions: Traffic Encryption Key (TEK) > 3600 sec • Key Encryption Key (KEK) > 86400 sec • • Insure rekey interval extends longer than routing convergence time BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 76 Transition from Clear-text to GETVPN SA Receive-Only Method § § § Goal •Incrementally deploy infrastructure without encryption •Immediate transition to encryption controlled by KS Method •Deploy KS with Receive-only SA’s (don’t encrypt, allow decryption) •Deploy GM throughout infrastructure and monitor rekey processes •Transition KS to Normal SA (encrypt, decrypt) Assessment •Pro: Simple transition to network-wide encryption •Con: Correct policies imperative •Con: Deferred encryption until all CE are capable of GM functions permit ip 10.1.4.0 0.0.1.255 10.1.4.0 0.0.1.255 10.1.4.0/24 KS 10.1.6.0/24 GM GET GM GM GM 10.1.5.0/24 10.1.7.0/24 permit ip 10.1.4.0 0.0.3.255 10.1.4.0 0.0.3.255 10.1.4.0/24 KS 10.1.6.0/24 GM GET GM GM GM 10.1.5.0/24 BRKRST-2041 10.1.7.0/24 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 77 Group Member For Your Reference Secured Group Member Interface interface Serial0/0 ip address 192.168.1.14 255.255.255.252 crypto map svn access-group pack-filter out <- WAN ENCRYPTION <- ALLOW IPsec and Control Packet filter (after encryption) ip access-list extended pack-filter permit esp any any permit ip host 192.168.1.14 host 192.168.1.13 permit tcp host 192.168.1.14 eq ssh any <- ALLOW IPsec <- ALLOW ROUTE ADJACENCY <- ALLOW SECURE SHELL Crypto Map Association to Group Security crypto map svn 10 gdoi<- GROUP CRYPTO MAP ENTRY set group secure-wan <- GROUP MEMBERSHIP match address control_plane <- LOCAL POLICY (EXCLUDE) Group Member Policy Exceptions ip access-list extended control_plane <- CONTROL PLANE PROTOCOLS deny ip host 192.168.1.14 host 192.168.1.13 <- PE-CE LINK (BGP, ICMP) deny tcp host 192.168.1.14 eq ssh any <- MANAGEMENT SECURE SHELL Group Member Association crypto gdoi group secure-wan identity number 3333 server address ipv4 <ks1_address> server address ipv4 <ks2_address> <<<<- GROUP ENCRYPTION MEMBER’S GROUP IDENTITY KS ADDRESS TO REGISTER ALTERNATE KS REGISTRATION © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 78 Key Server For Your Reference crypto gdoi group secure-wan identity number 3333 <- GROUP ID server local <- KEY SERVER rekey retransmit 40 number 3 <- REKEY RETRANSMITS rekey authentication mypubkey rsa my_rsa <- KS MSG AUTHENTICATION rekey transport unicast <- Unicast Rekey saipsec 10 <- SECURITY ASSOCIATION profile GETVPN-GDOI-PROFILE <- CRYPTO ATTRIBUTES SELECTION match address ipv4ipsec-policy <- ENCRYPTION POLICY no replay <- NO ANTI-REPLAY address ipv4 <ks_address> <- KS ADDRESS Crypto Attributes crypto ipsec profile GETVPN-GDOI-PROFILE set security-association lifetime seconds 7200 set transform-set AES256/SHA <- AES256 for Encryption and SHA for Hash Encryption IPsec Proxy ID’s (mandatory) ip access-list extended ipv4ipsec-policy deny udp any eq 848 any eq 848 permit ip 10.0.0.0 0.255.255.255 10.0.0.0 0.255.255.255 permit ip 10.0.0.0 0.255.255.255 232.0.0.0 0.255.255.255 <<<<- BRKRST-2041 ENCRYPTION POLICY ALLOW GDOI UNICAST MULTICAST © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 79 DMVPN Deployment over Internet Multiple Default Routes for VPN Headend § § VPN Headend has a default route to ASA firewall’s VPN-DMZ interface to reach Internet Remote site policy requires centralised Internet access default INSIDE default Internet Edge Block default § Enable EIGRP between VPN headend & Campus core to propagate default to remote § Static default (admin dist=0) remains active, § VPN-DMZ is wrong firewall interface for user traffic § Adjust admin distance so EIGRP route installed (to core) § VPN tunnel drops VPN-DMZ OUTSIDE default Internet Internet Internet default BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 80 DMVPN Deployment over Internet § The RED-VRF contains the default route to VPN-DMZ Interface needed for Tunnel Establishment default EIGRP § Enable FVRF with DMVPN to separate out the two default routes default INSIDE Internet Edge Block default § § § 2nd A default route exist on the Global Routing Table used by the user data traffic to reach Internet VPN-DMZ OUTSIDE To prevent split tunnelling the default route is advertised to spokes via Tunnel default default Internet Internet Spoke’s tunnel drops due to 2nd default route conflict with the one learned from ISP default BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 81 Best Practice – VRF-aware DMVPN Keeping the Default Routes in Separate VRFs Same Technique Enable FVRF DMVPN on the Spokes § Allow the ISP learned Default Route in the RED-VRF and used for tunnel establishment § § EIGRP § default default INSIDE Internet Edge Block default Global VRF contains Default Route learned via tunnel. User data traffic follow Tunnel to INSIDE interface on firewall VPN-DMZ OUTSIDE default default Allow for consistency for implementing corporate security policy for all users Internet Internet default BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 82 DMVPN and FVRF Configuration Example DMVPN Tunnel Interface Global Routing Table Global Branch LAN Traffic VRFs have independent routing and forwarding planes F-VRF ip vrf RED rd 65512:1 ! crypto keyring DMVPN-KEYRING vrf RED pre-shared-key address 0.0.0.0 0.0.0.0 key cisco123 ! ! crypto isakmp policy 10 encr aes 256 authentication pre-share group 2 ! crypto isakmp keepalive 30 5 ! crypto isakmp profile FVRF-ISAKMP-RED keyring DMVPN-KEYRING match identity address 0.0.0.0 RED ! Front Side Provider VRF Internet interface GigabitEthernet0/1 ip vrf forwarding RED ip address dhcp ! interface Tunnel10 ip address 10.4.132.201 255.255.254.0 …. tunnel mode gre multipoint tunnel vrf RED tunnel protection ipsec profile DMVPN-PROFILE ! router eigrp 200 network 10.4.132.0 0.0.0.255 network 10.4.163.0 0.0.0.127 eigrp router-id 10.4.132.201 BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 83 Best Practices — Enable Dead Peer Detection (DPD) Improve DMVPN Network Convergence § Dead Peer Detection (DPD) is a mechanism for detecting unreachable IKE peers § Each peer’s DPD state is independent of the others § Without DPD spoke routers will continue to encrypt traffic using old SPI which would be dropped at the hub. May take up to 60 minutes for spokes to reconverge § Use ISAKMP keepalives on spokes tun10 Internet crypto isakmp keepalives <initial> <retry> • • § ISAKMP invalid-SPI-recovery is not useful with DMVPN ISAKMP keepalive timeout should be greater than routing protocol hellos Not recommended for Hub routers – may cause an increase of CPU overhead with large number of peers Informational RFC 3706 BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 84 DMVPN Internet Deployment Dynamic IP Address Assignment on the Spokes § Spokes are receiving dynamic address assignment from the ISP § Spoke reboots and receive a new IP address from the ISP, VPN session is established but no traffic passes § Following error message appears on the spoke "%NHRP-3-PAKREPLY: Receive Registration Reply packet with error - unique address registered already(14)" § Hub router (NHS) reject registration attempts for the same private address that uses a different NBMA address § To resolve this issue, configure following command on spoke routers – Internet ip nhrp registration no-unique BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 85 Best Practices — Enable PIM NBMA-Mode Multicast over DMVPN § By default router uses OIL to correlate multicast group join to interface § This causes problem when hub is connected to multiple spokes over NBMA network § Any spoke that leaves a multicast group would case all the spokes to be pruned off the multicast group § Enable PIM NBMA mode under tunnel interface on hubs and spokes § § Multicast PIM Prune towards RP Internet ip pim nbma-mode § Allows the router to track multicast joins based on IP address instead of interface § Applies only to PIM sparse-mode Router treats NBMA network as a collection of pointto-point circuits, allowing remote sites to be pruned off traffic flows PIM Prune IGMP Leave Receiver BRKRST-2041 Receiver © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 86 Best Practices — Avoid Fragmentation with IPSec VPN GRE+IPsec MTU 1500 MTU 1400 MTU 1500 Tunnel Setting (AES256+SHA) Minimum MTU Recommended MTU GRE/IPSec (Tunnel Mode) 1414 bytes 1400 bytes GRE/IPSec (Transport Mode) 1434 bytes 1400 bytes IP fragmentation will cause CPU and memory overhead and resulting in lowering throughput performance § When one fragment of a datagram is dropped, the entire original IP datagram will have to be resent § Use ‘mode transport’ on transform-set § • § NHRP needs for NAT support and saves 20 bytes Avoid MTU issues with the following best practices • • ip mtu 1400 ip tcp adjust-mss 1360 BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 87 MACsec Layer 2 Encryption for WAN § Ethernet is not only in the campus and data centre any longer § Ethernet is growing rapidly as a WAN & Metro “transport” service § WAN/Metro SP offerings are replacing existing T1, ATM/FR, and SONET OC-x options § Ethernet services apply to many areas of the WAN/MAN: WAN links for core, edge, remote branch • PE-CE links (leveraging L3 VPN services) • Metro-E service hand-offs (P2P, P2MP) • § Cisco’s goal is to integrate MACsec as part of new Ethernet interface/LC development moving forward on routers leveraged in WAN, MAN, Branch Market Transition – Ethernet WAN Transport BRKRST-2041 ASR 1001-X XE3.14 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 88 WAN MACSec - Line Rate Encryption L2 Solution TrustSec Frame Format Authenticated Encrypted DMAC SMAC 802.1Q CMD 802.1AE Header ETYPE PAYLOAD ICV CRC 0x88e5 MISEec EtherType TCI/AN SL Packet Number SCI (optional) Simple • • Easy to configure. Simple configuration on interface level only. No GRE Tunnel establishment. Reduced interoperability issues with other L3 Features Secure • Leverage Advanced Encryption Standard Galios/Counter Mode (AES-GCM-128) Line Rate Encryption • • Leverages “line rate” Ethernet performance of the port (PHY). Speeds 1/10G, 40G, 100G Ethernet WAN deployments driving increasing need for higher crypto bandwidths. BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 89 MACsec Deployment Model Ring Topology IPv4/v6 IPv4/v6 IPv4/v6 P-t-P E-LINE P-t-MP E-LAN CE2 Routers peer per VLAN subinterface per PW Point-to-point dark fibre IPv4/v6 IPv4/v6 IPv4/v6 CE4IPv4/v6 Ethernet Sub-interface with 802.1q support CE3 IPv4/v6 Clear-Tag Feature: Leverage 802.1Q tag support for full/partial-mesh over Metro-Ethernet BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 90 WAN Extension into the Cloud Connectivity Options into AWS Cloud Corporate DC AWS Managed VPN Internet Cisco ISR/ASR VGW Private Network Corporate DC CSR 1000V AWS Direct Connect Colocation Facility CSR 1000V BRKRST-2041 Cisco ISR/ASR © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 92 AWS Direct Connect Overview § Dedicated private connection separated from Internet § Over of 40 Direct Connect locations worldwide § Each Direct Connect circuit attaches to one AWS Region § Uses 802.1q VLANS to partition into multiple virtual interfaces § Reduces network costs into/out of AWS for bandwidth-heavy workloads § Provides consistent network performance § Offers 50Mbps – 500Mbps, 1Gbps, and 10Gbps connectivity options § Uses eBGP peering for route exchange (<100 routes) BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 93 AWS Virtual Private Cloud (VPC) Overview § § § Represents logically isolated network q Separate IP range, Subnet, Routes, and Security q Overlap IP address allowed NO Support for Multicast/Broadcast traffic, VLANs, and Transit Routing Internet GW Subnet Router Network Controls Available to you: q Route Tables, aka VRFs q Internet Gateway (IGW), aka an Internet Router q Security Groups and Network ACLs q VPNaaS § Internet gateway forward traffic to outsides and in between VPCs § Subnet Router forward traffic within the VPC BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 94 CSR1000v placement in the AWS network NAT at the Internet GW § § § § § Will break services that do not work over NAT, such as GET-VPN Tunnel source will be a private address Tunnel destination from the perspective of VPN peers will be a public address Assign EC2 elastic IP address so that address does not change if the CSR1K is shutdown Other VPCs see Elastic IP address unless using VPC peering Maps to AWS Elastic IP 10.2.2.10 Gi2 10.2.1.10 Gi1 10.2.1.11 10.1.2.10 Gi2 10.1.1.10 Gi1 10.1.1.11 Internet IP 54.x.x.x • CSR should be the default gateway for the application VMs BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 95 AWS Transit Routing Limitations § You have a VPC peering connection between VPC A and VPC B § And between VPC A and VPC_C § But you cannot route packets directly from VPC B to VPC C through VPC B X C A BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 96 Inter-VPN Design using Transit VPCs with CSR1000v § End to end routings between VPCs in all regions and other none AWS network § Used for hierarchical designs • • Scaling beyond VPC peering limits Security/Monitoring Services in Transit VPC § Redundant CSRs for HA and fast convergence § Can use VRFs to resolve recursive routing issues App VPC 3rd Party VPC Shared Services VPC VGW VPC peering VPC peering Availability Zone 1 Availability Zone 2 Transit VPC Direct Connect Corporate Network/DC BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 97 Transit VPC to Interconnect AWS Regions AZ 1 AZ 2 Transit VPC US-West-2 AZ 1 AZ 2 Transit VPC US-East-2 Corporate Network/DC BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 98 CSR 1000v Advantages over… Virtual Private Gateway: VPC Peering: • Scalability • Overlapping CIDR blocks • Continuity of Operations • Peering between regions • Spoke-to-spoke routing • Transitive peering relationships • Richer routing features • Multiple peerings per VPC • Security/Application Visibility • Unicast Reverse Path Forwarding • Spoke-to-spoke routing BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 99 CSR 1000V Licensing Structure Example: Pick one option from each column… Technology Package (See next slide for details) IP Base Throughput 10 Mbps 50 Mbps 100 Mbps SEC License Type IP Base 250 Mbps 1-Year Subscription (1-year, 3-year or perpetual) 250 Mbps 500 Mbps AppX 1 Gbps 2.5 Gbps Service Provider License Agreement (SPLA) 5 Gbps AX 10 Gbps * CSR add-on license options not shown above BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 100 CSR 1000V Features Per Technology Package Technology Package IOS-XE Features § § § § § § IPBase (formerly Standard) SEC (formerly Advanced) Basic Networking: BGP, OSPF, EIGRP, RIP, ISIS, IPv6, GRE, VRF-LITE, NTP, QoS Multicast: IGMP, PIM High Availability: HSRP, VRRP, GLBP Addressing: 802.1Q VLAN, EVC, NAT, DHCP, DNS Basic Security: ACL, AAA, RADIUS, TACACS+ Management: IOS-XE CLI, SSH, Flexible NetFlow, SNMP, EEM, NETCONF IPBase Plus… § Advanced Security: Zone Based Firewall, IPSec VPN, EZVPN, DMVPN, FlexVPN, SSLVPN, GETVPN (express route/direct connect only) IPBase Plus… § Advanced Networking: L2TPv3, BFD, MPLS (tunneled), VRF, VXLAN § Application Experience: WCCPv2 (no clustering), AppXNAV, NBAR2, AVC, IP SLA § Hybrid Cloud Connectivity: LISP (L3 only), OTV, VPLS, EoMPLS § Subscriber Management: PTA, LNS, ISG AppX AX (formerly Premium) § ALL FEATURES Features in Red will not work in Amazon – infrastructure issues (lack of L2 support and Multicast support) BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 101 CSR1000v on AWS performance CSR1000v Amazon AMI Throughput Tests PV - Instance types emulate I/O devices, low to medium performance, cheaper instance type Build: 15.5(3)S2 (XE 03.16.02.S) HVM - Required to take advantage of hardware virtualisation acceleration (e.g. SR-IOV) Best performance, pricier instance types All numbers in Mbps CEF 64 CEF 1400 CEF IMIX IPSec 1400 IPSec IMIX PV m3.medium PV m3.large PV m3.xlarge PV c3.large PV c3.xlarge PV c3.2xlarge PV c3.4xlarge 35 67 76 79 73 80 70 313 696 682 534 760 1,031 1,069 108 215 297 255 281 303 306 237 454 782 514 742 1,005 1,023 97 230 250 302 313 336 346 HVM c3.large HVM c3.xlarge HVM c3.2xlarge HVM c3.4xlarge 80 101 203 362 526 758 1,032 2,067 306 382 766 1,254 515 741 1,009 2,020 348 436 874 1,318 BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 102 Summary Modern Hierarchical Global WAN Design East Theater Tier 1 West Theater Tier 2 Global IP/MPLS Core In-Theater IP/MPLS Core West Region East Region Tier 3 Internet Cloud Public Voice/Video Mobility Private IP Service Metro Service Public IP Service BRKRST-2041 Metro Service © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 104 Key Takeaways § Understand how WAN characteristics can affect your applications. • Bandwidth, latency, loss § A modular hierarchical network infrastructure is the foundation for a solid WAN architecture. Good WAN design have many well-utilised component § Encryption is a foundation component of all WAN designs and can be deployed transparently. § Understand how to build wide area network leveraging Internet transport with Intelligent WAN. § Design a network with consistent behaviour that provides predictable performance. § More is not always better. Keep it simple! BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 105 Recommended Reading BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 106 Suggested Sessions • BRKRST-2042 Highly Available WAN Design • BRKRST-2309 Introduction to WAN MACsec • BRKRST-2514 Application Optimisation and Provisioning the IWAN • BRKRST-3045 LISP A Next Generation Networking Architecture • BRKCRS-2000 Intelligent WAN 2.0 Update • BRKCRS-2040 WAN and Remote-Site Deployment using Cisco Validated Designs • BRKSEC-4054 Advanced Concepts of DMVPN BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 107 Continue Your Education • Demos in the Cisco campus • Walk-in Self-Paced Labs • Lunch & Learn • Meet the Engineer 1:1 meetings • Related sessions BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 108 Q&A Complete Your Online Session Evaluation Give us your feedback and receive a Cisco Live 2017 Cap by completing the overall event evaluation and 5 session evaluations. All evaluations can be completed via the Cisco Live Mobile App. Caps can be collected Friday 10 March at Registration. Learn online with Cisco Live! Visit us online after the conference for full access to session videos and presentations. www.CiscoLiveAPAC.com BRKRST-2041 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 110 Thank you