CUMULUS NETWORKS WHITEPAPER — VXLAN ROUTING WITH EVPN
VXLAN routing with EVPN
The VXLAN routing architectures,
operations, and standard configurations
for modern data center networks
Contents
Introduction
2
VXLAN routing overview
3
VXLAN routing architectures
4
Centralized architecture
4
Distributed architecture
5
VXLAN routing with EVPN overview
6
EVPN route types for VXLAN routing
6
EVPN integrated routing and bridging (IRB) models
9
Asymmetric IRB model
Symmetric IRB model
9
10
Cumulus EVPN VXLAN routing deployment scenarios and
configurations
12
Centralized VXLAN routing with bridging using EVPN
15
Distributed VXLAN routing with the asymmetric IRB model
23
Distributed VXLAN routing with the symmetric IRB model
27
NetQ for EVPN operations
32
Conclusion
EVPN: Introduction
Introduction
Modern day data centers are moving from layer 2 to layer 3 architectures to take advantage
of a vast array of benefits. A layer 3 infrastructure provides smaller failure domains, less
spanning tree troubleshooting, fewer proprietary protocols such as MLAG, as well as
increased redundancy and resilience over a strict layer 2 data center.
However, some distributed applications, load balancers, and storage appliances still require
layer 2 connectivity to communicate. Therefore, VXLAN was born to run as an overlay on a
robust layer 3 data center to offer the best of both worlds: maintaining layer 2 connectivity
between racks while deploying a layer 3 infrastructure. VXLAN also offers additional
benefits over layer 2 VLANs such as scale and flexibility.
To provide the layer 2 connectivity, VLANs are bridged into VXLAN tunnels, which run over
the IP infrastructure. Since these packets are bridged, they cannot reach outside the same
VLAN/VXLAN without performing some type of routing. However, many server applications
require both layer 2 connectivity to the same VLAN on a different rack and layer 3
connectivity to other VXLANs/VLANs within in the data center in a different rack (or even
outside the data center altogether). For example, in Figure 1 below, the green application
only communicates via layer 2, while the orange application needs access to the Internet
or other VLANs. Without static configuration, the front-end server cannot reach the Internet
without some type of VXLAN routing.
Internet
Layer 3 IP Fabric
VLAN 10
VLAN 10
Front-end Server
Back-end Database
Figure 1 - VXLAN routing example
In order to reach each other or the Internet, these individual tunnels must be integrated with
the larger routing fabric. Older ASICs in many data center switches, such as the Broadcom
Trident II, cannot support dual lookup routing — routing first on the outer header IP address
and then again on the decapsulated packet. Therefore, data centers with an external
2
CUMULUS NETWORKS — WHITE PAPER
EVPN: VXLAN routing overview
reachability requirement deployed external routing via a separate router/firewall or a
hyperloop cable to perform routing between the VXLAN tunnels and the Internet, usually
from a centralized location. The newer ASICs, such as the Broadcom Trident II+, Maverick,
Tomahawk (via internal loopback), and the Mellanox Spectrum can route directly on the
ASIC, which opens the door to more efficient VXLAN routing design options using BGP with
the Ethernet Virtual Private Network (EVPN) address family as the routing protocol.
This paper is meant as an addendum to the paper, BGP EVPN for VXLAN, which
describes using EVPN for VLAN/VXLAN bridging. Deploying BGP EVPN also as a layer
3 routing protocol for VXLAN dramatically simplifies the VXLAN routing deployment by
using the same protocol, BGP, for the underlay, for bridging between VLANs/VXLANs, and
for routing between VXLAN tunnels and to outside the data center. This paper provides a
VXLAN routing overview, describes some of the architectures and aspects of VXLAN routing
with EVPN, and provides some typical design examples along with configuration snippets.
VXLAN routing overview
VXLAN is an effective way to tunnel layer 2 frames over a layer 3 infrastructure, and EVPN can
be used to provide a resilient control plane for VXLAN tunnels. However, routing is also needed
for communication between VXLAN tunnels or between a VXLAN tunnel and the Internet.
In the past, customers with older generation silicon performed VXLAN routing using an
external router and/or a hyperloop. Newer generation ASICs such as the Broadcom Trident
II+, Trident III, Tomahawk1 , and Mellanox Spectrum support VXLAN routing directly on the
switch. For example, Figure 2 shows Host A on VLAN A communicating with a host on
VLAN B located on a different rack.
Routing
vni
Send to host C
on VLAN B on a
different rack
Bridge
vni
swp1
swp2
Host A VLAN A
Host B VLAN B
Figure 2 - Internal silicon-based VXLAN routing example
VXLAN routing internal to the switch allows for much more efficient architectures in a
network. You can efficiently deploy one of two different architectures to route across
VXLAN boundaries while still supporting VXLAN bridging when applicable. The two
architectures are discussed in the next section.
1 Tomahawk provides internal VXLAN routing via an internal loopback
CUMULUS NETWORKS — WHITE PAPER
3
EVPN: VXLAN routing architectures
VXLAN routing architectures
VXLAN routing is primarily deployed using two architectures: either centralized or
distributed. A centralized architecture involves using a centralized router (or pair of routers)
to route between all the local VXLAN tunnels and between a VXLAN tunnel and the outside.
A centralized architecture does not use EVPN for layer 3 routing; all the routing happens on
a centralized switch but EVPN routes are leveraged for VXLAN bridging and features such
as ARP suppression and mobility.
A distributed architecture involves distributing EVPN routes that are used for both routing
and bridging VXLAN tunnels to each top of rack leaf switch. A distributed architecture
provides routing closest to the hosts.
CENTRALIZED ARCHITECTURE
In a centralized architecture, only one or a pair of routers provide access between VXLANs.
All inter-VLAN traffic or traffic intended for outside the local data center needs to be VXLAN
tunneled to the centralized router and back to the destination. This "tromboning" of traffic
means additional east-west traffic will occur in the data center to support VXLAN routing.
For each VLAN/VXLAN pair in the local network, the centralized router must have a VNI
and a switch virtual interface (SVI) that is used as the VLAN's default gateway. The default
gateway IP and MAC are advertised via BGP EVPN extended community to all the leaf
switches. See Figure 3.
Internet
Layer 3 Network
Spine01
Leaf01
eBGP Peering
(ipv4 unicast)
Spine02
Leaf02
VTEP
Bridging only
VTEP
VTEP
Bridging only
Centralized Router
SVI B (default gtwy)
VLAN A
VLAN B
Server01
Server02
VLAN A
Server03
VLAN B
SVI A (default gtwy)
Server04
VXLAN Tunnel, VNI A
VXLAN Tunnel, VNI B
Figure 3 - Centralized routing architecture
For example, Server01 on VLAN A wants to communicate with Server04 on VLAN B. Traffic
exits Server01 and enters Leaf01 with a destination IP/MAC of the SVI on the centralized
router (the default gateway). Leaf01 bridges and encapsulates the traffic in a VXLAN tunnel
4
CUMULUS NETWORKS — WHITE PAPER
EVPN: VXLAN routing architectures
headed for the default gateway. The centralized router decapsulates the packet and routes
the packet to SVI B, which is seen as a connected route. The router then encapsulates the
packet in a new VXLAN header and sends it to Leaf02 on the destination VXLAN. Leaf02
decapsulates and bridges the packet to Server04.
DISTRIBUTED ARCHITECTURE
A distributed architecture involves configuring an SVI and thus enabling VXLAN routing
on each top of rack (ToR) leaf switch. Therefore, the VXLAN routing occurs closest to the
host, keeping traffic local which provides more efficient routing and lower latency than the
centralized architecture.
Typically, an anycast gateway is configured with a distributed architecture. Using an anycast
gateway, every leaf's SVI is configured with the same IP address per VLAN. For example,
looking at Figure 4 below, Leaf01 SVI A and Leaf02 SVI A would be configured with the
same IP address and subnet mask. Since all hosts within a VLAN are configured with the
same IP default gateway address, all hosts or VMs can be easily moved throughout the
data center without changing their configuration.
Internet
Layer 3 Network
Spine01
Leaf01
Leaf02
VTEP
Anycast gateway SVI A
VLAN A
eBGP Peering
(ipv4 unicast)
Spine02
SVI B
VLAN B
Server01
Server02
SVI A
VLAN A
Server03
VTEP
VTEP
SVI B
Exit01
VLAN B
Server04
VXLAN Tunnel, VNI A
VXLAN Tunnel, VNI B
Figure 4 - Distributed architecture
Using Figure 4 as an example, Server01 on VLAN A wants to communicate with Server04
on VLAN B. Traffic would exit Server01 with a destination MAC address of Leaf01, its default
gateway. Leaf01 would recognize its own MAC address, perform a lookup, add the correct
VXLAN header and route the packet directly towards Leaf02. The Exit01 router is only used for
traffic to/from the local data center/POD to outside the data center/POD.
CUMULUS NETWORKS — WHITE PAPER
5
EVPN: VXLAN routing with EVPN overview
VXLAN routing with EVPN overview
It is helpful to understand several aspects of EVPN that are specific to VXLAN routing.
We cover some of them in the next section to introduce the topics before moving into
deployment scenarios and configuration.
EVPN ROUTE TYPES FOR VXLAN ROUTING
BGP consists of multiple address families. Each address family exchanges a different type
of route. For example, the IPv4 address family exchanges IPv4 routes, and the IPv6 address
family exchanges IPv6 routes. EVPN also has its own BGP address family which exchanges
EVPN routes. Several different types of EVPN routes exist within the BGP EVPN address
family.
This paper covers the two route types directly used for VXLAN routing, type 2 and type 5.
Cumulus Linux also supports type 3, which is used for VTEP discovery.
Type 2 EVPN routes
A type 2 EVPN route is generally used for communication within a data center, although it could
used for "stretching" VXLAN tunnels between data centers. It consists of the following fields:
BGP EVPN Type 2 Route
Route Distinguisher
Ethernet Segment Identifier
Ethernet Tag ID
MAC Address Length
MAC Address
IP Address Length
IP Address
Label 1 (L2VNI)
Label 2 (L3VNI)
Figure 5 - BGP EVPN type 2 route
As seen in Figure 5, a type 2 route includes both an IP address and MAC address coupled
together. The MAC address in the advertisement enables all leafs to know the location of
every host and enables layer 2 reachability while eliminating data plane learning. Adding the IP
address onto the advertisement for each MAC address allows each leaf switch to route as well
as perform ARP suppression, which reduces broadcast traffic in the network.
6
CUMULUS NETWORKS — WHITE PAPER
EVPN: VXLAN routing with EVPN overview
A type 2 route is used for:
●● All VLAN/VXLAN bridging
●●
Intra-data center or intra-POD VXLAN routing
●●
Inter data center routing with stretched VXLAN tunnels (VXLAN tunnels between
data centers or PODs)
If external layer 3 connectivity is required, a separate route type, type 5, is used. Type 5 is
discussed below.
Type 5 EVPN routes
The type 5 EVPN route, IP prefix route, includes the following fields:
BGP EVPN Type 5 Route
Route Distinguisher
Ethernet Segment Identifier
Ethernet Tag ID
IP Prefix Length
IP Prefix
GW IP address
Label (L3VNI)
Figure 6 - Type 5 EVPN route
As is seen above in Figure 6, the route advertisement contains the IP address and IP prefix
length, but no MAC address. The type 5 route is used with a distributed architecture to
advertise external, inter-data center and/or inter-POD routes to all local leafs within a VRF
and can only be used with an L3VNI. More information about the L3VNI can be found in the
Symmetric IRB Model section of this paper.
Figure 7 depicts a setup where type 5 EVPN routes are used for external routing. In this
case, we are advertising the 172.16.0.0/16 route from the Internet router towards the border
leafs in the BGP IPv4 address family. It is inserted into vrf1 and sent to all the VTEPs in the
network via a type 5 EVPN route. A type 5 route may also be used for inter-POD or interdata center routing.
CUMULUS NETWORKS — WHITE PAPER
7
EVPN: VXLAN routing with EVPN overview
Layer 3 Network
AS65020
AS65020
Internet
Advertise
172.16.0.0/16
eBGP Peering
(ipv4 unicast)
L2 VNI
L3 VNI
Leaf01
VRF1
L2 VNI
L3 VNI
Leaf02
VRF1
AS65011
Host A VLAN A
Border Leaf
VRF1
AS65012
AS65041
Host A VLAN A
Rack 1
Rack 2
VXLAN Tunnel, L2 VNI (Type 2 routes)
VXLAN Tunnel, L3 VNI (Type 2/5 routes)
Figure 7 - Using Type 5 routes for external routing
cumulus@leaf01:mgmt-vrf:~$ net show bgp evpn route
BGP table version is 23, local router ID is 10.0.0.11
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-2 prefix: [2]:[ESI]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[ESI}[EthTag]:[Iplen]:[IP]:[GW}
Network
Next Hop
Metric LocPrf Weight Path
Route Distinguisher: 10.0.0.11:2
*> [2]:[0]:[0]:[48]:[44:38:39:00:00:15]
10.0.0.112
32768 i
*> [2]:[0]:[0]:[48]:[44:38:39:00:00:15]:[32]:[10.2.4.102]
10.0.0.112
32768 I
<snip>
Route Distinguisher: 10.0.0.41:2
* [5]:[0]:[0][16]:[172.16.0.0]
10.0.0.41
0 65020 65041 i
*> [5]:[0]:[0][16]:[172.16.0.0]
10.0.0.41
0 65020 65041 i
A type 5 EVPN route is used for:
●● All traffic destined to the Internet
●●
8
Inter-data center or inter-POD VXLAN routing
CUMULUS NETWORKS — WHITE PAPER
EVPN: VXLAN routing with EVPN overview
EVPN INTEGRATED ROUTING AND BRIDGING (IRB) MODELS
The IETF integrated routing and bridging in EVPN draft identifies two possible
distributed IRB models for use with distributed EVPN: asymmetric and symmetric. Cumulus
Linux supports both models.
Each model is useful in certain situations and has unique characteristics. While some
vendors support either one model or the other, a Cumulus Linux EVPN with VXLAN routing
implementation can support either model. Supporting both provides interoperability with
various other vendors as well as a choice for which model is best for your data center. This
section discusses the benefits and operation of each model used for VXLAN integrated
routing and bridging.
ASYMMETRIC IRB MODEL
The asymmetric IRB model is considered asymmetric as routed traffic between two hosts
travels on different VNIs (always the destination VNI). Traffic entering the VNI is bridged
from the VLAN and may be routed to a different VNI, whereas traffic exiting the VNI is
always bridged.
Layer 3 Network
Orange VNI
Leaf01
Green VNI
Routing on ingress
Bridging on egress
Orange VNI
Leaf02
Green VNI
Routing on ingress
Host A VLAN A
Host B VLAN B
Host D VLAN B
Host C VLAN A
Bridging on egress
VXLAN Tunnel, Orange VNI
VXLAN Tunnel, Green VNI
Figure 8 - Asymmetric IRB model
Consider the scenario in Figure 8. Host A on VLAN A wants to communicate with Host C
on VLAN A. Host A knows Host C is on its same subnet (using the destination address and
its own IP address and mask to determine) so it initiates the communication by ARPing for
Host C. Since Host C has already sent a frame in the past, Leaf02 learned its MAC address
and previously communicated it to Leaf01. Leaf01 responds to Host A's ARP request, telling
Host A Host C's MAC address. Host A then sends its packets directly to Host C, bridging
across the orange VXLAN tunnel.
CUMULUS NETWORKS — WHITE PAPER
9
EVPN: VXLAN routing with EVPN overview
Host A wants to now communicate with Host B, which is located on a different VLAN and
thus reachable via a different VNI. Since the destination is a different subnet from Host A,
Host A sends the frame to its default gateway, which is Leaf01. The Leaf01 southbound
interface towards Host A is configured with an anycast IP address, which is discussed
more deeply in the architectures section. Leaf01 recognizes that the destination MAC
address is itself and uses the routing table to route the packet to the Green VNI. Leaf01
then tunnels the frame in the Green VNI to Leaf02. Leaf02 removes the VXLAN header from
the frame, and bridges the frame to Host B.
The return traffic behaves similarly. Host B sends a frame to Leaf02, which recognizes its
own destination MAC address and routes the packet to the Orange VNI. The packet is
tunneled within the Orange VNI to Leaf01. Leaf01 removes the VXLAN header from the
frame and bridges it to Host A.
With the asymmetric model, all the required source and destination VNIs (that is, Orange and
Green) must be present on each leaf, even if that leaf doesn't have a host in that associated
VLAN in its rack. In many instances, this is needed for VM mobility anyway. As a result, all
leafs would be required to hold all routes and all MAC addresses that communicate with each
other. Deploying an asymmetric model is a simple solution as no additional VNIs need to be
configured and fewer routing hops occur to communicate between VXLANs. However, it does
not scale as well as the symmetric model covered below.
In the case of multitenancy, each set of VLANs can be placed into separate VRFs and
routed between them.
Use the asymmetric model if:
●● You plan to deploy all VNIs on all leaf switches anyway
●● You have a small number of VNIs in your data center or POD
●● Your data center can be broken into PODs, with the VLANs/subnets contained in
each POD
SYMMETRIC IRB MODEL
Routed VXLAN traffic within the symmetric IRB model travels across the same "transit" VNI
in both directions. The "transit" VNI must also have an associated VLAN to accommodate
traffic to be routed between VXLAN tunnels. We call the new transit VNI "L3VNI". All traffic
that needs to be routed is bridged onto the associated VLAN and then routed onto the
L3VNI. The ingress traffic is routed on ingress to the L3VNI, tunneled across the layer 3
infrastructure, and then routed again on egress off the L3VNI to the associated VLAN and
ultimately bridged to the destination host.
10
CUMULUS NETWORKS — WHITE PAPER
EVPN: VXLAN routing with EVPN overview
Layer 3 Network
To Leaf03
Leaf01
Leaf02
VNI 24
Routing & bridging
on ingress & egress
L3VNI
Routing & bridging
on ingress & egress
Host A VLAN 24
VNI 24
VNI 13
L3VNI
Host B VLAN 13
Host C VLAN 24
Rack 1
Rack 2
VXLAN Tunnel, Orange VNI 24 (Layer 2 communication only)
VXLAN Tunnel, Green VNI 13 (Layer 2 communication only (e.g. to Rack 3))
VXLAN Tunnel, L3VNI (Layer 3 communication only)
cumulus@leaf02:mgmt-vrf:~$ net show bgp evpn vni
Advertise Gateway Macip: Disabled
Advertise All VNI flag: Enabled
Number of L2 VNIs: 2
Number of L3 VNIs: 1
Flags: * - Kernel
VNI
Type RD
Import RT
* 24
L2
10.0.0.12:2
65012:24
* 13
L2
10.0.0.12:3
65012:13
* 104001
L3
10.2.4.12:4
65012:104001
Export RT
65012:13
5012:13
65012:104001
Tenant VRF
vrf1
vrf1
vrf1
Figure 9 - Symmetric IRB model
Using Figure 9 as an example, Host A on VLAN 24 needs to communicate with Host B on
VLAN 13. Since the destination is a different subnet from Host A, Host A sends the frame to
its default gateway, Leaf01.
Leaf01 recognizes that the destination MAC address is itself and uses the routing table to
route the packet to the egress leaf (Leaf02) over the L3VNI. The MAC address of Leaf02 is
communicated to Leaf01 via a BGP extended community. The VXLAN-encapsulated packet
has the egress leaf's MAC as the destination MAC address and the L3VNI as the VNI.
Leaf02 performs VXLAN decapsulation and recognizes that the destination MAC address
is itself and routes the packet to the destination VLAN to reach the destination host. The
return traffic is routed similarly over the same L3VNI. Routing and bridging happens on both
the ingress leaf and the egress leaf.
CUMULUS NETWORKS — WHITE PAPER
11
EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations
With the symmetric model, the leaf switches only need to host the VNIs/VLANs that are
located on their rack, as well as the L3VNI and its associated VLAN, since the ingress leaf
switch doesn't route directly to the destination VNI. The ability to host only the local VNIs
(plus one extra) provides additional scale over the asymmetric model. However, the data
plane traffic is more complex as an extra routing hop occurs and an extra VXLAN tunnel
and VLAN in your network is required.
Multitenancy requires one L3VNI per VRF, and all switches participating in that VRF must be
configured with the same L3VNI.
Use the symmetric model if:
●● Your VLANs/subnets/VNIs are widely dispersed
●● Your VLANs/subnets/VNIs are provisioned on the fly
●● You need external routing with Cumulus Linux 3.5
Cumulus EVPN VXLAN routing deployment scenarios and
configurations
VXLAN routing can be deployed in either a centralized or distributed architecture.
Centralized architecture involves bridging on the leaf switches and routing between
VXLANs on a centralized switch, usually a border switch. A distributed architecture can
be deployed in either the asymmetric or the symmetric IRB model. This section discusses
deployment scenarios for each model and includes configuration snippets for each element
of the design. More information on configuring VXLAN routing with EVPN can be found in
the Cumulus Linux User Guide.
The three solutions of this section use the Cumulus Linux Reference Architecture, as
discussed in each section. Although the examples are shown with the default, or one data
plane VRF, the same procedure as shown here could be used with multiple VRFs as well.
Although not depicted in the diagrams, all switches and hosts are connected to an outof-band management network. Management VRF is configured on the switches and
eth0 (connected to the out-of-band management network) is located in the management
VRF. Although an out-of-band management network connected to the switch via the
management VRF is always recommended, it is not required to deploy these solutions.
We set up an eBGP unnumbered underlay between the leafs, the spine and the exit leafs
in all models outlined in this paper. Using eBGP as the underlay routing protocol provides
a robust scalable infrastructure for the layer 2 overlay and sets the stage to also use BGP
EVPN as the overlay routing protocol.
We configure VXLAN tunnels as the overlay — providing layer 2 connectivity over the layer
3 infrastructure. BGP EVPN is used as the VXLAN routing and bridging control plane and
provides routing between VXLAN tunnels on the local, directly-connected leaf. All hosts are
12
CUMULUS NETWORKS — WHITE PAPER
EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations
advertised as type 2 EVPN routes in all scenarios.
For each of these examples, we deploy four servers, two in VLAN13 and two in VLAN24. We
run MLAG on the leaf switches to provide redundancy to the hosts. We deploy a border leaf
connected to an Internet router. For purposes of these examples, the following addresses are
used. Note that not all the addresses depicted in Table 1 below apply to all scenarios.
Table 1 - Addresses
ADDRESSES
SWITCH
LOOPBACK
SVIs
VNIs
VLANs
BGP AS
Internet
10.0.0.253/32
N/A
N/A
N/A
25253
Exit01
10.0.0.41/32
N/A
104001
(L3VNI)
4001 (mapped to L3VNI)
65041
Exit02
10.0.0.42/32
N/A
104001
(L3VNI)
4001 (mapped to L3VNI)
65042
Spine01
10.0.0.21/32
N/A
N/A
N/A
65020
Spine02
10.0.0.22/32
N/A
N/A
N/A
65020
10.1.3.11
10.1.3.1 (anycast gtwy)
13 (data)
13
10.2.4.11
10.2.4.1 (anycast gtwy)
24 (data)
24
N/A
104001
(L3VNI)
4001
10.1.3.12/24
10.1.3.1/24 (anycast gtwy)
13 (data)
13
10.2.4.12/24
10.2.4.1/24 (anycast gtwy)
24(data)
24
N/A
104001
(L3VNI)
4001
10.1.3.13/24
10.1.3.1/24
13 (data)
13
10.2.4.13/24
10.2.4.1/24 (anycast gtwy)
24(data)
24
N/A
104001
(L3VNI)
4001
Leaf01
Leaf02
Leaf03
10.0.0.11/32
10.0.0.12/32
10.0.0.13/32
CUMULUS NETWORKS — WHITE PAPER
65011
65012
65013
13
EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations
Leaf04
10.0.0.14/32
10.1.3.14/24
10.1.3.1/24 (anycast gtwy)
13 (data)
13
10.2.4.14/24
10.1.3.1/24 (anycast gtwy)
24(data)
24
N/A
104001
(L3VNI)
4001
65014
In all scenarios the spine switches are configured the same. They are configured with eBGP
unnumbered with the EVPN address family, peering with each leaf and any border/exit
routers. A sample configuration is show below in Figure 10 where the loopback is 10.0.0.21
and the BGP AS is 65020. Switch ports 1-4 connect to each leaf switch, respectively, and
swp29 and 30 connect to the exit switches.
router bgp 65020
bgp router-id 10.0.0.21
coalesce-time 1300
bgp bestpath as-path multipath-relax
neighbor swp1 interface remote-as external
neighbor swp2 interfac remote-as external
neighbor swp3 interface remote-as external
neighbor swp4 interface remote-as external
neighbor swp29 interface remote-as external
neighbor swp30 interface remote-as external
address-family ipv4 unicast
network 10.0.0.21/32
address-family l2vpn evpn
neighbor swp1 activate
neighbor swp2 activate
neighbor swp3 activate
neighbor swp4 activate
neighbor swp29 activate
neighbor swp30 activate
Figure 10 - Spine switch configuration snippet
CENTRALIZED VXLAN ROUTING WITH BRIDGING USING EVPN
An architecture describing a centralized routing scenario is depicted below in Figure 11.
Internet
Layer 3 Network
eBGP
Peering
Spines
Swp1-4, 29-30
Swp44
Swp51-52
Swp51-52
Leafs
VTEP
VTEP
Swp1-2
Swp1-2
VTEP
Swp1-2
Swp1-2
Exit Routers
SVI 24 (default gtwy)
SVI 13 (default gtwy)
10.1.3.101
10.2.4.102
VLAN 13
Server01
VLAN 24
Server02
10.1.3.103
VLAN 13
Server03
10.2.4.104
VLAN 24
Server04
VXLAN Tunnel, VNI 13
VXLAN Tunnel, VNI 24
Figure 11 - Centralized architecture
14
CUMULUS NETWORKS — WHITE PAPER
EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations
Each of the two racks hosts two servers and two leaf switches. The servers on a rack are
on separate VLANs from each other. One server is in VLAN 13 and the other server is in
VLAN 24. The leaf switches are configured with MLAG and each server is configured with a
bond interface. However, each member of the bond is connected to a different leaf switch
in the same rack.
Two VNIs and an anycast VTEP exist on each leaf switch, VNI 13 and VNI 24. An anycast
address is used for the VTEP to allow for VXLAN redundancy with the MLAG. Only bridging
occurs on the leaf switches in this design.
Each leaf is connected to each spine. The spines are configured for eBGP unnumbered
as the underlay routing protocol, and with BGP EVPN for the overlay. The spines are
connected to the centralized routers.
The centralized (exit) routers also have the same VNIs configured as the leafs and an
anycast VTEP. Since it hosts the SVI and thus is the default gateway for all the VNIs, the
centralized routers also advertises themselves as the default gateway by using a BGP
extended community with a type 2 EVPN route. Since we are running MLAG with VRR
between the centralized routers, the virtual IP and MAC addresses are advertised as well as
the physical addresses. The centralized routers peer to the spines for the underlay with the
IPv4 address family, and peer to the spines with the L2VPN EVPN address family. They also
peer with the Internet router with the IPv4 unicast address family.
This design can be expanded to accommodate many more servers and racks.
A sample configuration snippet for a leaf switch is shown in Figure 12. The MLAG, peerlink
and bond interface towards the servers are left off for brevity.
CUMULUS NETWORKS — WHITE PAPER
15
EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations
interface lo
address 10.0.0.11/32
clagd-vxlan-anycast-ip 10.0.0.112
interface vlan13
vlan-id 13
vlan-raw-device bridge
interface bridge
bridge-ports bond01 bond02 peerlink
vni13 vni24
bridge-pvid 1
bridge-vids 13 24
bridge-vlan-aware yes
interface vlan24
vlan-id 24
vlan-raw-device bridge
interface vni13
bridge-access 13
bridge-learning off
bridge-arp-nd-suppress on
mstpctl-bpduguard yes
mstpctl-portbpdufilter yes
mtu 9000
vxlan-id 13
vxlan-local-tunnelip 10.0.0.11
interface vni24
bridge-access 24
bridge-arp-nd-suppress on
bridge-learning off
mstpctl-bpduguard yes
mstpctl-portbpdufilter yes
mtu 9000
vxlan-id 24
vxlan-local-tunnelip 10.0.0.11
router bgp 65011
bgp router-id 10.0.0.11
bgp bestpath as-path multipath-relax
neighbor swp51 interface remote-as external
neighbor swp52 interface remote-as external
address-family ipv4 unicast
network 10.0.0.11/32
network 10.0.0.112/32
address-family l2vpn evpn
neighbor swp51 activate
neighbor swp52 activate
advertise-all-vni
# pointing to spine01
# pointing to spine02
Figure 12 - Centralized architecture leaf switch configuration snippet
The centralized switches are configured with a VTEP and all the same VNIs. An SVI is also
configured on each VLAN on the centralized switches.
BGP EVPN is configured to send out the default gateway community to all the leaf switches,
as highlighted below. The tenant VLANS are advertised via the BGP network command,
and a route-map is used to send only the tenant VLAN subnets to the Internet router.
Alternatively, redistribute connected could also be used with a route map. A route-map is
also used to send only the loopback and VTEP anycast addresses of the exit routers to the
underlay. The MLAG configurations were left off for brevity. Figure 13 shows a configuration
snippet on a centralized router.
16
CUMULUS NETWORKS — WHITE PAPER
EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations
interface lo
address 10.0.0.41/32
clagd-vxlan-anycast-ip 10.0.0.142
interface bridge
bridge-ports peerlink vni13 vni24
bridge-pvid 1
bridge-vids 13 24
bridge-vlan-aware yes
interface vni13
bridge-access 13
bridge-learning off
mstpctl-bpduguard yes
mstpctl-portbpdufilter yes
mtu 9000
vxlan-id 13
vxlan-local-tunnelip 10.0.0.41
interface vni24
bridge-access 24
bridge-arp-nd-suppress on
bridge-learning off
mstpctl-bpduguard yes
mstpctl-portbpdufilter yes
mtu 9000
vxlan-id 24
vxlan-local-tunnelip 10.0.0.41
interface vlan13
address 10.1.3.13/24
address-virtual 44:39:39:ff:00:13
10.1.3.1/24
vlan-id 13
vlan-raw-device bridge
interface vlan24
address 10.2.4.13/24
address-virtual 44:39:39:ff:00:24 10.2.4.1/24
vlan-id 24
vlan-raw-device bridge
router bgp 65041
bgp router-id 10.0.0.41
coalesce-time 1100
bgp bestpath as-path multipath-relax
neighbor swp44 interface remote-as external
neighbor swp51 interface remote-as external
neighbor swp52 interface remote-as external
address-family ipv4 unicast
Network 10.0.0.142/32
network 10.0.0.41/32
network 10.1.3.0/24
network 10.2.4.0/24
neighbor swp44 route-map TENANT _ VLANS out
neighbor swp51 route-map NO _ TENANT _ VLANS _ UNDERLAY out
neighbor swp52 route-map NO _ TENANT _ VLANS _ UNDERLAY out
address-family l2vpn evpn
neighbor swp51 activate
neighbor swp52 activate
advertise-all-vni
advertise-default-gw
ip prefix-list TENANT _ VLANS seq 5 permit 10.2.4.0/24
ip prefix-list TENANT _ VLANS seq 10 permit 10.1.3.0/24
ip prefix-list NO _ TENANT _ VLANS seq 5 permit
10.0.0.41/32
ip prefix-list NO _ TENANT _ VLANS seq 10 permit
10.0.0.142/32
route-map TENANT _ VLANS permit 10
match ip address prefix-list TENANT _ VLANS
route-map NO _ TENANT _ VLANS _ UNDERLAY permit 10
match ip address prefix-list NO _ TENANT _ VLANS
Figure 13 - Centralized switch configuration snippet
The configuration from the Internet router is very basic: we configure BGP unnumbered,
and advertise the loopback (BGP router ID) and the default route. Figure 14 shows the
configuration snippet. The management VRF was left off for brevity.
CUMULUS NETWORKS — WHITE PAPER
17
EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations
interface lo
address 10.0.0.253/32
interface eth0
address 192.168.0.253/24
interface swp1
ipv6 nd ra-interval 10
no ipv6 nd suppress-ra
router bgp 25253
bgp router-id 10.0.0.253
coalesce-time 1000
bgp bestpath as-path multipath-relax
neighbor swp1 interface remote-as external
neighbor swp2 interface remote-as external
address-family ipv4 unicast
network 10.0.0.253/32
neighbor swp1 default-originate
neighbor swp2 default-originate
line vty
Figure 14 - Internet router configuration
Moving back to the leaf, we can examine the EVPN route table. We see the addresses
to the virtual default gateways — 10.1.3.1 and 10.2.4.1— are installed as type 2 routes as
expected; each has two ECMP routes, one through each spine switch. The addresses to
the physical interfaces are displayed as well. Since the centralized architecture bridges at
the ToRs, EVPN sends only type 2 and 3 routes to the leafs. The EVPN routes in this case
are used for bridging only and are shown in Figure 15. Some routes to the other hosts were
left off for brevity.
18
CUMULUS NETWORKS — WHITE PAPER
EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations
cumulus@leaf01:mgmt-vrf:~$ net show bgp evpn route
BGP table version is 7, local router ID is 10.0.0.11
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-2 prefix: [2]:[ESI]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[ESI]:[EthTag]:[IPlen]:[IP]
<snip>
Route Distinguisher: 10.0.0.41:2
* [2]:[0]:[0]:[48]:[44:39:39:ff:00:13]:[32]:[10.1.3.1]
10.0.0.142
0 65020 65041 i
*> [2]:[0]:[0]:[48]:[44:39:39:ff:00:13]:[32]:[10.1.3.1]
10.0.0.142
0 65020 65041 i
* [2]:[0]:[0]:[48]:[44:39:39:ff:00:13]:[128]:[fe80::4639:39ff:feff:13]
10.0.0.142
0 65020 65041 i
*> [2]:[0]:[0]:[48]:[44:39:39:ff:00:13]:[128]:[fe80::4639:39ff:feff:13]
10.0.0.142
0 65020 65041 i
* [2]:[0]:[0]:[48]:[6a:eb:f2:b0:f3:f6]:[32]:[10.1.3.13]
10.0.0.142
0 65020 65041 i
*> [2]:[0]:[0]:[48]:[6a:eb:f2:b0:f3:f6]:[32]:[10.1.3.13]
10.0.0.142
0 65020 65041 i
* [2]:[0]:[0]:[48]:[6a:eb:f2:b0:f3:f6]:[128]:[fe80::68eb:f2ff:feb0:f3f6]
10.0.0.142
0 65020 65041 i
*> [2]:[0]:[0]:[48]:[6a:eb:f2:b0:f3:f6]:[128]:[fe80::68eb:f2ff:feb0:f3f6]
10.0.0.142
0 65020 65041 i
* [3]:[0]:[32]:[10.0.0.142]
10.0.0.142
0 65020 65041 i
*> [3]:[0]:[32]:[10.0.0.142]
10.0.0.142
0 65020 65041 i
Route Distinguisher: 10.0.0.41:3
* [2]:[0]:[0]:[48]:[44:39:39:ff:00:24]:[32]:[10.2.4.1]
10.0.0.142
0 65020 65041 i
*> [2]:[0]:[0]:[48]:[44:39:39:ff:00:24]:[32]:[10.2.4.1]
10.0.0.142
0 65020 65041 i
* [2]:[0]:[0]:[48]:[44:39:39:ff:00:24]:[128]:[fe80::4639:39ff:feff:24]
10.0.0.142
0 65020 65041 i
*> [2]:[0]:[0]:[48]:[44:39:39:ff:00:24]:[128]:[fe80::4639:39ff:feff:24]
10.0.0.142
0 65020 65041 i
* [2]:[0]:[0]:[48]:[6a:eb:f2:b0:f3:f6]:[32]:[10.2.4.13]
10.0.0.142
0 65020 65041 i
*> [2]:[0]:[0]:[48]:[6a:eb:f2:b0:f3:f6]:[32]:[10.2.4.13]
10.0.0.142
0 65020 65041 i
* [2]:[0]:[0]:[48]:[6a:eb:f2:b0:f3:f6]:[128]:[fe80::68eb:f2ff:feb0:f3f6]
10.0.0.142
0 65020 65041 i
*> [2]:[0]:[0]:[48]:[6a:eb:f2:b0:f3:f6]:[128]:[fe80::68eb:f2ff:feb0:f3f6]
10.0.0.142
0 65020 65041 i
* [3]:[0]:[32]:[10.0.0.142]
10.0.0.142
0 65020 65041 i
*> [3]:[0]:[32]:[10.0.0.142]
10.0.0.142
0 65020 65041 i
Route Distinguisher: 10.0.0.42:3
* [2]:[0]:[0]:[48]:[3a:ba:74:bb:6f:17]:[32]:[10.1.3.12]
10.0.0.142
0 65020 65042 i
*> [2]:[0]:[0]:[48]:[3a:ba:74:bb:6f:17]:[32]:[10.1.3.12]
10.0.0.142
0 65020 65042 i
* [2]:[0]:[0]:[48]:[3a:ba:74:bb:6f:17]:[128]:[fe80::38ba:74ff:febb:6f17]
10.0.0.142
0 65020 65042 i
*> [2]:[0]:[0]:[48]:[3a:ba:74:bb:6f:17]:[128]:[fe80::38ba:74ff:febb:6f17]
10.0.0.142
0 65020 65042 i
* [2]:[0]:[0]:[48]:[44:39:39:ff:00:13]:[32]:[10.1.3.1]
10.0.0.142
0 65020 65042 i
*> [2]:[0]:[0]:[48]:[44:39:39:ff:00:13]:[32]:[10.1.3.1]
10.0.0.142
0 65020 65042 i
Figure 15 - Leaf EVPN routing table with centralized architecture
CUMULUS NETWORKS — WHITE PAPER
19
EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations
*
[2]:[0]:[0]:[48]:[44:39:39:ff:00:13]:[128]:[fe80::4639:39ff:feff:13]
10.0.0.142
0 65020
*> [2]:[0]:[0]:[48]:[44:39:39:ff:00:13]:[128]:[fe80::4639:39ff:feff:13]
10.0.0.142
0 65020
* [3]:[0]:[32]:[10.0.0.142]
10.0.0.142
0 65020
*> [3]:[0]:[32]:[10.0.0.142]
10.0.0.142
0 65020
Route Distinguisher: 10.0.0.42:4
* [2]:[0]:[0]:[48]:[3a:ba:74:bb:6f:17]:[32]:[10.2.4.12]
10.0.0.142
0 65020
*> [2]:[0]:[0]:[48]:[3a:ba:74:bb:6f:17]:[32]:[10.2.4.12]
10.0.0.142
0 65020
* [2]:[0]:[0]:[48]:[3a:ba:74:bb:6f:17]:[128]:[fe80::38ba:74ff:febb:6f17]
10.0.0.142
0 65020
*> [2]:[0]:[0]:[48]:[3a:ba:74:bb:6f:17]:[128]:[fe80::38ba:74ff:febb:6f17]
10.0.0.142
0 65020
* [2]:[0]:[0]:[48]:[44:39:39:ff:00:24]:[32]:[10.2.4.1]
10.0.0.142
0 65020
*> [2]:[0]:[0]:[48]:[44:39:39:ff:00:24]:[32]:[10.2.4.1]
10.0.0.142
0 65020
* [2]:[0]:[0]:[48]:[44:39:39:ff:00:24]:[128]:[fe80::4639:39ff:feff:24]
10.0.0.142
0 65020
*> [2]:[0]:[0]:[48]:[44:39:39:ff:00:24]:[128]:[fe80::4639:39ff:feff:24]
10.0.0.142
0 65020
* [3]:[0]:[32]:[10.0.0.142]
10.0.0.142
0 65020
*> [3]:[0]:[32]:[10.0.0.142]
10.0.0.142
0 65020
Displayed 52 prefixes (94 paths)
65042 i
65042 i
65042 i
65042 i
65042 i
65042 i
65042 i
65042 i
65042 i
65042 i
65042 i
65042 i
65042 i
65042 i
Figure 15 - Leaf EVPN routing table with centralized architecture (continued)
We can see the default gateway community for the route distinguisher by using the net
show bgp l2vpn evpn route rd <rd> command, as shown below in Figure 16. Some of the
output has been snipped for brevity.
20
CUMULUS NETWORKS — WHITE PAPER
EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations
cumulus@leaf01:mgmt-vrf:~$ net show bgp l2vpn evpn route rd 10.0.0.41:2
EVPN type-2 prefix: [2]:[ESI]:[EthTag]:[MAClen]:[MAC]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[ESI]:[EthTag]:[IPlen]:[IP]
<snip>
BGP routing table entry for 10.0.0.41:2:[2]:[0]:[0]:[48]:[44:39:39:ff:00:13]:[32]:[10.1.3.1]
Paths: (1 available, best #1)
Advertised to non peer-group peers:
spine02(swp52)
Route [2]:[0]:[0]:[48]:[44:39:39:ff:00:13]:[32]:[10.1.3.1] VNI 13
65020 65041
10.0.0.142 from spine02(swp52) (10.0.0.22)
Origin IGP, localpref 100, valid, external, bestpath-from-AS 65020, best
Extended Community: RT:65041:13 ET:8 Default Gateway
AddPath ID: RX 0, TX 30
Last update: Thu Mar 8 19:58:56 2018
<snip>
Figure 16 - Default gateway community
Looking at the routing table on the leaf, we see only layer 3 routes to the loopbacks of the
switches; all the routing between VXLAN subnets is happening on the centralized switch.
Figure 17 depicts the output.
CUMULUS NETWORKS — WHITE PAPER
21
EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations
cumulus@leaf01:mgmt-vrf:~$ net show route
show ip route
=============
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
> - selected route, * - FIB route
C>* 10.0.0.11/32 is directly connected, lo, 02:39:46
B>* 10.0.0.12/32 [20/0] via fe80::4638:39ff:fe00:54, swp51, 02:38:03
*
via fe80::4638:39ff:fe00:25, swp52, 02:38:03
B>* 10.0.0.13/32 [20/0] via fe80::4638:39ff:fe00:54, swp51, 02:39:43
*
via fe80::4638:39ff:fe00:25, swp52, 02:39:43
B>* 10.0.0.14/32 [20/0] via fe80::4638:39ff:fe00:54, swp51, 02:39:43
*
via fe80::4638:39ff:fe00:25, swp52, 02:39:43
B>* 10.0.0.21/32 [20/0] via fe80::4638:39ff:fe00:54, swp51, 02:39:43
B>* 10.0.0.22/32 [20/0] via fe80::4638:39ff:fe00:25, swp52, 02:39:43
B>* 10.0.0.41/32 [20/0] via fe80::4638:39ff:fe00:54, swp51, 02:39:43
*
via fe80::4638:39ff:fe00:25, swp52, 02:39:43
B>* 10.0.0.42/32 [20/0] via fe80::4638:39ff:fe00:54, swp51, 02:39:43
*
via fe80::4638:39ff:fe00:25, swp52, 02:39:43
C>* 10.0.0.112/32 is directly connected, lo, 02:39:35
B>* 10.0.0.134/32 [20/0] via fe80::4638:39ff:fe00:54, swp51, 02:39:43
*
via fe80::4638:39ff:fe00:25, swp52, 02:39:43
B>* 10.0.0.142/32 [20/0] via fe80::4638:39ff:fe00:54, swp51, 00:43:58
*
via fe80::4638:39ff:fe00:25, swp52, 00:43:58
C>* 169.254.1.0/30 is directly connected, peerlink.4094, 02:38:04
Figure 17 - Leaf routing table
The routing table on the centralized switch is as expected, shown in Figure 18, where we
see the VLAN subnets as directly connected and the default pointing to the internet router:
22
CUMULUS NETWORKS — WHITE PAPER
EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations
cumulus@exit01:mgmt-vrf:~$ net show route
show ip route
=============
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
> - selected route, * - FIB route
B>* 0.0.0.0/0 [20/0] via fe80::4638:39ff:fe00:7, swp44, 18:06:20
B>* 10.0.0.11/32 [20/0] via fe80::4638:39ff:fe00:a, swp51, 02:40:33
*
via fe80::4638:39ff:fe00:5b, swp52, 02:40:33
B>* 10.0.0.12/32 [20/0] via fe80::4638:39ff:fe00:a, swp51, 02:38:52
*
via fe80::4638:39ff:fe00:5b, swp52, 02:38:52
B>* 10.0.0.13/32 [20/0] via fe80::4638:39ff:fe00:a, swp51, 19:15:53
*
via fe80::4638:39ff:fe00:5b, swp52, 19:15:53
B>* 10.0.0.14/32 [20/0] via fe80::4638:39ff:fe00:a, swp51, 19:15:53
*
via fe80::4638:39ff:fe00:5b, swp52, 19:15:53
B>* 10.0.0.21/32 [20/0] via fe80::4638:39ff:fe00:a, swp51, 19:15:53
B>* 10.0.0.22/32 [20/0] via fe80::4638:39ff:fe00:5b, swp52, 19:15:53
C>* 10.0.0.41/32 is directly connected, lo, 19:15:56
B>* 10.0.0.42/32 [20/0] via fe80::4638:39ff:fe00:a, swp51, 05:29:47
*
via fe80::4638:39ff:fe00:5b, swp52, 05:29:47
B>* 10.0.0.112/32 [20/0] via fe80::4638:39ff:fe00:a, swp51, 02:39:11
*
via fe80::4638:39ff:fe00:5b, swp52, 02:39:11
B>* 10.0.0.134/32 [20/0] via fe80::4638:39ff:fe00:a, swp51, 19:15:53
*
via fe80::4638:39ff:fe00:5b, swp52, 19:15:53
C>* 10.0.0.142/32 is directly connected, lo, 18:17:39
B>* 10.0.0.253/32 [20/0] via fe80::4638:39ff:fe00:7, swp44, 18:06:20
C * 10.1.3.0/24 is directly connected, vlan13-v0, 18:17:43
C>* 10.1.3.0/24 is directly connected, vlan13, 18:17:43
C * 10.2.4.0/24 is directly connected, vlan24-v0, 05:27:17
C>* 10.2.4.0/24 is directly connected, vlan24, 18:17:43
C>* 169.254.1.0/30 is directly connected, peerlink.4094, 18:17:47
<snip>
Figure 18 - Centralized routing table
This setup is available virtually on Cumulus Networks GitHub site and all the full
configurations are depicted.
DISTRIBUTED VXLAN ROUTING WITH THE ASYMMETRIC IRB MODEL
An architecture outlining an asymmetric model is depicted below in Figure 19. This design
can be also expanded to accommodate many more racks.
CUMULUS NETWORKS — WHITE PAPER
23
EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations
Layer 3 Network
Leaf01
Leaf02
VRF1
VTEP
VRF1
SVIs
Anycast gateway SVI A
10.1.3.101
VRF1
VTEP
Server02
VRF1
SVIs
10.1.3.101
VLAN 13
Server01
Leaf04
SVIs
10.2.4.102
VLAN 24
Leaf03
VLAN 24
Server03
SVIs
10.2.4.102
VLAN 13
Server04
VXLAN Tunnel, VNI 24
VXLAN Tunnel, VNI 13
Figure 19 - Architecture using asymmetric IRB model
Two servers are located in VLAN 24 and two servers are located in VLAN 13. The servers
are dual connected via MLAG to the leaf switches. Asymmetric VXLAN routing with EVPN is
deployed to provide inter-VLAN connectivity.
No special additional leaf (ToR) configuration is needed for VXLAN routing in the
asymmetric scenario. It is enabled by default when the associated VLAN is configured with
an SVI. However, to provide routing to a destination VLAN, the local switch must have a
local VNI and VLAN configured for that destination VLAN, even if no hosts on that same
destination VLAN exist on the rack.
A sample leaf switch configuration (leaf01) is shown in Figure 20. Each leaf is configured
similarly. The MLAG/bond configuration and the peerlink were left off for brevity. Note the
SVI is configured on each VLAN, along with Virtual Router Redundancy (VRR). For example,
a server on VLAN 13 would use 10.1.3.1 as its default gateway.
24
CUMULUS NETWORKS — WHITE PAPER
EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations
interface lo
address 10.0.0.11/32
alias loopback interface
clagd-vxlan-anycast-ip 10.0.0.112
interface bridge
bridge-ports bond01 bond02 peerlink vni13
vni24
bridge-pvid 1
bridge-vids 13 24
bridge-vlan-aware yes
interface vni13
bridge-access 13
bridge-arp-nd-suppress on
bridge-learning off
mstpctl-bpduguard yes
mstpctl-portbpdufilter yes
mtu 9000
vxlan-id 13
vxlan-local-tunnelip 10.0.0.11
interface vni24
bridge-access 24
bridge-arp-nd-suppress on
bridge-learning off
mstpctl-bpduguard yes
mstpctl-portbpdufilter yes
mtu 9000
vxlan-id 24
vxlan-local-tunnelip 10.0.0.11
interface vlan13
address 10.1.3.11/24
address-virtual 44:39:39:ff:00:13
10.1.3.1/24
vlan-id 13
vlan-raw-device bridge
interface vlan24
address 10.2.4.11/24
address-virtual 44:39:39:ff:00:24
10.2.4.1/24
vlan-id 24
vlan-raw-device bridge
router bgp 65011
bgp router-id 10.0.0.11
bgp bestpath as-path multipath-relax
neighbor swp51 interface remote-as external
neighbor swp52 interface remote-as external
address-family ipv4 unicast
network 10.0.0.11/32
network 10.0.0.112/32
address-family l2vpn evpn
neighbor swp51 activate
neighbor swp52 activate
advertise-all-vni
Figure 20 - Asymmetrical model Leaf01 switch configuration snippet
VRFs may be added for multitenancy, and routing between VXLANs happens only within a VRF.
Looking at the EVPN routing table shown in Figure 21, the four server IP addresses along
with their MAC addresses appear in the table as type 2 routes: 10.1.3.101, 10.2.4.102,
10.1.3.103 and 10.2.4.104:
CUMULUS NETWORKS — WHITE PAPER
25
EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations
cumulus@leaf01:mgmt-vrf:~$ net show bgp evpn route
BGP table version is 6, local router ID is 10.0.0.11
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-2 prefix: [2]:[ESI]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[ESI]:[EthTag]:[IPlen]:[IP]
Network
Next Hop
Metric LocPrf Weight Path
Route Distinguisher: 10.0.0.11:2
*> [2]:[0]:[0]:[48]:[44:38:39:00:08:01]
10.0.0.112
32768 i
*> [2]:[0]:[0]:[48]:[44:38:39:00:08:01]:[32]:[10.1.3.101]
10.0.0.112
32768 i
*> [2]:[0]:[0]:[48]:[44:38:39:00:08:01]:[128]:[fe80::4638:39ff:fe00:801]
10.0.0.112
32768 i
*> [2]:[0]:[0]:[48]:[46:38:39:00:08:01]
10.0.0.112
32768 i
*> [2]:[0]:[0]:[48]:[46:38:39:00:08:02]
10.0.0.112
32768 i
*> [3]:[0]:[32]:[10.0.0.112]
10.0.0.112
32768 i
Route Distinguisher: 10.0.0.11:3
*> [2]:[0]:[0]:[48]:[44:38:39:00:09:01]
10.0.0.112
32768 i
*> [2]:[0]:[0]:[48]:[44:38:39:00:09:01]:[32]:[10.2.4.102]
10.0.0.112
32768 i
*> [2]:[0]:[0]:[48]:[44:38:39:00:09:01]:[128]:[fe80::4638:39ff:fe00:901]
10.0.0.112
32768 i
*> [2]:[0]:[0]:[48]:[46:38:39:00:09:01]
10.0.0.112
32768 i
*> [2]:[0]:[0]:[48]:[46:38:39:00:09:02]
10.0.0.112
32768 i
*> [3]:[0]:[32]:[10.0.0.112]
10.0.0.112
32768 i
Route Distinguisher: 10.0.0.13:2
*> [2]:[0]:[0]:[48]:[44:38:39:00:0a:01]
10.0.0.134
0 65020
*> [2]:[0]:[0]:[48]:[44:38:39:00:0a:01]:[32]:[10.1.3.103]
10.0.0.134
0 65020
*> [2]:[0]:[0]:[48]:[44:38:39:00:0a:01]:[128]:[fe80::4638:39ff:fe00:a01]
10.0.0.134
0 65020
*> [2]:[0]:[0]:[48]:[46:38:39:00:0a:01]
10.0.0.134
0 65020
*> [2]:[0]:[0]:[48]:[46:38:39:00:0a:02]
10.0.0.134
0 65020
*> [3]:[0]:[32]:[10.0.0.134]
10.0.0.134
0 65020
Route Distinguisher: 10.0.0.13:3
*> [2]:[0]:[0]:[48]:[44:38:39:00:0b:01]
10.0.0.134
0 65020
*> [2]:[0]:[0]:[48]:[44:38:39:00:0b:01]:[32]:[10.2.4.104]
10.0.0.134
0 65020
*> [2]:[0]:[0]:[48]:[44:38:39:00:0b:01]:[128]:[fe80::4638:39ff:fe00:b01]
10.0.0.134
0 65020
*> [2]:[0]:[0]:[48]:[46:38:39:00:0b:01]
10.0.0.134
0 65020
*> [2]:[0]:[0]:[48]:[46:38:39:00:0b:02]
10.0.0.134
0 65020
*> [3]:[0]:[32]:[10.0.0.134]
10.0.0.134
0 65020
65013 i
65013 i
65013 i
65013 i
65013 i
65013 i
65013 i
65013 i
65013 i
65013 i
65013 i
65013 i
Figure 21 - EVPN routing table with asymmetric IRB model
26
CUMULUS NETWORKS — WHITE PAPER
EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations
Route Distinguisher: 10.0.0.14:2
*> [2]:[0]:[0]:[48]:[44:38:39:00:0a:01]
10.0.0.134
0 65020
*> [2]:[0]:[0]:[48]:[44:38:39:00:0a:01]:[32]:[10.1.3.103]
10.0.0.134
0 65020
*> [2]:[0]:[0]:[48]:[44:38:39:00:0a:01]:[128]:[fe80::4638:39ff:fe00:a01]
10.0.0.134
0 65020
*> [2]:[0]:[0]:[48]:[46:38:39:00:0a:01]
10.0.0.134
0 65020
*> [2]:[0]:[0]:[48]:[46:38:39:00:0a:02]
10.0.0.134
0 65020
*> [3]:[0]:[32]:[10.0.0.134]
10.0.0.134
0 65020
Route Distinguisher: 10.0.0.14:3
*> [2]:[0]:[0]:[48]:[44:38:39:00:0b:01]
10.0.0.134
0 65020
*> [2]:[0]:[0]:[48]:[44:38:39:00:0b:01]:[32]:[10.2.4.104]
10.0.0.134
0 65020
*> [2]:[0]:[0]:[48]:[44:38:39:00:0b:01]:[128]:[fe80::4638:39ff:fe00:b01]
10.0.0.134
0 65020
*> [2]:[0]:[0]:[48]:[46:38:39:00:0b:01]
10.0.0.134
0 65020
*> [2]:[0]:[0]:[48]:[46:38:39:00:0b:02]
10.0.0.134
0 65020
*> [3]:[0]:[32]:[10.0.0.134]
10.0.0.134
0 65020
Displayed 36 prefixes (36 paths)
65014 i
65014 i
65014 i
65014 i
65014 i
65014 i
65014 i
65014 i
65014 i
65014 i
65014 i
65014 i
Figure 21 - EVPN routing table with asymmetric IRB model (continued)
Looking at the kernel routing table, we can see that to get to either subnet, we access it
directly on the same leaf switch as shown in Figure 22.
cumulus@leaf01:mgmt-vrf:~$ net show route
show ip route
=============
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel,
> - selected route, * - FIB route
C>* 10.0.0.11/32 is directly connected, lo, 00:00:53
B>* 10.0.0.12/32 [20/0] via fe80::4638:39ff:fe00:701, swp52, 00:00:47
*
via fe80::4638:39ff:fe00:601, swp51, 00:00:47
B>* 10.0.0.13/32 [20/0] via fe80::4638:39ff:fe00:701, swp52, 00:00:47
*
via fe80::4638:39ff:fe00:601, swp51, 00:00:47
B>* 10.0.0.14/32 [20/0] via fe80::4638:39ff:fe00:701, swp52, 00:00:47
*
via fe80::4638:39ff:fe00:601, swp51, 00:00:47
B>* 10.0.0.21/32 [20/0] via fe80::4638:39ff:fe00:601, swp51, 00:00:47
B>* 10.0.0.22/32 [20/0] via fe80::4638:39ff:fe00:701, swp52, 00:00:47
C>* 10.0.0.112/32 is directly connected, lo, 00:00:42
B>* 10.0.0.134/32 [20/0] via fe80::4638:39ff:fe00:601, swp51, 00:00:40
*
via fe80::4638:39ff:fe00:701, swp52, 00:00:40
C * 10.1.3.0/24 is directly connected, vlan13-v0, 00:00:44
C>* 10.1.3.0/24 is directly connected, vlan13, 00:00:44
C * 10.2.4.0/24 is directly connected, vlan24-v0, 00:00:44
C>* 10.2.4.0/24 is directly connected, vlan24, 00:00:44
C>* 169.254.1.0/30 is directly connected, peerlink.4094, 00:00:49
<snip>
Figure 22 - Leaf routing table with asymmetric IRB model
CUMULUS NETWORKS — WHITE PAPER
27
EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations
This setup is available on Cumulus Network GitHub site.
DISTRIBUTED VXLAN ROUTING WITH THE SYMMETRIC IRB MODEL
An architecture using the symmetric model is depicted below in Figure 23.
Internet
Lo=172.16.1.1
Layer 3 Network
(BGP unnumbered underlay)
0.0.0.0/0
Spine01
Spine02
Leaf01
Leaf02
VRF1
Anycast gateway
VTEP
SVIs
Leaf04
VRF1
SVIs
10.1.3.101
10.2.4.102
VLAN 24
Leaf03
VRF1
VLAN 13
Server01
Server02
VTEP
VRF1
SVIs
SVIs
10.1.3.103
VLAN 24
Server03
VRF1
VRF1
VTEP
VTEP
Exit 01
Exit 02
10.2.4.104
VLAN 13
Server04
VXLAN Tunnel, VNI 24
VXLAN T
VXLAN Tunnel, VNI 104001, L3VNI
Figure 23 - VXLAN routing with a symmetric IRB model architecture
For purposes of this example, we deploy four servers and two VLANs. The servers use
MLAG to connect to the leaf switches for redundancy and are configured with an anycast
default gateway. The same virtual anycast IP address is configured on each switch's
southbound SVI, using VRR, towards the servers.
Each of the four leafs host two SVIs, one terminating with VLAN 13 and one with VLAN
24. VXLAN distributed routing occurs on these leafs. Each leaf switch also has VNIs 13
and 24 configured for the VXLAN tunnels to layer 2 transport both VLAN 13 and VLAN 24
respectively. Specifically for the symmetric routing model, each leaf and exit switch also
hosts VLAN 4001 (the transport VLAN) and VNI 104001 (the L3VNI).
All leaf switches host vrf1, and all servers are located in vrf1. Since we are using EVPN,
there is no need to configure vrf1 on the spine switches.
This design has no hosts attached to border leafs, but attaching hosts to the border leafs
as well is fully supported.
For this design, we generate a default route (0.0.0.0/0) from the Internet router and advertise
that default route to the exit leafs via the BGP IPv4 address family. A sample snippet
configuration for peering and generating a default route via eBGP unnumbered is shown in
Figure 24:
28
CUMULUS NETWORKS — WHITE PAPER
EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations
interface swp1
ipv6 nd ra-interval 10
no ipv6 nd suppress-ra
interface swp2
ipv6 nd ra-interval 10
no ipv6 nd suppress-ra
router bgp 25253
bgp router-id 10.0.0.253
bgp bestpath as-path multipath-relax
neighbor swp1 interface remote-as external
neighbor swp2 interface remote-as external
!
address-family ipv4 unicast
network 10.0.0.253/32
neighbor swp1 default-originate
neighbor swp2 default-originate
exit-address-family
Figure 24 - Internet router configuration snippet
The exit leaf's interface that peers with the Internet router (swp44) is placed in vrf1, which
also places the default route in vrf1. The Exit01 router injects vrf1's default route into
vrf1's EVPN address family as a type 5 route with the command advertise ipv4 unicast, as
highlighted below. EVPN then advertises this route via the local VLAN4001, which is also in
vrf1, across the L3VNI (VNI 104001) to all other leafs in the network that participate in vrf1.
The local leafs inject the type 5 default route into its vrf1 routing table. No other VNIs are
configured on the exit leafs since no hosts are directly connected to the exit leafs.
A sample configuration snippet from Exit01 is shown in Figure 25. Note the advertise ipv4
unicast command is only needed on routers where you want to redistribute IPv4 address
family routes into L2VPN EVPN address family routes, which is generally on the border/
exit leafs. We advertise the VXLAN subnets to the Internet router. Also, in this case we
need only the L3VNI on the exit leaf as there are no hosts attached to that leaf. If hosts
were attached to those leafs, we would also need the VNIs/SVIs configured here as well, as
shown in the leaf snippet.
CUMULUS NETWORKS — WHITE PAPER
29
EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations
interface bridge
bridge-ports vxlan4001
bridge-vlan-aware yes
router bgp 65041
bgp router-id 10.0.0.41
coalesce-time 1100
bgp bestpath as-path multipath-relax
neighbor swp51 interface remote-as external
neighbor swp52 interface remote-as external
interface vlan4001
vlan-id 4001
vlan-raw-device bridge
vrf vrf1
address-family ipv4 unicast
network 10.0.0.41/32
interface swp44
vrf vrf1
interface vxlan4001
bridge-access 4001
bridge-learning off
vxlan-id 104001
vxlan-local-tunnelip 10.0.0.41
vrf vrf1
vni 104001
interface swp44 vrf vrf1
ipv6 nd ra-interval 10
no ipv6 nd suppress-ra
address-family l2vpn evpn
neighbor swp51 activate
neighbor swp52 activate
advertise-all-vni
router bgp 65041 vrf vrf1
bgp router-id 10.0.0.41
network 10.1.3.0/24 # advertise VLAN to Internet
network 10.2.4.0/24 # advertise VLAN to Internet
coalesce-time 1050
neighbor swp44 interface remote-as external
address-family l2vpn evpn
advertise ipv4 unicast # sends Type 5 routes
Figure 25 - Exit01 configuration snippet
The Exit02 switch is configured similarly to Exit01.
The spine switches are configured for both the IPv4 unicast address family as well as
the EVPN address family, which is fairly straightforward, as shown below. A snippet from
Spine01 is shown below. Interfaces swp1-4 connect to the leaf switches, and swp29-30
connect to the exit switches. Spine02 is configured similarly.
In a distributed architecture, VXLAN routing is done on the leaf switches. A snippet of a
sample configuration from a leaf switch is in Figure 26.
30
CUMULUS NETWORKS — WHITE PAPER
EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations
interface bridge
bridge-ports bond01 bond02 peerlink
vni13 vni24 vxlan4001
bridge-pvid 1
bridge-vids 13 24
bridge-vlan-aware yes
interface vni13
bridge-access 13
bridge-learning off
mstpctl-bpduguard yes
mstpctl-portbpdufilter yes
mtu 9000
vxlan-id 13
vxlan-local-tunnelip 10.0.0.11
interface vni24
bridge-access 24
bridge-arp-nd-suppress on
bridge-learning off
mstpctl-bpduguard yes
mstpctl-portbpdufilter yes
mtu 9000
vxlan-id 24
vxlan-local-tunnelip 10.0.0.11
interface vxlan4001
bridge-access 4001
bridge-learning off
vxlan-id 104001
vxlan-local-tunnelip 10.0.0.11
interface vrf1
vrf-table auto
interface vlan13
address 10.1.3.11/24
address-virtual 44:39:39:ff:00:13
10.1.3.1/24
vlan-id 13
vlan-raw-device bridge
vrf vrf1
interface vlan13
address 10.1.3.11/24
address-virtual 44:39:39:ff:00:13 10.1.3.1/24
vlan-id 13
vlan-raw-device bridge
vrf vrf1
interface vlan24
address 10.2.4.11/24
address-virtual 44:39:39:ff:00:24 10.2.4.1/24
vlan-id 24
vlan-raw-device bridge
vrf vrf1
interface vlan4001
hwaddress 44:39:39:FF:40:94
vlan-id 4001
vlan-raw-device bridge
vrf vrf1
vrf vrf1
vni 104001
router bgp 65011
bgp router-id 10.0.0.11
bgp bestpath as-path multipath-relax
neighbor swp51 interface remote-as external
neighbor swp52 interface remote-as external
address-family ipv4 unicast
network 10.0.0.11/32
address-family l2vpn evpn
neighbor swp51 activate
neighbor swp52 activate
advertise-all-vni
Figure 26 - Leaf01 switch configuration snippet for symmetric IRB model
Although not depicted above for brevity, the leaf's southbound ports are placed in vrf1
and a tenant VLAN. Each tenant VLAN is mapped to a VNI that provides the layer 2
VXLAN tunneling. In addition, VNI 104001 (associated with interface vxlan4001) and VLAN
4001 are configured to create the L3VNI. This provides the hop between the local VLANs
and the L3VNI.
Finally, let's look at the leaf's EVPN routing table in Figure 27, showing both type 2 and type
5 routes (the type 3 routes provide VTEP discovery). We can see the type 2 routes to the
servers in the data center, as well as the type 5 routes to reach outside the data center. Part
of the routing table has been left off for brevity.
CUMULUS NETWORKS — WHITE PAPER
31
EVPN: Cumulus EVPN VXLAN routing deployment scenarios and configurations
cumulus@leaf01:mgmt-vrf:~$ net show bgp evpn route
BGP table version is 10, local router ID is 10.0.0.11
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-2 prefix: [2]:[ESI]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[ESI]:[EthTag]:[IPlen]:[IP]
Network
Next Hop
Metric LocPrf Weight Path
Route Distinguisher: 10.0.0.11:2
*> [2]:[0]:[0]:[48]:[44:38:39:00:00:17]
10.0.0.112
32768 i
*> [2]:[0]:[0]:[48]:[44:38:39:00:00:17]:[32]:[10.1.3.101]
10.0.0.112
32768 i
* [3]:[0]:[32]:[10.0.0.134]
10.0.0.134
0 65020
<snip>
Route Distinguisher: 10.0.0.14:3
* [2]:[0]:[0]:[48]:[44:38:39:00:00:32]
10.0.0.134
0 65020
*> [2]:[0]:[0]:[48]:[44:38:39:00:00:32]
10.0.0.134
0 65020
* [2]:[0]:[0]:[48]:[44:38:39:00:00:32]:[32]:[10.2.4.104]
10.0.0.134
0 65020
*> [2]:[0]:[0]:[48]:[44:38:39:00:00:32]:[32]:[10.2.4.104]
10.0.0.134
0 65020
* [2]:[0]:[0]:[48]:[44:38:39:00:00:32]:[128]:[fe80::4638:39ff:fe00:32]
10.0.0.134
0 65020
*> [2]:[0]:[0]:[48]:[44:38:39:00:00:32]:[128]:[fe80::4638:39ff:fe00:32]
10.0.0.134
0 65020
* [2]:[0]:[0]:[48]:[46:38:39:00:00:23]
10.0.0.134
0 65020
*> [2]:[0]:[0]:[48]:[46:38:39:00:00:23]
10.0.0.134
0 65020
*> [2]:[0]:[0]:[48]:[46:38:39:00:00:32]
10.0.0.134
0 65020
* [2]:[0]:[0]:[48]:[46:38:39:00:00:32]
10.0.0.134
0 65020
*> [3]:[0]:[32]:[10.0.0.134]
10.0.0.134
0 65020
* [3]:[0]:[32]:[10.0.0.134]
10.0.0.134
0 65020
Route Distinguisher: 10.0.0.41:2
* [5]:[0]:[0]:[0.0.0.0]
10.0.0.41
0 65020
*> [5]:[0]:[0]:[0.0.0.0]
10.0.0.41
0 65020
Route Distinguisher: 10.0.0.42:2
* [5]:[0]:[0]:[0]:[0.0.0.0]
10.0.0.42
0 65020
*> [5]:[0]:[0]:[0]:[0.0.0.0]
10.0.0.42
0 65020
Displayed 36 prefixes (61 paths)
65014 i
65014 i
65014 i
65014 i
65014 i
65014 i
65014 i
65014 i
65014 i
65014 i
65014 i
65014 i
65014 i
65041 25253 i
65041 25253 i
65042 25253 i
65042 25253 i
Figure 27 - EVPN routing table with symmetric IRB model
Looking in the leaf's routing table in Figure 28, we can see all the routes are available in vrf1
as well as the default route that is reachable over vlan4001, which is attached to the L3VNI:
32
CUMULUS NETWORKS — WHITE PAPER
EVPN: NetQ for EVPN operations
cumulus@leaf01:mgmt-vrf:~$ net show route vrf vrf1
show ip route vrf vrf1
=======================
Codes: K - kernel route, C
O - OSPF, I - IS-IS,
T - Table, v - VNC,
> - selected route,
VRF
B>*
*
K *
B>*
*
C *
C>*
B>*
C *
C>*
B>*
B
V
*
connected, S - static, R - RIP,
- BGP, P - PIM, E - EIGRP, N - NHRP,
- VNC-Direct, A - Babel, D - SHARP,
- FIB route
vrf1:
0.0.0.0/0 [20/0] via 10.0.0.41, vlan4001 onlink, 00:05:53 #Type 5 with L3VNI
via 10.0.0.42, vlan4001 onlink, 00:05:53
0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 00:10:03
10.0.0.253/32 [20/0] via 10.0.0.41, vlan4001 onlink, 00:05:53
via 10.0.0.42, vlan4001 onlink, 00:05:53
10.1.3.0/24 is directly connected, vlan13-v0, 00:10:03
10.1.3.0/24 is directly connected, vlan13, 00:10:03
10.1.3.103/32 [20/0] via 10.0.0.134, vlan4001 onlink, 00:04:56 #Type 2 with L3VNI
10.2.4.0/24 is directly connected, vlan24-v0, 00:10:03
10.2.4.0/24 is directly connected, vlan24, 00:10:03
10.2.4.104/32 [20/0] via 10.0.0.134, vlan4001 onlink, 00:04:16 #Type 2 with L3VNI
Figure 28 - Symmetric routing table
A virtual setup with this topology is available at Cumulus Linux GitHub site.
NetQ for EVPN operations
Cumulus NetQ greatly simplifies EVPN deployment and operations. NetQ provides
configuration checks and simplifies troubleshooting EVPN in a network from one location
anywhere in the network.
Upon initial setup, it is wise to check the configurations to be sure nothing is missing
and all VNIs are configured — NetQ can accomplish this in simply one step, as shown
in Figure 29 below.
cumulus@oob-mgmt-server:~$ netq check evpn
Total Nodes: 10, Failed Nodes: 1, Total Sessions: 8 , Failed Sessions: 1, Total VNIs: 2
Hostname Peer Name
Peer Hostname Error
Last Changed
--------- ----------------------- -----------------------------------leaf04
swp51
spine01
AFI/SAFI evpn not activated on peer 1m:35.655s
Figure 29 - NetQ `check evpn`
In this example, we are missing activating the L2VPN EVPN address family on spine01. We
can also check BGP and it will tell us the same information as shown in Figure 30.
CUMULUS NETWORKS — WHITE PAPER
33
EVPN: NetQ for EVPN operations
cumulus@oob-mgmt-server:~$ netq check bgp
Total Nodes: 10, Failed Nodes: 2, Total Sessions: 16 , Failed Sessions: 2,
Hostname
Peer Name Peer Hostname Reason
Last Changed
---------- ---------- ------------- --------------------------------------------- --------leaf04
swp51
spine01
evpn address families not activated on peer 34.140580s
spine01
swp4
leaf04
evpn address families not activated on node 33.140629s
Figure 30 - NetQ `check bgp`
After this is fixed, all will look good, as shown in Figure 31 below.
cumulus@oob-mgmt-server:~$ netq check evpn
Total Nodes: 10, Failed Nodes: 0, Total Sessions: 8, Failed Sessions: 0, Total VNIs: 2
cumulus@oob-mgmt-server:~$ netq check bgp
Total Nodes: 10, Failed Nodes: 0, Total Sessions: 16, Failed Sessions: 0
Figure 31 - Good EVPN and BGP configurations
NetQ also displays all the VNIs in the entire network in one simple command, to check and
be sure the correct VNIs are where they belong, as shown in Figure 32:
cumulus@oob-mgmt-server:~$ netq
Matching evpn records:
Hostname
VNI
VTEP IP
------------ ------------leaf01
13
10.0.0.112
leaf02
13
10.0.0.112
leaf03
13
10.0.0.134
leaf04
13
10.0.0.134
show evpn vni 13
In Kernel
--------yes
yes
yes
yes
Export RT
---------65011:13
65012:13
65013:13
65014:13
Import RT
----------65011:13
65012:13
65013:13
65014:13
Last Changed
-------------15m:57.782s
16m:0.939s
15m:56.334s
15m:57.467s
Figure 32 - Showing all VNIs on a node
We can also easily display all VNIs per node, as shown in Figure 33:
cumulus@oob-mgmt-server:~$ netq leaf01 show evpn
Matching evpn records:
Hostname
VNI
VTEP IP
In Kernel
Export RT
---------------------------------------leaf01
13
10.0.0.112
yes
65011:13
leaf01
24
10.0.0.112
yes
65011:24
Import RT
---------65011:13
65011:24
Last Changed
-----------17m:37.430s
17m:37.430s
Figure 33 - NetQ displays all VNIs on a node
This is just a few examples of how NetQ can help with operations in a EVPN environment.
There are many more features available. Visit our NetQ User's Guide for more information.
34
CUMULUS NETWORKS — WHITE PAPER
EVPN: Conclusion
Conclusion
Modern data centers deploy layer 3 with the BGP routing protocol in order to scale and
provide a robust, easy to troubleshoot infrastructure that also supports a multi-vendor
environment. However, since some applications still require layer 2 connectivity, VXLAN
tunnels, deployed over a layer 3 fabric, have become a popular way to achieve layer 2
connectivity between racks.
In order for the VXLAN tunnels to reach each other or the outside world, VXLAN routing
must be enabled. Newer merchant silicon supports this functionality directly in the ASIC,
thereby making a cost effective and simple deployment. EVPN, which is already popular
for VXLAN bridging, now integrates with VXLAN routing, unifying the control plane and
streamlining deployment.
Cumulus Linux EVPN is the ideal control plane solution for VXLAN routing. It uses the
same routing protocol preferred for data center infrastructures — BGP — for both VXLAN
bridging and routing. Additionally, it has inherent support for multitenancy.
Try it out for yourself on this ready-to-go demo using Cumulus VX with NetQ and Vagrant
or try it out in Cumulus in the Cloud.
ABOUT CUMULUS NETWORKS®
Cumulus Networks is leading the transformation of bringing web-scale networking to enterprise cloud. Its network
switch, Cumulus Linux, is the only solution that allows you to affordably build and efficiently operate your network like the
world’s largest data center operators, unlocking vertical network stacks. By allowing operators to use standard hardware
components, Cumulus Linux offers unprecedented operational speed and agility, at the industry’s most competitive cost.
Cumulus Networks has received venture funding from Andreessen Horowitz, Battery Ventures, Capital, Peter Wagner and
four of the original VMware founders.
For more information visit cumulusnetworks.com or follow @cumulusnetworks.
©2018 Cumulus Networks. All rights reserved. CUMULUS, the Cumulus Logo, CUMULUS NETWORKS, and the Rocket Turtle Logo (the “Marks”) are
trademarks and service marks of Cumulus Networks, Inc. in the U.S. and other countries. You are not permitted to use the Marks without the prior
written consent of Cumulus Networks. The registered trademark Linux® is used pursuant to a sublicense from LMI, the exclusive licensee of Linus
Torvalds, owner of the mark on a worldwide basis. All other marks are used under fair use or license from their respective owners.
04032018
CUMULUS NETWORKS — WHITE PAPER
35