Matěj Grégr
RESILIENT NETWORK DESIGN
PAGE
1/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
Campus Best Practices - Resilient network design
 Campus Best Practices documents share knowledge within
several technical areas (physical infrastructure, campus
networking, wireless, security, etc.)
 Campus Best Practice Documents are available at:
http://www.terena.org/activities/campus-bp/bpd.html
 Resilient network design is described mainly in:
 Recommended Resilient Campus Network Design, March 2010 (CBPD114,
the Czech Republic)
 Recommended configuration of switches in campus networks, May 2010
(UFS105, Norway)
PAGE
2/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
Resilient network design
 Enterprise campus requires highly available and secure
network infrastructure, to support business solutions such as
voice, video, wireless, and mission-critical data applications.
 Resiliency—Ability to provide non-stop business
communication with rapid sub-second network recovery
during abnormal network failures or even network upgrades.
 The goal of resilient topology is to eliminate downtime and
convergence time during crashes and device upgrades
PAGE
3/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
Network design ①




Single broadcast domain
Single security domain
No backup
Central switch performance
PAGE
4/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
Network design ②
 Routers can separate network
 Smaller broadcast domains
 Possibility to control traffic path
 Routers are pretty expensive
 Number of ports in an ordinary router is limited
PAGE
5/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
Resilient Campus design
Servers and users access
ports
Aggregation traffic from
access layer
High speed switching/routing
PAGE
6/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
Access layer






Entry point for clients into the network
Provides Layer 2 (VLAN) connectivity between users
High port density
PoE
Security mechanism – 802.1x
QoS classification
PAGE
7/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
Access layer
 Problems:
 Access switches are single points of failure in a network
• Redundant connection for end users is very expensive
• Resiliency has to be integrated into the device
• Redundant supervisor, power outlet …
 Recommendations:
 Disable Etherchannel and trunk negotiation for end users
• Prevents VLAN hopping attacks
 Enable edge ports (PortFast) for access ports
PAGE
8/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
Server access layer redundancy
 Servers need redundant network connection
 Possible solutions:




PAGE
9/36
Link aggregation protocol (LACP, PAgP)
Ethernet card bonding
Server virtualization
Service load balancing
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
Link aggregation (Etherchannel)
 Allows combination of several physical links to one
logical channel
Physical view
Logical view
 Load balancing
 MAC, IP, IP+TCP/UDP
 Simplify configuration
 only logical port configuration is necessary
 Simplify other protocol operation
 STP sees only Etherchannel link
PAGE
10/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
Link aggregation protocol
 Two signaling protocols exist (PAgP, LACP)
 LACP – IEEE 802.3ad is recommended, PAgP is Cisco
proprietary
 Modes:
 Active/Passive: request/response channel establishment
 On/Off: static configuration
 Recommendation:
 Use static configuration (mode on): dynamic configuration
delays channel establishment
PAGE
11/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
Etherchannel configuration
 Prerequisites to established channel
 Same speed/duplex
 Same mode (trunk – same vlans enabled, access – same
access vlan)
 Same STP cost and mode (edge/non-edge)
 Cisco
Switch1(config)# interface range gi 0/1 - 2
Switch1(config-if-range)# channel-group 1 mode active
 HP
Switch1(config)# trunk g1-g2 trk1 lacp
PAGE
12/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
Ethernet card bonding
 Useful, when server does not support aggregation
protocol
 Linux - supported in kernel
 Windows – supported in Ethernet card drivers
 Several modes:
 backup
 transmit load balancing: load balance in transmit direction
 adaptive load balancing: rewrite MAC addresses, different
peers use different MAC address, no switch support is
necessary
PAGE
13/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
Server virtualization
 Virtualization brings simplification in network
resilient design
 Virtual servers are connected through Virtual Switch
 Virtual switch – redundant connection to resilient
network
 All virtual servers have resilient connection to
Internet
PAGE
14/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
Server Load Balancing
 SLB provides a virtual server IP address to which
clients can connect, representing a group of real
physical servers in a server farm
 Load balancing: according to:
 L4 - L7 information
 Software implementation
 Hardware implementation
• Cisco Application Control Engine modul needed
 Advantages:
 Reduced server load
 Higher security – real IP address is not visible
 Downtime elimination if more servers are used
PAGE
15/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
SLB modes
 Dispatched mode
 Every server in server farm has own real IP together with virtual
IP address (secondary IP or loopback IP) of whole server farm
 Traffic redirection: packet with virtual IP is put in Ethernet frame
with MAC address of real server
 All servers in SLB farm have to be in same IP subnet
 Directed mode
 Every server has only own real IP address
 Servers do not know virtual IP address of whole server farm
 NAT is used – virtual IP address of the farm is translated to real
IP address of a server
PAGE
16/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
Distribution layer
 Purpose is to provide L2 distribution through
switched network
 Topology contains loops (needed for redundancy)
 L2 loop protocol (STP) is needed
 Gateway redundancy, high availability
 Packet filtering,
PAGE
17/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
Spanning tree protocol
 Necessary for loop elimination
 802.1D
 Original version has very long convergence time > 30s
 Did not support VLAN – support added in 802.1t, extend
BID, now integrated into 802.1D
 Recommendation is to use RSTP (802.1w), RPVSTP+
or MSTP (802.1s)
PAGE
18/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
Rapid Spanning Tree (RSTP)




IEEE 802.1w
Convergence time < 1s
Backward compatibility with 802.1D
Several Cisco 802.1D improvements were integrated
into 802.1w standard (UplinkFast, BackboneFast …)
 Configuration:
 Cisco: Switch(config)# spanning-tree mode rapid-pvst
 HP: Switch(config)# spanning-tree force-version
rstp-operation
PAGE
19/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
STP load balancing
 Scenario:
 Port-priority or port cost can be used
 Example:
Left(config)# interface gi1/6
Left(config)# spanning-tree vlan 200 port priority 112
 Recommendation:
 Useful is to set higher priority on undesired port instead of
setting lower priority on desired port
PAGE
20/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
MSTP
 STP or RSTP support only one STP tree for all VLANs
 RPVSTP+ (Cisco proprietary): STP tree per every VLAN
 Main idea of MSTP:
 Administrator can configure several STP instances
 VLANS are mapped to instances
 MSTP internally use RSTP
PAGE
21/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
MSTP configuration
Switch(config)# spanning-tree mode mst
Switch(config)# spanning-tree mst configuration
Switch(config-mst)# name region1
Switch(config-mst)# revision 1
Switch(config-mst)# instance 1 vlan 100
Switch(config-mst)# instance 2 vlan 200
 Configuration needed for every switch
 Increase complexity
 Proprietary solution:
 Use VTPv3
PAGE
22/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
VTPv3




More flexible protocol – can distribute “any” database
Better authentication
VTP can be turned on/off per port
Client can not rewrite database as it was common in
previous versions
 Server/client/transparent switch for databases
 VLAN, MST, Unknown (another database in future)
 Primary/secondary server
 Primary server (only) can modify a database
• Only one server in a domain
 Secondary server: backup server, could be primary
PAGE
23/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
VTPv3 – MSTP configuration
 Configure VTPv3
Switch(config)# vtp
Switch(config)# vtp
Switch(config)# vtp
Switch(config)# end
Switch# vtp primary
version 3
domain NAME
mode server mst
mst
 MSTP configuration similar to previous slide
 MSTP config is distributed within VTPv3 domain
PAGE
24/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
Gateway redundancy
 Historic attempts
 Proxy ARP, ICMP Router Discovery Protocol, routing support in
end station
 Does not scale well, software support is needed
 Solution:
 Redundancy using virtual router
 Virtual IP, virtual MAC
 No host configuration needed
Virtual Router
 Proprietary solutions
 HSRP, GLBP
Backup in
Standby
Forwarder
 Standard solution
 VRRP
PAGE
25/36
Internet,
Backbone, etc.
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
VRRP
 Open standard
 IETF RFC 3768 – version 2
 IETF RFC 5798 – version 3 (IPv4 + IPv6)
 VRRP Group – virtual router with virtual IP address
 Virtual MAC address - 0000.5e00.01xx - last byte is
group number
 Master router
 Highest priority
 IP address same as virtual IP (IP address owner) – always win
master role
 Backup router
 Other routers in VRRP group
PAGE
26/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
VRRP configuration
 Cisco configuration
SwitchA(config)# interface vlan10
SwitchA(config-if)# ip address 10.1.10.5 255.255.255.0
! Virtual IP for vrrp group 10
SwitchA(config-if)# vrrp 10 ip 10.1.10.1
! Priority for router in group 10 (standard priority is 100)
SwitchA(config-if)# vrrp 10 priority 150
! Preempt delay
SwitchA(config-if)# vrrp 10 preempt delay minimum 380
 HP configuration
hp
hp
hp
hp
hp
(config)# vlan 223
(vlan-224)# vrrp vrid 1
(vlan-224-vrid-1)# owner
(vlan-224-vrid-1)# virtual-ip-address 10.1.10.5 255.255.255.0
(vlan-224-vrid-1)# enable
 Recommendation
 Use the first IP address from subnet for Master router
 Set preempt-delay-time to let routing protocol converge
PAGE
27/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
Gateway Load Balancing Protocol
 HSRP, VRRP may have inactive routers in a group
 Standby and Other routers: cannot be used by end station (do not have
virtual IP/MAC)
 Possible solution: several HSRP/VRRP groups with end stations distributed
among them
 Static configuration!
 GLBP goal is to utilize all routers equally
 Several members of GLBP group should participate in packets
switching/routing
 GLBP solution
 Virtual IP per group
 max. 4 virtual MAC addresses per group
PAGE
28/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
GLBP – fundamental concept
 GLBP group contains two types of members
 Active Virtual Gateway
 Active Virtual Forwarder
 Active virtual gateway (AVG)




Router with highest priority (highest IP address)
There is only one AVG per group
Assign virtual MAC addresses of other members of the GLBP group
Reply to virtual IP ARP requests
• GLBP group controller: end device requesting virtual IP address
obtains some of the assigned virtual MAC addresses
 Active virtual forwarder (AVF)




PAGE
29/36
Max. 4 AVF per group, other routers are in backups
AVF are responsible for assigned virtual MAC/IP address
AVG is also AVF
Communication is done via Hello messages every 3 sec, multicast
address 224.0.0.102 UDP 3222
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
GLBP operation
PAGE
30/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
GLBP load balancing techniques
 Weighted load-balancing algorithm
 Based on weighting parameter
 Per-host (Host-dependent load-balancing algorithm)
 End station has always the same AVF
 Round-robin load-balancing algorithm (default)
 Virtual MAC addresses are rotated per ARP request
PAGE
31/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
Core layer




High speed routing
Fast convergence is necessary
Aggregate links from distribution layer
Try to avoid any packet manipulation, (access lists
and filtering), which would slow down the switching
of packets.
 Smaller campus can combine core and distribution
layer functions
PAGE
32/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
Is the core layer necessary?
PAGE
33/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
Is the core layer necessary?
PAGE
34/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
Summary
 Resilient network design eliminates downtime and
convergence time in a network
 If is it properly deployed
 Always depends of “How much money do You have”
PAGE
35/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz
PAGE
36/36
nes.at.fit
© 2011 Brno University of Technology, Faculty of Information Technology, Matěj Grégr, igregr@fit.vutbr.cz