Real world FabricPath Deployment at IBM
Data Centers
Santiago Freitas
CCIE#18776 (R&S / SP)
Consulting Systems Engineer
Cisco-IBM Global Team
safreita@cisco.com
Lasse Leegaard
IT Architect
AT&T
lasse@intl.att.com
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Cisco
Confidential
Confidential
1
What?
 IBM has achieved tangible benefits by migrating its infrastructure from
Catalyst 6500 to a Nexus 2000, 5000, 7000 Architecture.
 IBM has adopted FabricPath on the Nexus 5K and 7K and MPLS L3VPN on
the Nexus 7K.
 The solution was extensively tested at Cisco ECATS.
 FabricPath was a key differentiator when competing with Juniper.
 We learned a lot from this deployment.
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
2
Session Objectives
At the end of the session, you should be able to:
Articulate to your customers the Business Benefits that IBM has achieved by
migrating to a Nexus 2K/5K/7K Architecture.
Explain the reasons why they adopted FabricPath.
Understand the Tests performed to validate the solution before deployment.
Understand IBM’s future direction and how they plan to get there.
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
3
IBM Nordic Strategic Outsourcing
One of the company’s largest Integrated Market Teams (IMT) globally
 IBM SO provides outsourcing services that offer management of applications
and other IT components in either an onsite or hosted arrangement.
 Eight Data Centers located in Denmark, Sweden and Finland.
 Serve around 200 customers
‒ Some have dedicated infrastructures
‒ Over 100 served by a shared, multitenant
infrastructure
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
4
Overview of the Network Infrastructure
Overall network structure
Customers
Telcos
Internet
AS25384
Si
Dedicated gateways hold 1 telco
Shared gateways holds multiple telcos
One shared set hold direct customer
connections on 1G and internet access
MPLS gateway: 6500, 7200 or 7300
Si
MPLS route
reflectors
VLAN + IP
MPLS LDP link
Si
Si
MPLS gateways
MPLS P routers
MPLS layer
MPLS P router: 6500 or 7600
L2 trunk
VPLS PE routers
Si
Si
Si
Server access block
Si
MPLS PE routers /
Aggregation switches
Layer 2 trunks
10G or Nx1G
Si
Si
Service switches
FWSM/NAM/ACE
Si
Si
Access switch: 6500
Service switch: 6500
MPLS PE router / Aggregation switch: 6500
VPLS PE router: 7600+ES20
Limit of 20 access switches is based on
un-oversubscribed port density in the
Core routers
Up to 20 access
switches
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
5
Overview of the Network Infrastructure
 ~150 Cisco 6500/7600
 ~50 Cisco 7200/7300
 ~ 3000 VLANs and 290
virtual firewalls
 ~26000 Ethernet ports
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
6
One of the Access Blocks Reached EoL
A portion of the shared infrastructure was approaching end of life
 Cisco and AT&T performed
EoL analysis.
 Factual discussion: Vital to
demonstrate the need for a
full network refresh.
- 22 Access Switches
- 4080 access ports
 1G or 2x1G uplinks
- 2 pair of FWSM in Service
Switches
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
7
Hardware Refresh Options
Replace only of the parts
that reached End of Life
Technology Refresh using
Catalyst 6500
Technology Refresh using
Nexus 2K/5K/7K
Does not solve the High
Risks and Technology
Limitations
Does not solve some of High
Risks and Technology
Limitations. Limited evolution 10G uplinks and Resolves
single point of failure issues
Solves all of High Risks and
Technology Limitations
Hidden cost of line cards
replacement within 3 years
$2.440.430,00 (now) +
$2.460.000,00 (2014) =
$4.900.430,00
Total Rack Space – 390 RU
Total Rack Space – 154 RU
Total Power – 156 KW
Total Power – 60 KW
Total cost - $5.378.635,00
Total cost - $2.639.300,00
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
8
Business Benefits of the Nexus-based Solution
Why IBM chose to deploy Nexus and FabricPath
 Significant OPEX savings when compared with the existing infrastructure:
‒ Reduced the power consumption by 61%
‒ Reduced the rack space used by network switches by 60%
‒ Reduced the number of managed devices in the network by 38.5% (from 26 to 16)
 Easier way to scale, supports more access blocks on the same Core devices,
therefore less expensive per customer port
 Reduction in the time to onboard and configure the network for new customers
 CAPEX savings – Next Generation DC based on Cisco Nexus and FabricPath
was 46% cheaper than building similar architecture using Catalyst 6500
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
9
Juniper!!!
Yes – we did consider doing it differently
 Like-for-like (EX8200/4500/4200 + MX routing + 6500/FWSM firewall)
 Qfabric (Qfabric switching + MX routing + 6500/FWSM firewall)
 No FCoE capable hardware
 10G server density not impressive
 FCoE development is beginning to catch up
 However, Nexus has more/longer field exposure than Juniper kit in this area.
 Organizational inertia and training would have to be overcome
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
10
What IBM actually deployed?
 2x Nexus 7010
MPLS Backbone
‒ M1/F1 combination
‒ MPLS L3 VPN PE
 12x Nexus 5548UP
FabricPath
‒ Across 3 DCs
 70x Nexus 2200
‒ 3360 access ports
 2x 6500 Service chassis for FWSM modules
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
11
FabricPath Flexibility
The Network Can Evolve With No Disruption
 Need more edge ports?
 Need more bandwidth?
L3
→ Add more leaf switches
→ Add more links and spines
L3
FabricPath FabricPath
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
Why IBM adopted FabricPath?
vPC and traditional STP topologies were considered
 Better utilization of links
 Increased Agility
‒ New PODs and/or links for more capacity can be added non-disruptively
‒ Any VLAN anywhere
 Simplicity of Configuration
‒ Much simpler to implement and configure than vPC
 Very fast convergence - sub-second in most cases
 Need to route over the Fabric
‒ Layer 3 over FabricPath
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
13
FabricPath enablement
Was that really it?
install feature-set fabricpath
vpc domain 11
feature-set fabricpath
role priority 100
vlan 3865
peer-keepalive destination 10.1.20.46
source 10.1.20.45
mode fabricpath
spanning-tree mst configuration
name IBMMST02
revision 10
peer-gateway
auto-recovery
fabricpath switch-id 1000
instance 1 vlan 1-2048
instance 2 vlan 204
interface Ethernet1/5
switchport mode fabricpath
fabricpath domain default
spf-interval 50 50 50
lsp-gen-interval 50 50 50
root-priority 255 / 254 (N7K)
fabricpath switch-id 1
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
14
MPLS L3 VPN on Nexus 7000
Works together with the rest of the infrastructure
 Nexus 7010 as the MPLS L3VPN PE.
 Customer VLANs mapped into VRF/VPN in
the Aggregation Layer.
 Remote Sites are 6500, 7600, 7300 and
7200, working well with the rest of the
infrastructure.
Nexus 7010
 Advantage over Juniper, extra layer
required.
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
15
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
Migration plan
How to get from here to there (or from there to here depending on your point of view)
MPLS P
VPLS PE
MPLS PE +
Aggregation
L3
L3
FabricPath
VLANs VLANs
FW/LB service +
Access
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
17
ECATS End of Test Report
Cisco Enhanced Customer Aligned Testing Services - http://ecats
 36 Major Tests Areas
 Detailed Results
 DDTS/Bugs Found and workarounds
 Technical Notes
 Convergence Summary Table
 HW and SW utilized
 Lessons Learned
 Configuration Files
See Additional Resources Slides for link to it
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
18
Migration plan
How to get from here to there (or from there to here depending on your point of view)
MPLS P
VPLS PE
MPLS PE +
Aggregation
L3
L3
FabricPath
VLANs VLANs
FW/LB service +
Access
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
19
ECATS End of Test Report
Cisco Enhanced Customer Aligned Testing Services - http://ecats
 36 Major Tests Areas
 Detailed Results
 DDTS/Bugs Found and workarounds
 Technical Notes
 Convergence Summary Table
 HW and SW utilized
 Lessons Learned
 Configuration Files
See Additional Resources Slides for link to it
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
20
ECATS testing experience
Cisco Enhanced Customer Aligned Testing Services - http://ecats
 Vital on the success of this deployment.
 Gives us experience before having used it
 Test overlap with rollout
 Reduction of risk of introducing new technology
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
21
IXIA
4/9
IXIA
4/10
9/1
Testing Topology
9/2
Access5 6500
10.53.234.172
L2 Access Layer
9/47
 Remote Site
9/48
L2
1G
8/1
8/1
Core3 7600
10.53.234.170
L2/L3 Aggregation
MPLS PE
BGP / ISIS / MP-BGP / LDP
9/3
9/1
M1 ports
1/1
Core1 Nexus 7010
10.53.234.166
10.53.234.167
L2/L3 Aggregation
MPLS PE
BGP / ISIS /MP-BGP / LDP
9/4
MP-BGP
Peering
between
PEs
9/2
L3 - 10G
MPLS/LDP
enabled links
ISIS
9/2
3/9
Po
10
4/9
4/9
F1 ports3/31
VPC+ peer link
4/1
3/2
- ISIS / MP-BGP / LDP
‒ Access Layer Cat6500 (Layer 2)
M1 ports
1/1
2/1
3/9
3/1
- L2/L3 Aggregation
9/1
L3 - 10G
MPLS/LDP
enabled links
ISIS
2/1
3/31
‒ Agg/MPLS PEs (7600)
Core4 7600
10.53.234.171
L2/L3 Aggregation
MPLS PE
BGP / ISIS / MP-BGP / LDP
9/3
Po
10
9/4
4/2
3/1
4/1
F1 ports
3/2
Core2 Nexus 7010
10.53.234.168
10.53.234.169
L2/L3 Aggregation
MPLS PE
BGP / ISIS / MP-BGP / LDP
 ISIS and MPLS in the core
4/2
 Site Under Test
F1 ports
LACP
Gi 0/1
Gi 0/2
‒ Nexus 7010 as Agg/MPLS PE (L2/L3)
Switch 3
10.53.234.148
FaE 0/3
IXIA
4/16
L2-10Gbs FabricPath Enabled Links
1/1
Access1
Nexus
5548UP
10.53.234.162 2/13
1/2
1/3
1/4
2/14
Po
10
1/3
FEX 100 Po20
1
100
/1/10
100/1/1 100
100/1/2 /1/48
IXIA
4/1
Gi 0/1
L3 BGP / OSPF
connections
BGP
FaE 0/0/0
CE1-2851
Access3 1/1
Nexus
5548UP
10.53.234.164 2/13
Access2
Nexus
5548UP
2/14 10.53.234.163
1
100/1/46
100
/1/47
1/2
FEX 100 Po20
2
Nexus 2248
IXIA
4/11
1/1
1/4
2/13
IXIA
4/2
IXIA
4/3
Gi 0/1
2
100
/1/10
1
100
/1/47
2/13
IXIA
4/5
FaE 0/3
1
Gi 0/1
BGP
CE2-2821
IXIA
4/7
OSPF
FaE 0/1/8
100
/1/48
‒ Nexus 5548UP/Nexus 2248 as Access
Switch 2
‒ FabricPath
‒ Servers attached with vPC+
IXIA
4/8
FaE 0/2
10.53.234.147
FaE 0/3
CE4-2851
10.53.234.174 10.53.234.169
IXIA
4/14
100
/1/10
FaE 0/1 LACP
‒ vPC+ at the Core for Active/Active HSRP
2
Nexus 2248
100
/1/47
IXIA
4/6
FaE 0/0/0
CE3-2851
10.53.234.173 10.53.234.167
2/14
Access4
Nexus
5548UP
10.53.234.165
FEX 100 Po20
100/1/1 100
100/1/2 /1/48
Gi 0/1
L3 BGP / OSPF
connections
1/2
100
/1/10
100/1/46
FaE 0/2
Switch 1 10.53.234.146
OSPF
FaE 0/1/8
1/1
1/3
Po
10 1/4
2
Nexus 2248
IXIA
4/13
100
/1/48
IXIA
4/4
FaE 0/1 LACP
1/3
1/4
2/14
FEX 100 Po20
Nexus 2248
100
/1/47
1/2
‒ OSPF/BGP over FP
IXIA
4/15
Note any IP addresses shown are for management purposes only.
Title:
Description:
IBM Voyager
Author:
mraines
Proposed topology for ECATS Nexus 7K/5K 7600 testing
Updated:
13/03/2012
Filename:
Minimal-Testing-topology-FabricPathv19.vsd
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
22
Testing Topology and Scale Numbers
For Your
Reference
Hardware and Software Versions and Scale Numbers
 Access Layer
‒ Nexus 5548UP – NX-OS 5.1(3)N1(1a)
‒ FEX Nexus 2248
 Core
‒ Nexus 7010 – NX-OS 5.2(3a)
‒ 2x M1 8x 10GE (N7K-M108X2-12L)
‒ 2x F1 32x 1/10GE (N7K-F132XP-15)
 Remote Site PE
‒ 7609 with RSP-720 – IOS 15.1(1)S
 300 VLANs
 300 SVIs and 300
HSRP
 200 VRFs / MPLS
L3 VPN
 3000 MAC
addresses injected
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
 IMIX Ethernet Traffic
‒ 4Gbps within Nexus
Access Block (East
23
Convergence Times
Sub-second on
FabricPath link failures
Failover Test Result Convergence Summary
 Layer 3 Link Failure on Core towards Remote site – 64 ms / 30 ms on Recovery
 M1 Line Card Failure on Core - 950 ms (North-South) / 75 ms on Recovery
 Fabric Path Link Failures (multiple tests) – 117 ms / 241 ms on Recovery
 F1 Line Card failure on Core - 1380 ms / 319 ms on Recovery
 Core Node Failure (power off N7010) - 2584 ms / 2703 ms on Recovery
 Access Node Failure - 316.52 ms for vPC+ attached servers / 181 ms on Recovery
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
24
Dynamic Routing Protocol and FabricPath
You can run OSPF and BGP over FabricPath, you can’t over vPC
 The OSPF CE routers CE-3 and CE-4 were configured with “ip ospf priority 0”
interface configuration so they don’t participate in DR/BDR election process
 FULL OSPF neighborships are formed with both Core1 and Core2
CE3-2851-RK18#sh ip ospf neighbor
Neighbor ID
Pri
State
Dead Time
Address
Interface
10.10.101.5
1
FULL/BDR
00:00:36
10.10.101.5
GigabitEthernet0/1
10.10.101.7
1
FULL/DR
00:00:33
10.10.101.7
GigabitEthernet0/1
10.10.101.8
0
2WAY/DROTHER
00:00:30
10.10.101.8
GigabitEthernet0/1
CE3-2851-RK18#
 Traffic still forwarded even when crossing peer-link
 FabricPath doesn’t have same limitations as vPC
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
25
Technical Lessons Learned
It would be a session on its own…
 No Show Stopper DDTS
‒ One cosmetic, one catastrophic but with an easy workaround (already fixed) and one
Unreproducible.
 Several Technical Lessons Learned on the areas of:
‒ Peer-Link Failure and vPC+ attached devices
‒ MAC Learning with vPC+ domain
‒ Multidestination tree and vPC+
‒ MAC Learning on N7K with M1/F1 for L2 Traffic
Details on the hidden slides and on Additional Resources page
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
26
Further developments
Where do we see the rest of the infrastructure go?
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
27
Evolution Plan
MPLS P Routers
Layer 3 / MPLS
MPLS PE/
Agg Switches
Layer 2
IPv4
Services Switches
(FWSM/ACE/NAM)
SAN A
MPLS PE/
Agg Switches
Layer 2
IPv4
Services Switches
Up to 20
Access Switches (FWSM/ACE/NAM)
SAN B
SAN A
VPLS
PE
Layer 2
IPv4
Services Switches
Up to 20
Access Switches (FWSM/ACE/NAM)
SAN B
SAN A
VPLS
PE
Services Switches
Up to 20
Access Switches (FWSM/ACE/NAM)
SAN B
Layer 2
IPv4
Layer 2
IPv4
SAN A
Services Switches
Up to 20
Access Switches (FWSM/ACE/NAM)
SAN B
SAN A
Up to 20
Access Switches
SAN B
28
Management
Orchestration
Provisioning
Automation
Evolution Plan
MPLS P Routers
Dynamic Infrastructure
Layer 3 / MPLS
2^12 = 4096 VLANs…
2^24 = 16777216 Segment IDs
MPLS PE/
Agg Switches
Layer 2
IPv4
IPv4/IPv6
Services Switches
(FWSM/ACE/NAM)
MPLS PE/
Agg Switches
Layer
Layer22
IPv4IPv6
IPv4/IPv6
IPv4/
Services Switches
Up to 20
Access Switches (FWSM/ACE/NAM)
SAN B
SAN A
Layer 2
IPv4
Services Switches
Up to 20
Access Switches (FWSM/ACE/NAM)
Storage
1 2
FC/FCoE/NAS
3 4
SAN A
VPLS
PE
SAN A SAN B
VPLS
PE
Layer 2
IPv4
Layer 2
IPv4
Services Switches
Up to 20
Access Switches (FWSM/ACE/NAM)
Services Switches
Up to 20
Access Switches (FWSM/ACE/NAM)
Up to 20
Access Switches
Storage
1 2
FC/FCoE/NAS
3 4
A
SAN B
SAN B
SAN A
SAN B
SAN A
SAN B
29
Key Takeaways
The Key Takeaways of this presentation were:
 IBM has achieved OPEX and CAPEX savings by migrating to a Nexus
2K/5K/7K Architecture in their Data Centers.
 IBM has adopted FabricPath and is very happy with its Flexibility, Easy to
Implement and Use and Convergence Time.
 FabricPath was extensively tested and validated at Cisco ECATS.
 FabricPath and MPLS on N7K were differentiators against Juniper.
 You can reuse the lessons learned and additional resources available from this
deployment to position FabricPath to your customers.
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
30
Additional Resources
 You can find the following additional information on the
link below
‒ Customer Requirements and Business Case for Catalyst 6500
-> Nexus and FabricPath
‒ Joint Technical Plan of Record (test requirements)
‒ Detailed Test Plan
‒ Complete end of Test Report (including detailed test results
and configurations)
‒ Lessons Learned Presentation
‒ INTERNAL Case Study of IBM Nordic Adoption of Nexus and
FabricPath
‒ EXTERNAL version of the Case Study
http://bock-bock.cisco.com/wiki/User:Safreita:FabricPath_Testing
Real world FabricPath deployment at IBM Data © 2012 Cisco and/or its affiliates. All rights reserved.
Centers
Cisco Confidential
31
Presentation_ID
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public