High Performance Network
Analysis
Enterprise Operate Practice
Cisco Services
Andrew Wojtkowiak – Network Consulting Engineer
© 2011 Cisco and/or its affiliates. All rights reserved.
1
• Background
Cisco Services performed an assessment of the wired
infrastructure to serve as a holistic health check of the
University Corporation of Atmospheric Research
network
• Goal of the assessment
To identify immediate remediation needs
Provide Opportunities for network improvement
© 2011 Cisco and/or its affiliates. All rights reserved.
2
Background
And
Key Areas
Assessed
© 2011 Cisco and/or its affiliates. All rights reserved.
High Level
Findings
Executive
Level
Findings
Remediation
Steps
Strengths and
Concerns
Encompassing
Projects
Looking
Forward
3
• The High Performance Network Analysis (HPNA) was performed to assure the stability of
the core routing and switching infrastructure
• Performed as a holistic network health check
• Emphasis placed on Availability and Resiliency with the Campus environments
• On-site interviews and data collection
• Analyzed ~80 devices as part of the HPNA
• Collected detailed network data such as topology diagrams, software, network
standards, protocols, etc…
© 2011 Cisco and/or its affiliates. All rights reserved.
4
• Network Topology
• Protocol Resiliency
• Network Service Resiliency
• Hardware and Software
© 2011 Cisco and/or its affiliates. All rights reserved.
5
© 2011 Cisco and/or its affiliates. All rights reserved.
6
• Dedicated and professional network staff
Everyone we worked with was very open, professional and accommodating
• Excellent Hardware and Software replacement strategies
Hardware and Software is kept up to date and staff is knowledgeable of bugs and vulnerabilities
• Change Management Process
Well documented and followed change management process
• Individualized tools for Network Management
Tools for deployments, configurations, backups, and management
© 2011 Cisco and/or its affiliates. All rights reserved.
7
• Single Points of Failure
Increased risk of a pervasive network incident; scalability and availability concerns
• Process Documentation
Lack of formal process to follow. No repeatable steps that all team members can use.
• Global Configuration Templates
Templates will help reduce configuration inconsistencies and ensure services are configured according to
policy
• Configuration Inconsistencies
Increased time to repair due to troubleshooting overhead; decreased network security; compliance risk
© 2011 Cisco and/or its affiliates. All rights reserved.
8
• A few single points of
failure
TCOM switch for internet
connectivity
Foothills Lab secondary
switch
NWSC second switch
• Major risk with TCOM
Higher latency backup
• Foothills under
• Foothills and NWSC would
limit connectivity from
those locations to the rest
of the network.
construction, second
switch in move
• NWSC secondary switch
is being considered
© 2011 Cisco and/or its affiliates. All rights reserved.
9
© 2011 Cisco and/or its affiliates. All rights reserved.
10
• Processes are well defined
by the individuals who
perform the tasks
Software and Hardware
replacement
Standards for implementing
new devices
• No actual defined
• Only certain people are
well versed in processes
• Not easily reproducible
• No defined steps for
changes
• Allocate time to turn
processes into
documentation
• Allocate someone to
review the documents
• Keep them up to date as
they change.
documentation
© 2011 Cisco and/or its affiliates. All rights reserved.
11
Total CatOS Summary
Cisco 6500 Series Switches IOS
12.2(33)SXH3
1
8.4
33%
12.2(33)SXH4
2
3
1
15
8.6
8.6
67%
8.4
5
2
12.2(33)SXH8
12.2(33)SXI4a
12.2(33)SXI5
12.2(33)SXI6
12.2(33)SXI8a
All CatOS has reached End of SW Maintenance, and will no
longer receive attention with regards to defect or security
vulnerability patching
© 2011 Cisco and/or its affiliates. All rights reserved.
12
• Configuration standards
are adhoc; without formal
documentation
• No way to perform
configuration compliance
to a template*
• Number of configuration
inconsistencies and errors
• Network unpredictability
• Potential increased
troubleshooting overhead
and operational difficulty
• Prolonged loss of
connectivity and service
interruption to critical
applications
• Increased exposure to
security vulnerabilities
• Increased cost associated
with operating the network
(Protocol, Service, Security)
© 2011 Cisco and/or its affiliates. All rights reserved.
13
• HSRP inconsistencies
• Partially configured
advanced spanning tree
features
• Optimize/Standardize
Spanning-tree priorities
• OSPF passive interface
© 2011 Cisco and/or its affiliates. All rights reserved.
• Some routers do not have
a peer
• Possible loops or rouge
switches influencing the
network
• Routing updates are not
limited
• Implement changes to the
network to remediate the
smaller configuration
inconsistencies
• The standard templates will
assist in ensuring fewer
deviations from standard.
14
© 2010 Cisco and/or its affiliates. All rights reserved.
15
• Three buildings connected
in a partial mesh topology
• Collapsed connections to
each other
• Port density growth at
N*(N-1) rate for every new
building
• Lack of modularity and
scalability
• Large fault domains
across all buildings
• Network disruption and
outages
• Increased troubleshooting
overhead
© 2011 Cisco and/or its affiliates. All rights reserved.
• Quantifiable cost increase
in both capital and
operational expenditure
Cost to Add 4th Building
Nx(N-1) = 12 Ports (6 Links)
Additional Capital Expenditure
associated with running fiber
Additional Operational Expenditure
associated with design complexity
16
Current Topology - No Core
• Fully-meshed distribution layers
• Physical cabling requirement
• Routing complexity
© 2011 Cisco and/or its affiliates. All rights reserved.
17
Center Green
Mesa Lab
Foothills
This leading practice hierarchical design
has been proven to:
New Location
 Promote easy growth and ease of
troubleshooting
 Reduce capital and operational
expenditure
 Create small fault domains
Dedicated Core
 Promote deterministic traffic flows
Dedicated WAN /
Internet Switch Block
Research Networks
Internet
© 2011 Cisco and/or its affiliates. All rights reserved.
 Enable logical and physical
topology mapping
TCOM/FRGP
Firewalls
18
• Monitoring facing the
Internet
Intrusion Prevention
SPAN Sessions to security
team
• Extensive ACLs on core
switches
• Limited methods to log
and account for network
incidents
• Increased CPU usage on
switches
• Create method to evaluate
internal ACLs routinely
• Consider Control Plane
Policing for basic
router/switch services
Routing
Switching
• No Control Plane Policing
to protect devices
© 2011 Cisco and/or its affiliates. All rights reserved.
19
Correlating business impact (risk
reduction) to ease of execution and
exemplar implementation time
High priority
> year
9months
Must Do – Reduce Risk
0-6 months
Quick Wins – High Business Impact
1
2
Project List:
1) Remediate single points of failure
2)
Create, utilize and maintain global configuration
standard templates
3)
Create, utilize and maintain process
documentation
4)
Remediate configuration inconsistencies within
the network
4
Very Hard
Easy But Low Return
3
Low priority
More complex to implement
© 2011 Cisco and/or its affiliates. All rights reserved.
Easy to implement
20
© 2011 Cisco and/or its affiliates. All rights reserved.
21
Thank you.