CEG 2400 FALL 2012
Chapter 13
Troubleshooting Network Problems
1
Approach to Troubleshooting
• Proceed slowly, logically and methodically
• Use experience when necessary
• Benefits of a logical approach
– Prevents wasteful, time-consuming efforts
– Unnecessary software purchases
– Unnecessary hardware replacements
2
Troubleshooting Steps
• Troubleshooting steps
1. Identify problem
•
•
•
•
Gather information
Identify symptoms
Question users
Determine if anything has changed
2. Establish theory
•
Always question the obvious
3
Troubleshooting Steps
• Troubleshooting steps (cont’d.)
3. Test theory to determine cause
• If theory confirmed, determine next steps
• If not, establish new theory or escalate
• Test must not be destructive
4.
5.
6.
7.
8.
Establish action plan
Implement solution or escalate
Verify full functionality
Implement preventative measures if applicable
Document findings, actions, outcomes
4
1. Identify the Problem and Its
Symptoms
• Ask questions
• Avoid jumping to conclusions
• Pay attention to:
– Users
– System and network behaviors
– Error messages
• Treat each symptom uniquely
5
Identify the Problem and Its Symptoms
• Determine the problem’s scope
– Identify affected problem area
• Find out how many users or network segments are
affected
– Narrow down the time frame during which problem
occurred
• Discover time and frequency of problem
– May reveal more subtle network problems
• Benefits of narrowing scope
– Eliminate causes, may point to other problems
• Take your time to troubleshoot correctly
6
Identify the Problem and Its Symptoms
• Question users
– Ensure human error is not source of problem
– Watch the user if possible
• Use remote desktop software if you can’t be there to
watch
• Watch each step user takes
• Determine if anything has changed
– Be aware of recent network changes
• Possible actions if network change created problem
– Correct problem
– Reverse change
• Have/make network change records available to
staff members
7
2. Establish a Theory
• Re-create the symptoms if can be done safely
– Follow same steps as person reporting symptom
• Verify Physical layer connectivity
– Cabling and network adapters a common source of
problems
– Diagnosing Physical layer problems
•
•
•
•
Ask questions
Verify connections between devices
Verify soundness of connection hardware
Check cable
8
Establish a Theory
• Verify logical connectivity
–
–
–
–
Consider error messages
Note changes in operating system or applications
Determine if the problem is repeatable
Verify firmware and software configurations of
devices in path
– Try Ping utility
9
3. Test the Theory to Determine Cause
• Test Physical layer theories
• Example approaches
–
–
–
–
–
–
Use a cable testing tool
Check to make sure NIC is seated firmly in slot
Use wireless analyzer
Follow cables to verify physical connectivity
Try different port, data jack
Connect/use another device
10
Test the Theory to Determine Cause
• Test logical connectivity theories
• Example approaches
– View switch configuration to determine which nodes
are included in VLANs
– Investigate user permissions
– Examine NIC configuration (dhcp, ip address, etc)
– Ensure routing table includes valid entries
– Use tools such as ping, netstat, route, traceroute
– Check wireless client settings
11
Test the Theory to Determine Cause
• Escalate if necessary
– Ask colleague with more experience or knowledge
• Help desk analysts/ first-level support
– Proficient in basic workstation, network
troubleshooting
• Network specialist / Second-level support
• Network coordinator/ Third-level support
• Follow escalation procedures
12
4. Establish a Plan of Action to Resolve
the Problem
• Consider how the solution affects users/network
functionality
• Scope
– Assess solution’s scope before implementing
• Trade-offs
– Solution may restore functionality for one user group
but may remove functionality for others
• Security
– Be aware of security implications
• Understand access changes for authorized,
unauthorized users
13
Establish a Plan of Action to Resolve
the Problem
• Scalability of solution
– Position network for future additions, enhancements
– Try not to use temporary fix organization will outgrow
• Cost
– Weigh options carefully
• Use vendor information
– Manufacturer documentation
– Free online troubleshooting information (phone
support/web support)
– Consult with others within, outside your organization
14
5. Implement the Solution or Escalate
as Necessary
• Solution implementation
– Time required may be minimal or long
– Use methodical and logical approach
– Problem causing catastrophic outages should be
solved as quickly as possible
– Follow series of steps
• Implement a safe, reliable solution
• Large-scale fixes
– Roll out changes in stages
15
6. Verify Full System Functionality
• Verify problem solution is complete
– Type of testing depends on solution
– Type of testing depends on area
– May not be able to test solution immediately
• Determine how and why solution worked
• Evaluate effects on users (no negative
consequences)
16
7. Implement preventative measures if
applicable
• Consider how similar problems may be prevented in
the future
–
–
–
–
Increased maintenance
Thoughtful planning
Perform network health checks
Upgrading network
• Good planning
• Good design
17
8. Document Findings, Actions, and
Outcomes
• Necessary to record:
– Problem symptoms and causes and solutions
• Justification for recording
– Impossible to remember each incident’s
circumstances
– People leave/Job changes
• Use centrally located database accessible to all
networking personnel
18
Document Findings, Actions, and
Outcomes
• Have a good call tracking system (help desk
software)
• Supported services list document
– Contact personnel list both local and software
companies
– Lists all supported hardware, services, and software
• Follow-up with user who reported problem
19
Document Findings, Actions, and
Outcomes
• Notify others in organization of changes
– Record resolution in call tracking system
– Alert others about problem, solution, and network
changes made
• Have a change management system
– Means of documenting network changes
20
Troubleshooting Tools
• Utilities help troubleshoot network problems
– Ping
– Specialized tools
• Simple continuity testers
• Protocol analyzers
• Tool selection dependencies
– Problem being investigated
– Network characteristics
21
Tone Generator and Tone Locator
• Probe kit
– Generator and locator combination
• Tone generator (toner)
– Small electronic device that issues signal on wire pair
• Tone locator (probe)
– Emits tone when electrical activity detected
• Testing requires trial and error
• Used to determine where wire pair terminates
• Not used to determine cable characteristics
22
Use of a tone generator and tone locator
23
Multimeter
• Specialized tool used to test cables for faults
– Isolate problems with network cables
– Measures electric circuit characteristics
• Multimeter
– Single instrument for measuring impedance,
resistance, and voltage on a wire
– Has several uses
– Sophistication, features, and costs vary
24
A multimeter
25
Cable Continuity Testers
• Cable checkers (continuity testers, cable testers)
– Tests whether cable carries signal to destination
• Series of lights, audible tone
– Used to signal pass/fail
• Some continuity testers verify UTP, STP wires
paired correctly
– Not shorted, exposed, crossed
• Fiber optic continuity tester
– Issues light pulses on fiber
– Determines whether pulses reach other end
26
Cable continuity tester
27
Cable Performance Testers
• Determines if cable carrying current
• Continuity testers versus performance testers
– Differ in sophistication and price
– Performance tester accomplishes same tests
•
•
•
•
Can also perform additional tasks
Issue signal, measures signal bounce back
Indicates distance between nodes/ measure length
Indicates whether terminators properly installed,
functional, determine faulty splice locations, breaks,
connectors, bends
• Expensive
28
A high-end cable performance tester
29
Butt Set
• Lineman’s handset, telephone test set
• Uses
– Determine if line functioning (detects dial tone)
– Receives signal
– Picks up noise affecting signal
30
Butt set
31
Network Monitors
• Software-based tool
–
–
–
–
Continually monitors network traffic
Interprets up to Layer 3
Determines protocols passed by each frame
Cannot interpret frame data
• Tools developed by other software companies
– Purchase or free download
• Network adapter
– Must support promiscuous mode
32
Protocol Analyzers
• Protocol analyzer (network analyzer)
– Captures traffic, analyzes frames
• Typically to Layer 7
• Same features as network monitor but includes
additional features
– Generates traffic to reproduce network problem
– Can collect more information than can be reasonably
processed
– Set filters on gathered data
33
Protocol Analyzers
• Before using network monitor or protocol analyzer:
– Learn what network traffic normally looks like
– Capture data for time period on regular basis
• Establish a baseline to compare with future analyses
• Sniffer (packet sniffer)
– Older term referring to hardware device
34
Traffic captured by a protocol analyzer
35
Wireless Network Testers
• Tools containing wireless NICs, running wireless
protocols
• A workstation wireless network connection
– Shows little about wireless environment
– Only applies to one workstation
• Programs that scan for wireless signals
– Both software and hardware based
– Discover access points, wireless stations transmitting
36
Wireless Network Testers (cont’d.)
• Hardware instruments for wireless network testing
– Typically more portable than workstation with
software tools
– Preinstalled with network analysis tools
– Accessible from simple, graphical interface
– Contain powerful antennas
• Spectrum analyzer tool
– Tool to assess wireless signal quality
37
Wireless network testing tool
38
Network Monitors/Protocol analyzer
• Network Monitor software
–
–
–
–
Microsoft’s Network Monitor
Nagios
Ganglia
Plus there are commercial software also
• Variety of protocol analyzer software available
–
–
–
–
Wireshark
Ethereal
SmartSniff
Plus there are commercial software also
39
Summary
• Methodical, logical troubleshooting methodology
– 8 troubleshooting steps
• Troubleshooting tools
–
–
–
–
–
–
–
Tone generator, tone locator
Multimeters
Cable continuity testers
Cable performance tester
Butt set
Network monitors and protocol analyzers
Wireless network testing tools
40
End of Chapter 13
Questions
41