Internet Routing (COS 598A) Today: Root-Cause Analysis Jennifer Rexford http://www.cs.princeton.edu/~jrex/teaching/spring2005 Tuesdays/Thursdays 11:00am-12:20pm Outline • Network troubleshooting – Motivation for network troubleshooting – Investigating from the edge vs. inside • Active probing – Traceroute – Mapping IP addresses to AS numbers • Passive monitoring – Analyzing BGP update streams – Identifying location and cause of routing change – Limitations of the approach Network Troubleshooting “Why can’t I reach www.cnn.com?” “Why is the performance bad?” Internet www.cnn.com Reachability Problems: What Could be Wrong? • End-host problem – Web server down – DNS server down, or misconfigured • Forwarding-path problem – Packet filter or firewall restricting access – Mismatch in Maximum Transmission Unit (MTU) • Routing problem – User or server disconnected from Internet – Blackhole dropping all packets – Persistent loop Performance Problem: What Could be Wrong? • End-host problems – Overloaded Web server – Overloaded DNS server – Overloaded user machine • Forwarding-path problem – High round-trip time – Link congestion • Routing problem – Long-term routing instability – Transient disruption during convergence Motivation for Troubleshooting • Improving performance – Detect, diagnose, and fix the problem – Pick a path through another provider – Pick a different path in any overlay network • Establishing accountability – Enforce Service Level Agreements – Rate service providers • Characterizing the Internet – Understand causes of performance problems – Understand challenges of troubleshooting Troubleshooting Outside vs. Inside • Outside: from network edge Today – Who: users and researchers, and operators troubleshooting problems outside their network – Data: ping/traceroute, public feeds of BGP updates, and public measurement platforms – Challenges: inference from very limited data • Inside: from inside the network – Who: operators running a network – Data: SNMP, fault data, traffic measurement, route monitors, and router configuration files – Challenges: collecting and joining the data Active Probing Pros and Cons of Active Probing • Advantages – Can run from any end system – Measure the actual forwarding path • See black-holes, loops, and delays directly • Disadvantages – Effects of routing changes, not the cause – Current path, not the path used in the past • Requires frequent probes to observe the changes – Shows only properties of round-trip path • Hard to tell if problem is on forward vs. reverse Traceroute: Measuring the Forwarding Path • Time-To-Live field in IP packet header – Source sends a packet with a TTL of n – Each router along the path decrements the TTL – “TTL exceeded” sent when TTL reaches 0 • Traceroute tool exploits this TTL behavior TTL=1 source Time exceeded destination TTL=2 Send packets with TTL=1, 2, 3, … and record source of “time exceeded” message Example Traceroute Output (Berkeley to CNN) Hop number, IP address, DNS name No response from router 1 169.229.62.1 inr-daedalus-0.CS.Berkeley.EDU 2 169.229.59.225 soda-cr-1-1-soda-br-6-2 3 128.32.255.169 vlan242.inr-202-doecev.Berkeley.EDU 4 128.32.0.249 gigE6-0-0.inr-666-doecev.Berkeley.EDU 5 128.32.0.66 qsv-juniper--ucb-gw.calren2.net 6 209.247.159.109 POS1-0.hsipaccess1.SanJose1.Level3.net 7 * ? 8 64.159.1.46 ? 9 209.247.9.170 pos8-0.hsa2.Atlanta2.Level3.net No name resolution 10 66.185.138.33 pop2-atm-P0-2.atdn.net 11 * ? 12 66.185.136.17 pop1-atl-P4-0.atdn.net 13 64.236.16.52 www4.cnn.com Example Troubleshooting Results • No packets go beyond your gateway – Gateway’s connection to Internet is dead • Traceroute stops at intermediate point – Perhaps a blackhole • Traceroute path has a loop – Transient or persistent forwarding loop • Traceroute shows a very long path – Routing anomaly, route hijacking, etc. • Traceroute shows very long delays – Delay or congestion on forward or reverse path Problems with Traceroute • Missing responses – Routers might not send “Time-Exceeded” – Firewalls may drop the probe packets – “Time-Exceeded” reply may be dropped • Misleading responses – Probes taken while the path is changing – Name not in DNS, or DNS entry misconfigured • Mapping IP addresses – Mapping interfaces to a common router – Mapping interface/router to Autonomous System Map Traceroute Hops to ASes Traceroute output: (hop number, IP) 1 169.229.62.1 AS25 2 169.229.59.225 AS25 Berkeley 3 128.32.255.169 AS25 4 128.32.0.249 AS25 5 128.32.0.66 AS11423 Calren 6 209.247.159.109 AS3356 7 * AS3356 8 64.159.1.46 AS3356 9 209.247.9.170 AS3356 10 66.185.138.33 AS1668 11 * AS1668 12 66.185.136.17 AS1668 13 64.236.16.52 AS5662 CNN Level3 AOL Need accurate IP-to-AS mappings (for network equipment). Candidate Ways to Get IP-to-AS Mapping • Routing address registry – Voluntary public registry such as whois.radb.net – Used by prtraceroute and “NANOG traceroute” – Incomplete and quite out-of-date • Mergers, acquisitions, delegation to customers • Origin AS in BGP paths – Public BGP routing tables such as RouteViews – Used to translate traceroute data to an AS graph – Incomplete and inaccurate… but usually right • Multiple Origin ASes, no mapping, wrong mapping Example: BGP Table (“show ip bgp” at RouteViews) Network * 3.0.0.0/8 * * * * *> * * 9.184.112.0/20 * *> * * * Next Hop Metric LocPrf Weight Path 205.215.45.50 0 4006 701 80 i 167.142.3.6 0 5056 701 80 i 157.22.9.7 0 715 1 701 80 i 195.219.96.239 0 8297 6453 701 80 i 195.211.29.254 0 5409 6667 6427 3356 701 80 i 12.127.0.249 0 7018 701 80 i 213.200.87.254 929 0 3257 701 80 i 205.215.45.50 0 4006 6461 3786 i 195.66.225.254 0 5459 6461 3786 i 203.62.248.4 0 1221 3786 i 167.142.3.6 0 5056 6461 6461 3786 i 195.219.96.239 0 8297 6461 3786 i 195.211.29.254 0 5409 6461 3786 i AS 80 is General Electric, AS 701 is UUNET, AS 7018 is AT&T AS 3786 is DACOM (Korea), AS 1221 is Telstra Why Would IP-to-AS Mapping Be Wrong? • IP addresses of equipment – Interfaces on the routers, not end hosts – Identifies equipment in routing protocols – Doesn’t need to be globally visible consistent • Three reasons the mappings may be “wrong” – Addresses of Internet Exchange Points – Sibling ASes that share address space – ASes that don’t announce their addresses • Look at traceroute path vs. BGP AS path – Traceroute path after IP-to-AS mapping – BGP AS path taken from the BGP table Extra AS due to Internet eXchange Points • IXP: shared place where providers meet – E.g., Mae-East, Mae-West, PAIX – Large number of fan-in and fan-out ASes A B C D E A E F B F G C G Traceroute AS path BGP AS path Ignore extra traceroute AS hop with high fan-in and fan-out Extra AS due to Sibling ASes • Sibling: organizations with multiple ASes: – E.g., Sprint AS 1239 and AS 1791 – AS numbers equipment with addresses of another A B C H D E A F B G C Traceroute AS path E D F G BGP AS path Merge sibling ASes “belong together” as if they were one AS. Unannounced Infrastructure Addresses 12.0.0.0/8 A B C does not announce part of its address space in BGP (e.g., 12.1.2.0/24) ACAC C AC BAC BC Fix the IP-to-AS map to associate 12.1.2.0/24 with C Refining Initial IP-to-AS Mapping • Start with initial IP-to-AS mapping – Mapping from BGP tables is usually correct – Good starting point for computing the mapping • Collect many BGP and traceroute paths – Signaling and forwarding AS path usually match – Good way to identify mistakes in IP-to-AS map • Successively refine the IP-to-AS mapping – Find add/change/delete that makes big difference – Base these “edits” on operational realities http://www.cs.princeton.edu/~jrex/papers/sigcomm03.pdf http://www.cs.princeton.edu/~jrex/papers/infocom04.pdf Research Areas • Better version of traceroute – Router support for active measurement – IPPM (IP Performance Measurement) – http://www1.ietf.org/mail-archive/web/imrg/current/msg00154.html • Peer-to-peer troubleshooting www.cnn.com “Yes” “No” Passive Monitoring Limitations of Active Measurements • Active measurements: traceroute-like tools – Can’t probe in the past – Shows the effect, not the cause AS 2 AS 4 AS 1 User (s) AS 3 Web Server (d) Appealing to Peek Inside • Passive measurements: public BGP data BGP update feeds Data Correlation Data Collection (RouteViews, RIPE) root cause Inspect BGP Routing Changes • Changes in paths to reach destination d – AS – AS – AS – AS 1: 2: 3: 4: “1 3 4” “1 2 4” “2 4” (no change) “3 4” “3 1 2 4” “4” (no change) AS 2 AS 4 AS 1 User (s) AS 3 Web Server (d) Idea #1: ASes in Paths Undergoing Change • Key assumption – “The AS responsible for the change appears in the old and/or the new AS path to the destination.” • If an AS has a routing change – All ASes in old and new paths may be responsible – Call these ASes the “suspect set” • Combining across vantage points – Consider all ASes that had a routing change – Perform the intersection across the suspect sets Idea #2: Excluding ASes in Non-Changing Paths • Key assumption – “If an AS has no routing change, the ASes in the path are not responsible and can be excluded.” • Example – AS 1: “1 2 4” “1 2 3 4”: suspects {1, 2, 3, 4} – AS 2: “2 4” “2 3 4”: suspects {2, 3, 4} – AS 3: “3 4” (no change): non-suspects {3, 4} AS 3 AS 1 AS 2 AS 4 Idea #3: Blaming the ASes in the Better Path • Key assumption – “The better path is the one that contains the AS responsible for the change.” • Example – “1 2 4” “1 2 3 4”: better path to worse path, with ASes {1,2,4} as the suspects (not AS 3) • Heuristics for identifying the “better” path – E.g., the shorter AS path AS 3 AS 1 AS 2 AS 4 Idea #4: Combining Across Destinations • Key assumption – “All destinations experiencing routing changes in a short period of time have a common cause.” • Exploiting the observation – Form suspect sets for each destination – Perform intersections of the sets across the destinations Difficulties With Root-Cause Analysis • Misleading BGP routing changes – Responsible AS not on old or new path – Looking across destinations doesn’t resolve • Missing routing changes – Some routers in an AS don’t have a change – Some subnets are not visible in BGP – Some internal changes are not visible in BGP Misleading BGP Changes Myth:The AS responsible for the change appears in the old or the new AS path. BGP data collection old: 1,2,8,9,10 new: 1,4,5,6,7,10 1 2 4 8 3 5 9 6 11 7 10 Misleading BGP Changes Myth:Looking at routing changes across prefixes resolves causes d2 AS 3 d3 AS 2 AS 1 d1 A B 7 10 12 C BGP data collection Changes for d2, but not for d1 and d3 Missing Routing Changes Myth: The BGP updates from a single router accurately represent the AS dst AS 2 AS 1 A B 7 6 12 C 10 D BGP data collection No change Missing Routing Changes Myth:BGP data from a router accurately represents changes on that router. 12.1.1.0/24 BGP data collection A 12.1.0.0/16 Missing Routing Changes Myth:Routing changes visible in eBGP have greater impact end-to-end impact than changes with local scope. dst AS 2 AS 1 A B 5 6 12 C 10 7 D BGP data collection Hybrid of Active and Passive Monitoring Omni 2 AS 2 i User (s) AS 4 AS 1 Omni 1 j AS 3 (i,s,d,t) failure link (3,4) (j,s,d,t’) failure link (3,4) Omni 4 Omni 3 Web Server (d) Research Questions • Understanding if root-cause analysis can work – How many vantage points are needed? – Do the assumptions usually hold? – Can algorithms tolerate occasional violations? – Can some additional information help? • Distributed algorithms for root-cause analysis – Can ASes cooperate in distributed fashion? – How to prevent or detect ASes that cheat? – Do all ASes have to participate? – Other hybrids of active and passive monitoring? Conclusions • Troubleshooting is important – Detect, diagnose, and fix problems – Accountability and service-level agreements • Troubleshooting is hard – Active measurement (e.g., traceroute) not enough – Root-cause analysis techniques are not enough • New innovation necessary – Hybrid active/passive approaches – Router support for active measurement – Routing protocol extensions for troubleshooting For Next Time: From Inside an AS • Two papers – “OSPF monitoring: Architecture, design, and deployment experience” – “Finding a needle in a haystack: Pinpointing significant BGP routing changes in an IP network” • Optional reading – Materials from Packet Design and Ipsum Networks • Review only of first paper – Summary – Why accept – Why reject – Future work