Pinpointing Delay and Forwarding Anomalies Using Large-Scale Traceroute Measurements

Transcription

1 Pinpointing Delay and Forwarding Anomalies Using Large-Scale Traceroute Measurements Romain Fontugne 1, Emile Aben 2, Cristel Pelsser 3, Randy Bush 1 November 1, IIJ Research Lab, 2 RIPE NCC, 3 University of Strasbourg / CNRS 1 / 25

2 Understanding Internet health? 2 / 25

3 Understanding Internet health? 2 / 25

4 Understanding Internet health? 2 / 25

5 Understanding Internet health? (Problems) Manual observations Traceroute / Ping / Operators group mailing lists Slow process Small visibility 3 / 25

6 Understanding Internet health? (Problems) Manual observations Traceroute / Ping / Operators group mailing lists Slow process Small visibility Our goal: Systematically pinpoint network disruptions Delay changes Forwarding anomalies (not covered here, see the paper) 3 / 25

7 Silly solution: frequent traceroutes to the whole Internet! Doesn t scale Overload the network 4 / 25

8 Better solution: mine results from deployed platforms Cooperative and distributed approach Using existing data, no added burden to the network 5 / 25

9 RIPE Atlas Actively measures Internet connectivity Multiple types of measurement: ping, traceroute, DNS, SSL, NTP and HTTP active probes! Data for numerous measurements is made publicly available 6 / 25

10 RIPE Atlas: traceroutes Two repetitive large-scale measurements Builtin: traceroute every 30 minutes to all DNS root servers ( 500 server instances) Anchoring: traceroute every 15 minutes to 189 collaborative servers Analyzed dataset May to December billion IPv4 traceroutes 1.2 billion IPv6 traceroutes 7 / 25

11 Monitor delays with traceroute? 300 Traceroutes from CZ to BD RTT (ms) Challenges: Noisy data Traffic asymmetry Packet loss Number of hops 8 / 25

12 Monitor delays with traceroute? Traceroute to Round Trip Time (RTT) between B and C? Report abnormal RTT between B and C? 9 / 25

13 What is the RTT between B and C? Differential RTT: CB =RTT C - RTT B? = RTTCB 10 / 25

14 What is the RTT between B and C? RTT C - RTT B = RTT CB? No! Traffic is asymmetric RTT B and RTT C take different return paths! 11 / 25

15 What is the RTT between B and C? RTT C - RTT B = RTT CB? No! Traffic is asymmetric RTT B and RTT C take different return paths! Differential RTT: CB = RTT C RTT B = d BC + e p 11 / 25

16 Problem with differential RTT Monitoring CB over time: RTT Time Delay change on BC? CD? DA? BA??? 12 / 25

17 Proposed Approach: Use probes with different return paths Differential RTT: CB = x 0 13 / 25

18 Proposed Approach: Use probes with different return paths Differential RTT: CB = {x 0, x 1 } 13 / 25

19 Proposed Approach: Use probes with different return paths Differential RTT: CB = {x 0, x 1, x 2, x 3, x 4 } 13 / 25

20 Proposed Approach: Use probes with different return paths Differential RTT: CB = {x 0, x 1, x 2, x 3, x 4 } Median CB : Stable if a few return paths delay change Fluctuate if delay on BC changes 13 / 25

21 Median Diff. RTT: Example Tier1 link, 2 weeks of data, 95 probes: Differential RTT (ms) (Cogent, Zurich) (Cogent, Munich) Raw values 5.6 Differential RTT (ms) Median Diff. RTT Normal Reference Jun Jun Jun Jun Jun Jun Jun Stable despite noisy RTTs Normally distributed Conf. interval: Wilson score Normal ref.: exp. smooth. 14 / 25

22 Detecting Delay Changes Differential RTT (ms) Nov Nov (HE, Frankfurt) (DE-CIX (RIPE)) Median Diff. RTT Normal Reference Detected Anomalies Nov Nov Nov Dec Significant RTT changes: Confidence interval not overlapping with the normal reference 15 / 25

23 Results Analyzed dataset Atlas builtin/anchoring measurements From May to Dec Observed 262k IPv4 and 42k IPv6 links We found a lot of delay changes! Let s see only two prominent examples 16 / 25

24 Case study: DDoS on DNS root servers Two attacks: Nov. 30th 2015 Dec. 1st 2015 Almost all server are anycast Congestion at the 531 sites? Found 129 instances altered by the attacks 17 / 25

25 Observed delay changes Differential RTT (ms) Nov Differential RTT (ms) Nov Differential RTT (ms) Nov Nov (K-root) (1&1, Kansas City) Median Diff. RTT Normal Reference Detected Anomalies Nov Nov Nov Nov Nov Nov Nov Dec (HE, Frankfurt) (DE-CIX (RIPE)) Median Diff. RTT Normal Reference Detected Anomalies Nov Nov Nov Nov Dec (Selectel, St. Petersburg) (Selectel, Moscow) Dec Certain servers are affected only by one attack Continuous attack in Russia 18 / 25

26 Unaffected root servers Differential RTT (ms) Nov Nov (K-root) (Poznan, PL) Nov Median Diff. RTT Normal Reference Nov Nov Dec Very stable delay during the attacks Thanks to anycast! Far from the attackers 19 / 25

27 Congested links for servers F, I, and K Concentration of malicious traffic at IXPs 20 / 25

28 Case study: Telekom Malaysia BGP leak 21 / 25

29 Case study: Telekom Malaysia BGP leak 22 / 25

30 Case study: Telekom Malaysia BGP leak 22 / 25

31 Case study: Telekom Malaysia BGP leak 22 / 25

32 Case study: Telekom Malaysia BGP leak Not only with Google... but about 170k prefixes! 22 / 25

33 Congestion in Level3 Rerouted traffic has congested Level3 (120 reported links) Example: 229ms increase between two routers in London! Differential RTT (ms) Jun Median Diff. RTT Normal Reference Detected Anomalies Jun Jun Jun Jun Jun / 25

34 Congestion in Level3 Reported links in London: Delay increase Delay & packet loss Traffic staying within UK/Europe may also be altered 24 / 25

35 Summary Detect and locate delay and forwarding anomalies in billions of traceroutes Non-parametric and robust statistics Diverse root causes: remote attacks, routing anomalies, etc... Give a lot of new insights on reported events Online detection for network operators 25 / 25