Pinpointing Anomalies in Large-Scale Traceroute Measurements. Romain Fontugne & Kenjiro Cho November 10, 2016

Transcription

1 Pinpointing Anomalies in Large-Scale Traceroute Measurements Romain Fontugne & Kenjiro Cho November 10, / 31

2 This presentation On going research work conducted at IIJ-II In collaboration with: Emile Aben (RIPE NCC) Cristel Pelsser (University of Strasbourg) Randy Bush (IIJ) 2 / 31

3 Agenda Background: Understanding Internet health Challenges Detect and locate Internet congestion: Analysis of traceroutes from RIPE Atlas Differential RTT and robust statistics Results: Study cases: DDoS attack and BGP leak 3 / 31

4 Understanding Internet health? 4 / 31

5 Understanding Internet health? 4 / 31

6 Understanding Internet health? 4 / 31

7 Understanding Internet health? (Problems) Manual observations and operations Traceroute / Ping / Operators group mailing lists Time consuming Slow process Small visibility Our goal: Pinpointing network disruptions (i.e. congestion and packet loss) 5 / 31

8 Silly solution: frequent traceroutes to the whole Internet! Doesn t scale Overload the network 6 / 31

9 Better solution: mine results from deployed platforms Cooperative and distributed approach Using existing data, no added burden to the network 7 / 31

10 RIPE Atlas Actively measures Internet connectivity Ethernet port Automatically perform active measurements: ping, traceroute, DNS, SSL, NTP and HTTP All results are collected by RIPE NCC 8 / 31

11 RIPE Atlas: coverage active probes! 9 / 31

12 RIPE Atlas: traceroutes Two repetitive large-scale measurements Builtin: traceroute every 30 minutes to all DNS root servers ( 500 server instances) Anchoring: traceroute every 15 minutes to 189 collaborative servers Analyzed dataset May to December billion IPv4 traceroutes 1.2 billion IPv6 traceroutes 10 / 31

13 Monitor delays with traceroute? Traceroute to Round Trip Time (RTT) between B and C? Report abnormal RTT between B and C? 11 / 31

14 Monitor delays with traceroute? Challenges: Noisy data RTT (ms) Traceroutes from CZ to BD Number of hops 12 / 31

15 Monitor delays with traceroute? 300 Traceroutes from CZ to BD RTT (ms) Challenges: Noisy data Traffic asymmetry Packet loss Number of hops 12 / 31

16 What is the RTT between B and C? RTT C - RTT B = RTT CB? 13 / 31

17 What is the RTT between B and C? RTT C - RTT B = RTT CB? No! Traffic is asymmetric RTT B and RTT C take different return paths! 14 / 31

18 What is the RTT between B and C? RTT C - RTT B = RTT CB? No! Traffic is asymmetric RTT B and RTT C take different return paths! Differential RTT: CB = RTT C RTT B = d BC + e p 14 / 31

19 Problem with differential RTT Monitoring CB over time: RTT Time Delay change on BC? CD? DA? BA??? 15 / 31

20 Proposed Approach: Use probes with different return paths Differential RTT: CB = x 0 16 / 31

21 Proposed Approach: Use probes with different return paths Differential RTT: CB = {x 0, x 1 } 16 / 31

22 Proposed Approach: Use probes with different return paths Differential RTT: CB = {x 0, x 1, x 2, x 3, x 4 } 16 / 31

23 Proposed Approach: Use probes with different return paths Differential RTT: CB = {x 0, x 1, x 2, x 3, x 4 } Median CB : Stable if a few return paths delay change Fluctuate if delay on BC changes 16 / 31

24 Median Diff. RTT: Example Tier1 link, 2 weeks of data, 95 probes: Differential RTT (ms) (Cogent, Zurich) (Cogent, Munich) Raw values 5.6 Differential RTT (ms) Median Diff. RTT Normal Reference Jun Jun Jun Jun Jun Jun Jun Stable despite noisy RTTs (not true for average) Normally distributed 17 / 31

25 Detecting congestion Differential RTT (ms) Nov Nov (HE, Frankfurt) (DE-CIX (RIPE)) Median Diff. RTT Normal Reference Detected Anomalies Nov Nov Nov Dec Significant RTT changes: Confidence interval not overlapping with the normal reference 18 / 31

26 Packet loss Worst case: router is not responding Cannot obtain RTT values Need to identify the faulty link 19 / 31

27 Packet forwarding model Learn usual paths from past traceroutes: 20 / 31

28 Identifying faulty links In case of packet loss: Query the model for the expected next hop Link AB is dropping packets! 21 / 31

29 Results Analyzed dataset Atlas builtin/anchoring measurements From May to Dec Observed 262k IPv4 and 42k IPv6 links We found a lot of congested links! Let s see only two significant examples 22 / 31

30 Study case: DDoS on DNS root servers Two attacks: Nov. 30th 2015 Dec. 1st 2015 Almost all server are anycast Congestion at the 531 sites? Found 129 instances altered by the attacks 23 / 31

31 Observed congestion Differential RTT (ms) Nov Differential RTT (ms) Nov Differential RTT (ms) Nov Nov (K-root) (1&1, Kansas City) Median Diff. RTT Normal Reference Detected Anomalies Nov Nov Nov Nov Nov Nov Nov Dec (HE, Frankfurt) (DE-CIX (RIPE)) Median Diff. RTT Normal Reference Detected Anomalies Nov Nov Nov Nov Dec (Selectel, St. Petersburg) (Selectel, Moscow) Dec Certain servers are affected only by one attack Continuous attack in Russia 24 / 31

32 Unaffected root servers Differential RTT (ms) Nov Nov (K-root) (Poznan, PL) Nov Median Diff. RTT Normal Reference Nov Nov Dec Very stable delay during the attacks Thanks to anycast! Far from the attackers 25 / 31

33 Congested links for servers F, I, and K Concentration of malicious traffic in IXPs 26 / 31

34 Study case: Telekom Malaysia BGP leak 27 / 31

35 Study case: Telekom Malaysia BGP leak 28 / 31

36 Study case: Telekom Malaysia BGP leak 28 / 31

37 Study case: Telekom Malaysia BGP leak 28 / 31

38 Study case: Telekom Malaysia BGP leak Not only with Google... but about 170k prefixes! 28 / 31

39 Congestion in Level3 Rerouted traffic has congested Level3 (120 reported links) Example: 229ms increase between two routers in London! Differential RTT (ms) Jun Median Diff. RTT Normal Reference Detected Anomalies Jun Jun Jun Jun Jun / 31

40 Congestion in Level3 Reported links in London: Delay increase Delay & packet loss Traffic staying within UK/Europe may also be altered 30 / 31

41 Summary Monitor delays with the Atlas platform Billions of (noisy) traceroutes Detect and locate Internet congestion Robust statistical analysis Diverse root causes: remote attacks, routing anomalies, etc... Give a lot of new insights on reported events On going work with RIPE NCC: Online detection and reports for network operators References: 31 / 31