RTTometer: Measuring Path Minimum RTT with Confidence

Transcription

1 RTTometer: Measuring Path Minimum RTT with Confidence Amgad Zeitoun Zhiheng Wang Sugih Jamin Department of Electrical Engineering and Computer Science, The University of Michigan, Ann Arbor Abstract Internet path delay is a substantial metric in determining path quality. Therefore, it is not surprising that RTT plays a tangible role in several protocols and applications, such as overlay network construction protocol, peer-to-peer services, and proximity-based server redirection. Unfortunately, current RTT measurement tools report delay without any further insight about path condition. Therefore, applications usually estimate minimum RTT by sending a large number of probes to gain confidence in the measured RTT. Nevertheless, large number of probes does not directly translate to better confidence. Usually, minimum RTT can be measured using few probes. Based on observations of path RTT presented in [], we develop a set of techniques not only to measure minimum path RTT but also to associate it with confidence level that reveals the condition on the path during measurement. Our tool, called RTTometer, is able to provide confidence measure associated with path RTT. Besides, given a required confidence level, RTTometer dynamically adjusts the number of probes based on path s condition. In this paper, we describe our techniques implemented in RTTometer and present our preliminary experiences using RTTometer to estimate RTT on various representative paths of the Internet. I. INTRODUCTION Internet path delay is one of the substantial metrics in determining path quality. Path delay, either one-way time or round-trip time (RTT), is often used to indicate distance between hosts and estimate the proximity of Internet hosts. It may also indicate the expected performance of an application that uses network as a transport medium. For example, the throughput of TCP (Transport Control Protocol) is partly dependent on the round-trip time between communicating hosts [2], [3]. Therefore, shorter RTT usually indicates smaller service time, i.e., faster transfer rate. Furthermore, Internet RTT is highly correlated with geographical distance; in fact, GeoPing [4] exploits the correlation between geographic distance and RTT to determine IP s geographic location. It maps an IP address into a geographic location by measuring RTT from multiple points on the Internet. Even though the primary goal of Internet architecture is not to facilitate performance measurements between endhosts [5], several tools to measure RTT exploit features in the IP protocol. One of the common tools to measure path RTT is ping. The tool works by sending an ICMP (Internet Control Message Protocol) packet, usually called a probe, which forces end-host to elicit a reply message. The RTT, then, is the elapsed time between the sending of the ICMP packet and the recipient of the reply. The main goals of ping are two folds. First, it is often used by operators to determine host reachability. Second, it provides a way to observe the dynamics of the RTTs along a path to determine any path anomalies. Besides, ping reports the RTT of each probe and it briefly provides statistics on probes (e.g., minimum, average, and maximum RTTs) as well as probe losses. Due to the preferential treatment of ICMP packets by different routers and hosts, other flavors of ping-like tools have emerged. All these tools, however, follow the same philosophy of the ping tool in measuring and reporting delays with minimal statistics. Services that use RTT probes as a regular routine to enhance their performance [6], [7], [8], [9], [0], [4], such as, service mirroring, distributed games, peer-to-peer applications, and measurement infrastructures have been proliferated nowaday. These services and applications usually require estimating minimum path RTT between hosts. The accuracy, or, confidence, of the measured path RTT is critical to their operational performance. To achieve certain level of confidence in their measured path RTT, these services usually send a large number of probes. For example, some services even send 220 probes to estimate minimum RTT! These services, however, select the value of a single probe among all these probes (e.g., minimum RTT) to reflect the property of the path. They often do not make use of the information provided by the remaining probes. Therefore, the main motivation to justify sending these large number of probes is to add confidence to the captured minimum RTT. Even with these large number of probes, no service has shown the confidence in the captured minimum RTT or the justification for the number of probes chosen. In this paper, we present RTTometer, a tool to estimate the minimum RTT of a path with an associated confidence measure. RTTometer provides the same functionality of the ping and other ping-like tools, but it differs in several ways. First, RTTometer uses TCP probes instead of ICMP probes. Second, other than the basic statistics for probes, it provides a measure of accuracy, i.e., confidence, associated with the minimum RTT observed along the path. The associated confidence takes into account the condition on the path during RTT measurement (e.g., congestion and route change); therefore, it uses the information provided by all probes. Finally, RTTometer dynamically determines the number of probes required to capture the minimum RTT with a given accuracy. An application can determine the tolerable error range and the confidence measure according to its requirements. Our philosophy comes from the belief that there is no general solution that fits all applications, but rather, each application can determine the best number of probes required to achieve a given accuracy in the measured RTT. Even for the same

2 path the required number of probes needed to capture RTT may change according to the path s condition, e.g., congestion level. RTTometer is potentially useful for applications that rely on measuring path RTT as a regular routine of their operation. Specifically, these applications that depend on measuring minimum RTT of a path. The minimum path RTT is a property related to the topology of the network rather than the load of the network, hence it is of most interest to applications and services that are sensitive to network topology. Potential use of RTTometer is in the field of overlay network such as Content-Addressable Network (CAN) [6], peer-to-peer applications such as Napster [7], various multiplayer game servers such as Gamespy [8], end-to-end Internet distance estimation services such as GNP [8] and IDMaps [9], geographical IP mapping services such as GeoPing [4], or end-host multicast applications such as Narada [0] and HMTP [7]. All these services and applications rely on measured RTT as a fundamental tool to determine the location of an Internet host relative to one or more measurement points. Inaccurate measured RTT would result in performance degradation in these applications. The rest of this paper is organized as follows. In Section II, we present the set of measurement concepts and techniques used in RTTometer. We, then, describe the design and implementation of our tool in Section III. We present some preliminary experiences studying a large number of paths on the Internet and provide general observations in Section IV. We briefly summarize several related work in Section V Finally, the paper concludes in Section VI with final remarks. II. RTTOMETER CONCEPT The concept of RTTometer tool is based on observations developed in [], where authors identified several challenges in capturing the minimum RTT. This paper, however, addresses the issue of implementing a tool for operational networks. It provides a better approach in calculating the confidence of the measured minimum RTT, and empirically tests this tool on a large number of paths on the Internet. Estimating the minimum RTT along a path requires sending a number of probes, K, with inter-probe interval of τ time units. The probing period, thus, lasts for L time units, where L = Kτ. There is a set of challenges that face the process of measuring minimum path RTT, namely, self-interference, persistent congestion, and route changes. Self-interference occurs when the inter-probe interval τ is too small, the rate at which probe packets are sent may be high that they queue up behind each other. Persistent congestion makes all probes sent during the measurement period L experience queueing delays or some losses; the captured RTT, then, would not represent the minimum RTT of the path. Finally, route change is another factor that may change the value of the measured RTT and, hence, can be misinterpreted as persistent congestion. In [], authors studied the effect of each controllable parameter, K, τ, and L on the accuracy of the measured RTT. For example, K determines the amount of traffic required by Fig.. II I III II An example of a phase-plot showing different congestion regions. the measurement process, and the choice of τ can avoid selfinterference. In addition, they developed a technique to detect inaccuracies in the measured RTT due to congestion and route changes (uncontrollable factors). So, not only the minimum RTT of a path is measured but also a confidence level, which reflects the congestion condition along the path, is provided to indicate how strong is the estimated minimum RTT. A. Phase Plot RTTometer s main mechanism depends on the use of the phase-plot technique described in []. Given a set of RTT measurements, the phase-plot shows the correlation between two adjacent measurements. In a phase-plot, the x-axis ticks represent RTT n and the y-axis ticks represent RTT n+, where n and n+ are indices of consecutive probes. Fig. shows an example of a phase-plot. For probes that see an empty queue, the experienced delay for each probe is RTT n+ = RTT n = p + ε, where p is the propagation delay of the path and ε represents some random delay due to media access contention, router processing overhead, ARP resolution, etc. We discuss how to estimate ε in Section II-B. If there are no queueing delays experienced by the probes then delays are clustered around the point of value p at the bottom left corner of the phase-plot. When a probe packet experiences queueing delay, the corresponding data point in the phase-plot deviates from p. Self-interference happens when inter-probe interval is small such that probes start to queue behind each other []. If two or more adjacent probes are queued behind each other, then RTT n+ = RTT n τ, where τ is the interprobe interval. With the concept of laying points on the phaseplot, RTTometer uses techniques described in [] to detect self-interference. B. Detecting Congestion and Route Change Other than self-interference, RTTometer needs to detect the condition of the network during measurement, mainly congestion and route changes. Minimum path RTT may change due to congestion or route change conditions on the measured path. Different applications may react differently when the change in minimum RTT is due to one or the other of Transmission delay is negligible due to small probe size.

3 these causes. Therefore, RTTometer provides a confidence level associated with the minimum RTT; based on that the application decides according to its requirements. The notion of congestion regions and congestion components are introduced in the phase-plot to capture network dynamics during RTT measurement. In a phase-plot of a given measurement period there are three congestion regions as shown in Fig. : a) Region I contains probe pairs that see empty queues and experience minimum RTT plus minor random overheads ε due to media access contention, router processing overhead, etc. b) Region III contains probe pairs that always see a queue. This is the region of persistent congestion. c) Region II contains probe pairs where one of the probes experiences queueing delay but the other does not, i.e., there is a transition in congestion state between adjacent probes. This is the region of transient congestion. The congestion regions are determined based on two values, the minimum RTT and ε. The minimum RTT determines the bottom left corner of region I, while ε delimits the boundaries of regions I, II, and III. The setting of ε accommodates for a marginal error of the measured minimum RTT. In addition, on a non-congested path, region I should contain most of the points, and hence the most frequent value of the measured RTT. Therefore, ε is equal to the size of a window around the most frequent values of the RTT observed on a non-congested path (i.e., the statistical mode). This also implies that the window size is double the distance between the minimum RTT and the mode RTT. Given a trace of RTT measurements over a path, to detect the congestion level of the measured path, RTTometer computes the distribution of data points in the three regions of the phase-plot. Assume that the value of minimum RTT on the path is minrtt. Point p i = (RTT i, RTT i+ ) is in region I if RTT i (minrtt + ε) and RTT i+ (minrtt + ε). A point is in region II if MAX(RTT i, RTT i+ ) > (minrtt+ε) and MIN(RTT i, RTT i+ ) (minrtt+ε). Finally, a point is in region III if RTT i > (minrtt+ε) and RTT i+ > (minrtt +ε). Given N points in the phase-plot, the computation of congestion component C I is: C I = N N p i,i= (RTT i ), where () (RTT i+ ) { : x = minrtt (x) = x minrtt ε : x > minrtt For all points in region II, the computation of congestion region C II is: C II = N [K (MAX(RTT i, RTT i+ )) ], p i region II (2) where K is the number of points in region II. Finally, for all points in region III, the computation of congestion region C III is: C III = N [M p i region III (RTT i ) ], (3) (RTT i+ ) where M is the number of points in region III. Since each point in the phase-plot represents two consecutive measurements of path RTT, the calculation of C I, C II, and C III depends on the value of both RTTs for each point. The closer the RTT value to region I the more weight it has. Different than the approach followed in [] in calculating C I, instead of assigning weight of to each point in region I and 0 otherwise, in equation () each point in the phase-plot contributes to the value of C I, the closer the point to region I the more weight it adds. In the current approach, we divide the phase-plot into squares, where each square has a height and width of value ε, and the bottom left square is region I. Given a value of RTT, function ( ) calculates the distance between this RTT and minrtt in terms of the number of squares, e.g., if RTT (minrtt + ε), the distance is. For each point in the phase-plot, the distances of its RTTs are used to calculate its weight. The maximum weight a point can add is (when the point is within region I). The weight of a point is calculated by multiplying the inverse of its RTTs distances. Therefore, a point close to region I has more weight than a point that is further away. Therefore, C I indicates the confidence level in the measured RTT (e.g., if all points are in region I, C I equals to ) and C III indicates the condition on the path during measurements. A reported minimum RTT along with a large value of C III may indicate either persistent congestion or route change. To distinguish between the two cases, RTTometer provides the number of lost probes in addition to the reported C j s, j = I, II, and III. Given the fact that some probes get lost due to path congestion, an application can determine whether the change is due to congestion or route change. That is, applications should consider probe losses to signal for congestion even if C I is high. C. Summary We briefly summarize some of the main conclusions in [] regarding the RTT measurement process. Inter-probe interval has minimum effect on the accuracy of the captured minimum RTT. The inter-probe interval τ can be set to a reasonable value that does not cause self-interference among probes. Given a small error range, i.e., ε, minimum RTT of a path can be captured for a stationary non-congested path with a few number of probes. To minimize the effect of route changes on the measured minimum RTT, the choice of the measurement period L should be small enough that the probability of route change occurring during this period is small (one minute or less). By leveraging the findings in [], we develop the measurement tool, RTTometer, to estimate the minimum path RTT

4 with certain confidence level. In the subsequent sections we present our implementation and experiences of RTTometer. III. IMPLEMENTATION ISSUES In this section we outline our implementation for RTTometer. RTTometer does not require any changes on the receiver s side. Similar to ping and other ping-like tools, it elicits replies from target hosts by exploiting protocol s semantics. Due to the limitations and disadvantages of ICMP probes [2] (mainly, due to security concerns and preferential treatment at routers), RTTometer uses TCP probes to measure RTT, often called TCP ping. The mechanism of TCP ping was introduced in previous studies [3], [4], and it has been evaluated in []. To measure RTT between a source and a target host using TCP ping, the source sends a TCP SYN packet to the target. The time elapsed until a TCP SYN-ACK or RST packet returns from the target is the RTT. RTTometer uses raw socket to send TCP probes and intercepts the replies through packet sniffing using libpcap [5]. The timestamps of packets captured by libpcap are used to indicate the send and receive time of probes and probe replies, respectively. The RTT is, then, the difference in timestamps between the probe reply and the probe. In addition, we choose not to suppress the TCP replies that emerge from the source host s operating system. For example, let s imagine the scenario where RTTometer sends a SYN request to the target host and the target host replies with a SYN-ACK. RTTometer captures the reply packet through libpcap and another copy is delivered to the operating system s TCP stack, which subsequently replies with a RST packet because there are no states for the received packet. We decided to leave this behavior as it helps to guarantee that the connection at the target machine is not left in a half-open state, just a matter of being courteous. There are two main modes of operation for the RTTometer. The first mode is used to estimate the value of ε of a given path. In this mode, RTTometer sends a number of probes and estimates the value of ε as double the distance between the minimum and mode RTTs observed from these measurements. Preferably, the number of probes is large enough so that the mode RTT is statistically strong. RTTometer reports the number of probes lost and the values of C j, j = I, II, and III, a user should take the lost probes and the values of congestion regions into consideration when estimating the value of ε. Estimating ε during congestion periods may introduce errors in the estimated value. Should a user experiences congestion while estimating ε (packet loss and C III are used to indicate congestion), a user is advised to repeat the estimation some time later. The second mode of operation is the minimum RTT estimation of a path. In this mode a user specifies the maximum number of probes to be sent and may specify the value of ε and the value of the confidence required (i.e., the threshold on C I ). The default value of ε is 2, 4, or 6 ms based on the path minimum RTT (see Section IV-B) with a confidence level of 0.8. RTTometer starts sending probes and calculating the value of congestion regions. The probing routine stops when the required value of confidence has been reached for a given value of ε, or when the maximum number of probes is reached. In either case, RTTometer reports the minimum RTT captured along with the values of C I, C II, and C III. In addition, the number of probes lost, 0-percentile, 25- percentile, median, mode, 75-percentile, 90-percentile, and the maximum of the observed RTTs are reported. In both modes, user can specify the inter-probe interval (minimum value is 0 ms and the default value is 500 ms). Besides these parameters, a user can set the timeout value for probes. If RTTometer does not receive a reply to a probe within a timeout time units, the probe is considered lost (default value is 3 seconds). In addition, a user can also specify the port number of the TCP probes (default is port 80). Finally, the value of the Time-To-Live (TTL) of the probes can be specified. This allows RTTometer to measure the RTT to a given hop along a path without addressing the probes directly to that node (useful for probing routers along a path). IV. EXPERIENCES In this section, we present some preliminary observations from a broad set of experiments to measure the delay on different paths. First, we study the characteristics of the distance between the mode RTT and the minimum RTT of a path, which is used to determine ε. We then discuss the issue of estimating ε from the operational point-of-view. Our main motivation is to understand whether the estimation of ε needs to be done quite frequently, or there is a reasonable default value for ε. Finally, we look at the dynamics of the number of probes required to capture the minimum RTT along a path with a specific setting of ε and at a given confidence level. A. Observations on Mode RTT In this experiment we study the behavior of path RTT. Specifically, we are interested in studying the distribution of the RTT and how close the mode RTT to the minimum RTT on a given path. We run RTTometer from the University of Michigan to a set of 5,937 hosts, mostly Web servers. Due to the diversity of connectivities on the Internet, choosing a representative set of paths is not a simple task. However, we make sure that our set of target hosts contains hosts at different continents, connected through different links (high speed links, DSL, cable modem, etc.), and belonging to different organizations (educational, commercial, governmental, etc.). For each target host, we send 00 probes with inter-probe interval of 500 ms. To better understand and quantify the behavior of the path RTT, we classify the set of paths into three groups based on their minimum RTT. The first group (G) contains all paths with minimum RTT less than or equal to 50 ms. The second group (G2) contains paths with minimum RTT between 50 ms and 50 ms. The third group (G3) contains the remaining set of paths, i.e., larger than 50 ms. Table I shows the number of paths in each group. Probe losses are not uniformly distributed among all paths. Instead, some paths suffer from severe probe losses while others are not, i.e., the

5 TABLE I CLASSIFICATION OF TARGET PATHS. Group Delay range Paths Avg. loss G 50 ms 2403 (40.5%) 2.3 G ms 2594 (43.7%) 2.4 G3 > 50 ms 940 (5.8%) 4.2 TABLE II STATISTICS OF (MODE RTT - MINIMUM RTT) FOR DIFFERENT GROUPS. Percentile G G G Cumulative Distribution Function G G2 G (Mode - Minimum) (ms) Mode - Minimum (ms) Mode Strength G G2 G3 Fig. 2. CDF of the difference between minimum and mode RTTs on each path at different groups. Fig. 3. The distance between minimum RTT and mode RTT and its relation to the strength of the observed mode RTT. loss distribution has a long tail. For example, for G,,724 paths did not suffer any probe loss while more than 70 probes were lost on few paths. For G2,,823 did not suffer any probe loss, and similarly 555 paths in G3. Recall that on the phase-plot ε delimits the set of congestion regions, we, thus, look at the setting of ε for different paths. The main question we investigate is whether there is a reasonable default value of ε for different paths. In [], authors found that for different paths the third quartile of the difference between mode RTT and minimum RTT is ms. For each group, we look at the distance between the minimum RTT and the mode RTT along each path. Fig. 2 shows the cumulative fraction of the distance between minimum and mode RTTs for paths that did not suffer any probe losses in G, G2, and G3. The distribution of the difference between mode and minimum RTTs is long-tail, we only display the distribution up to 50 ms in the figure. Table II shows statistics about the distribution in each group. We can see that the difference between mode and minimum RTTs slightly depends on the path RTT itself, e.g., even for paths with large RTT (G3) we can see that 50% of the paths have differences less than.4 ms. However, the tail of the distribution gets longer for paths with large RTT. Since we do not have access to the set of target hosts, we could not check if the route did not change during the course of our RTT measurement. 2 Therefore, some of the large differences between mode RTT and minimum RTT may be due to route changes. Besides, some of the target hosts are connected through relatively slow-speed links which have weird behavior (See IV-D for discussion on behavior of DSL and cable modem hosts). The result seems intuitive as on paths 2 We could identify some route changes by observing changes in the RTT values, e.g., on one path, RTT changed from 00 ms to 2 sec after the 20 th probe and stayed on that value till the end of our probing period. with large RTT the delay fluctuates more when crossing long haul links or satellite connections. We study the relationship between the distance between the minimum RTT and the mode RTT and the strength of the mode. The strength of the mode is defined as the fraction of observations that consist the mode of a path. Fig. 3 depicts this relationship. The y-axis represents the distance between minimum RTT and mode RTT in log scale, while x-axis represents the mode strength. We first note that most of the samples have less than 5 ms between their minimum RTTs and mode RTTs (Fig. 2). This suggests a small ε value. Furthermore, while such distance of some of the paths falls beyond 5 ms, all these paths have weak mode strength. This observation suggests that during ε estimation mode, the stronger the mode RTT is the more accurate is the estimation of ε. B. Dynamics of ε As pointed out earlier, ε of a path is important in delimiting different congestion regions. We have shown in the previous subsection that the distance between mode and minimum RTT on a given path may slightly depend on the minimum RTT of the path. Since ε is double the difference between mode and minimum RTTs, we investigate the stability of ε over time. The main motivation is to study whether an estimated ε at a given measurement period can be used at subsequent measurements as well. Can we choose a default value for ε that would suite different paths? We conduct an experiment where we pick 5 representative paths from each group. The minimum RTTs for the paths selected from G are 5 ms (path ), 5 ms (path 2), 25 ms (path 3), 35 ms (path 4), and 45 ms (path 5). For G2, we pick paths with minimum RTTs of 55 ms (path 6), 75

6 Mode - Minimun (ms) G G2 G Path Fig. 4. The dynamics of (mode RTT - minimum RTT) on 5 paths in each group. Paths -5 are from G, paths 6-0 are from G2, and paths -5 are from G3. The solid line represents the interquartile range where the point is the median, and the dotted line represents the range from 0%-tile to 90%-tile. ms (path 7), 95 ms (path 8), 5 ms (path 9), and 35 ms (path 0). Finally, from G3 we pick paths with minimum RTTs of 60 ms (path ), 25 ms (path 2), 240 ms (path 3), 300 ms (path 4), and 670 ms (path 5). For each path we send 00 probes every one hour over a period of one day. For each path we calculate the distance between mode and minimum RTTs. Fig. 4 shows the range of values for the difference between mode and minimum RTTs on each path. We can see that the ranges for path 6 and path 0 in G2 are quite large compared to other paths. By looking into their measurements, we find that path 6 experienced some probe losses (between 0% to 20%) during 4 measurement periods. For path 0, we do not observe many probe losses, but the minimum RTT is 35 ms and the distance between the mode and the minimum, although larger than other paths, still less than 4% of the minimum RTT. On path 2 in G3 we observe that the minimum RTT shifts from 25 ms to 200 ms and then to 230 ms. We conjecture that it may be due to some route changes (probe losses are less than 2%). Still the mode is close to the minimum RTT, i.e., the distance between mode and minimum is less than 0% of the minimum RTT for the worst case. From the figure, we can see that the variation in ε is relatively small compared to path s minimum RTT. Therefore, the estimated ε at a given time can still be used to delimit the boundaries of the congestion regions at other times. In fact, given the diversity of the minimum RTT on our observed paths, we believe that a default value of ε can be chosen for each group. For example, we can see that the variation in G is less than that in G2 and G3. Currently, RTTometer sets the default values of ε as 2, 4, or 6 ms for paths in G, G2, or G3, respectively. RTTometer can set the default value of ε, given that the user did not specify it on the command line, by observing the value of the RTTs for the first few probes. C. Number of Probes Given that the estimated ε at a given measurement period can be used during other times, in this subsection we study Fig. 5. Distribution of the required number of probes to estimate minimum RTT for different groups. the required number of probes to estimate the minimum RTT of a path. To elaborate, given a value of ε and a required confidence level, what is the required number of probes needed to estimate the minimum RTT? RTTometer stops probing once it reaches the required confidence level or after it sends the maximum number of probes. We study how the required number of probes changes over time. Again, we use the 5 paths chosen previously, where we estimate the minimum RTT with confidence level of 0.8. The setting of ε is 2 ms for paths in G, 4 ms for paths in G2, and 6 ms for paths in G3. We repeat our measurements every hour for a period of one day. We set the maximum number of probes sent for a given path to 30 probes. RTTometer starts calculating the confidence once a minimum number of probes have been sent. In our experiment, we set this minimum threshold to 5 probes. Fig. 5 shows the distribution of the number of probes. Note that the minimum number of probes sent by RTTometer is 5, therefore the value on the x-axis starts at 5 in the figure. As we see, for the majority of measurements, 6 or less probes are enough to estimate minimum path RTT with confidence 0.8. During some measurements, 30 probes were not enough to estimate the minimum RTT at the required confidence level. Specially, for paths in G and G3. This may happen due to the small value of ε in G or due to the large delays for paths in G3. Fig. 6 shows the estimation of minimum RTT for path 5 from G3 along with the confidence level. The left y-axis represents the minimum delay in milliseconds, while the right y-axis represents the confidence level. From the figure we can see that for a couple of measurement periods, the confidence level dropped severely. By looking at the measurements for these periods, we find that the first dip occurred due to a delay shift from 20 ms to 440 ms. The first two probes experienced the low delay while the remaining 28 probes experienced the high delay. RTTometer correctly signaled for low accuracy level due to route changes. In the second dip, we find that the delay did not change, but the confidence level was low. During this measurement period, probes experienced high loss rate (about 40%). Even though the minimum RTT still

7 Minimum RTT (ms) RTT Confidence Measurement Index Confidence RTT (ms) End-host Next-to-last hop Probe Sequence Fig. 6. The estimated minimum RTT on path 5 from G3 and the corresponding confidence level. conforms with previous measurements, RTTometer signaled for congestion behavior on the path. D. DSL and Cable Modem End-hosts During our course of data collection, we observed some weird behavior on some paths. Closely investigating these paths we find that they are paths where the last hop is a broadband link (DSL or cable). On these paths the RTT fluctuates more often. For example, RTT may fluctuate more than 00 ms. We have observed that the fluctuation is more prevalent for hosts connected with cable modem. In one case, RTT measured to the end host fluctuates between values from 26 ms up to 400 ms! When we measure the RTT on the path up to the next-to-last hop, the RTT is 67 ms and without much fluctuation (Fig. 7). This indicates that the delay on the last hop is the main cause for these fluctuations. This fluctuation of RTT on the last hop may be due to routing mechanism done inside the cable network. In addition, cable networks tend to over-subscribe their network and rate limit user s traffic; it is common to have asymmetrical bandwidth on these links. Due to the prevalence of broadband links technology as connection media for users to the Internet, understanding the dynamics of RTT on these links is the main goal in our future works. V. RELATED WORK Given the administratively decentralized nature of the Internet, the ping utility has turned out to be the common primitive to all extant RTT measurement tools. To measure the RTT between a source and a target host, ping sends an Echo-Request ICMP message from the source host to the target host, and measures the elapsed time until an Echo-Response ICMP message returns from the target host. Due to the recent heightened security awareness of network administrators [2], increasingly large number of routers and firewalls do not forward ICMP packets. Furthermore, some system administrators turn off support for ICMP request-reply in their operating system kernel. Other ping-like tools have emerged due to the limitations and disadvantages of ping. TCP ping, for example, was introduced and used in [4], [3]. Application-level ping was Fig. 7. Behavior of RTT measured to end host connected through cable modem and RTT measured to the next-to-last hop. used in previous studies [3], e.g., HTTP ping measures the service time by a Web server to serve a GET request. However, similar to ping, these tools only provide some reachability test and simple statistics on the measured RTT. Other measurement tools leverage the existent of services to measure path delay. For example, King [6] estimates the delay between any pair of arbitrary hosts by measuring the latency between their authoritative DNS (Domain Name Service) servers. In all these tools, the main goal is to measure the delay between end hosts. Users of these tools usually specify the number of probes and the tools report the delay of each probe. There is no other measure associated with the captured delay. On the other hand, RTTometer not only measures the delay but also associates measure of confidence that reveals the condition of the path during measurements. In addition, RTTometer dynamically adapts the number of probes required for a given confidence level. VI. CONCLUSION In this paper, we have described techniques based on the use of phase-plot presented in [] to associate a confidence level to a measured path RTT. By laying out subsequent probes in different congestion regions in the phase-plot, we can assess the condition on a path during RTT measurement. Factors that may affect the accuracy of the measured RTT are selfinterference, congestion, and route changes. In addition, these techniques make use of the information provided by all probes, which reveals insight about path conditions. Using these techniques the RTTometer tool can associate an accuracy level with a measured RTT. RTTometer dynamically adjusts the number of probes required to measure the RTT with a given confidence level. Consequently, applications can make better decision not only based on the measured RTT but also on the revealed path condition. REFERENCES [] Z. Wang, A. Zeitoun, and S. Jamin, Challenges and Lessons Learned in Measuring Path RTT for Proximity-based Applications, Proceedings of Passive & Active Measurement Workshop (PAM 03), San Diego, CA, April 2003.

8 [2] T. Lakshman and U. Madhow, The Performance of TCP/IP for Networks with High Bandwidth-Delay Products and Random Loss, ACM/IEEE Transactions on Networking, June 997. [3] J. Padhye, V. Firoiu, D. Towsley, and J. Kurose, Modeling TCP Throughput: A Simple Model and its Empirical Validation, Proc. of ACM SIGCOMM 98, pp , 998. [4] V. N. Padmanabhan and L. Subramanian, An Investigation of Geographic Mapping Techniques for Internet Hosts, Proc. of ACM SIG- COMM 0, August 200. [5] D. Clark, The Design Philosophy of the DARPA Internet Protocols, Proc. of ACM SIGCOMM 88, pp. 06 4, 988. [6] S. Ratnasamy, P. Francis, M. Handley, R. Karp, and S. Shenker, A Scalable Content-Addressable Network, Proc. of ACM SIGCOMM 0, August 200. [7] B. Zhang, S. Jamin, and L. Zhang, Host Multicast: A Framework for Delivering Multicast to End Users, Proc. of IEEE INFOCOM 02, June [8] T. S. Eugene Ng and H. Zhang, Towards Global Network Positioning, ACM Internet Measurement Workshop 200, November 200. [9] P. Francis et al., IDMaps: A Global Internet Host Distance Estimation Service, ACM/IEEE Transactions on Networking, vol. 9, no. 5, Oct [0] Y. Chu, S. Rao, and H. Zhang, A Case for End System Multicast, Proc. of ACM SIGMETRICS 00, [] J.-C. Bolot, Characterizing End-to-End Packet Delay and Loss in the Internet, Proc. of ACM SIGCOMM 93, pp , Sep [2] S. Savage, Sting: a TCP-based Network Measurement Tool, USENIX Symposium on Internet Technologies and Systems 99, Oct [3] S. G. Dykes, C. L. Jeffery, and K. A. Robbins, An Empirical Evaluation of Client-Side Server Selection Algorithms, Proc. of IEEE INFOCOM 00, [4] M. Horneffer, Assessing Internet Performance Metrics Using Large- Scale TCP-syn Based Measurements, Passive & Active Measurement Workshop (PAM 00), [5] libpcap, [6] K. Gummadi, S. Saroiu, and S. Gribble, King: Estimating Latency Between Arbitrary Internet End Hosts, Proc. of ACM Internet Measurement Workshop (IMW 02), [7] Napster, [8] GameSpy Arcade,