Binary Increase Congestion Control (BIC) for Fast Long-Distance Networks

Transcription

1 Binary Increase Congestion Control () for Fast Long-Distance Networks Lisong Xu, Khale Harfoush, an Injong Rhee Department of Computer Science North Carolina State University Raleigh, NC lxu2, harfoush, Abstract High-spee networks with large elays present a unique environment where TCP may have a problem utilizing the full banwith. Several congestion control proposals have been suggeste to remey this problem. The existing protocols consier mainly two properties: TCP frienliness an banwith scalability. That is, a protocol shoul not take away too much banwith from stanar TCP flows while utilizing the full banwith of high-spee networks. This paper presents another important constraint, namely, RTT (roun trip time) unfairness where competing flows with ifferent RTTs may consume vastly unfair banwith shares. Existing schemes have a severe RTT unfairness problem because the congestion winow increase rate gets larger as the winow grows ironically the very reason that makes them more scalable. RTT unfairness for high-spee networks occurs istinctly with rop tail routers for flows with large congestion winows where packet loss can be highly synchronize. After ientifying the RTT unfairness problem of existing protocols, this paper presents a new congestion control scheme that alleviates RTT unfairness while supporting TCP frienliness an banwith scalability. The propose congestion control algorithm uses two winow size control policies calle aitive increase an binary search increase. When the congestion winow is large, aitive increase with a large increment ensures square RTT unfairness as well as goo scalability. Uner small congestion winows, binary search increase supports TCP frienliness. The simulation results confirm these properties of the protocol. Keywors Congestion control, High-spee networks, RTT unfairness, TCP frienliness, Scalability, Protocol Design. I. INTRODUCTION The Internet is evolving. The eployment of wie-area high-spee networks like Abilene an ESNet has heightene eman for ata-intensive, high-performance computing. Applications like scientific collaboration, telemeicine, an real-time environment monitoring benefit from this eployment. These applications require access to high-banwith real time ata, images, an vieo capture from remote sensors such as satellite, raars, an echocariography. They, in aition, require preictable, low-latency access to this ata, in real time. TCP has been wiely aopte as a ata transfer protocol for these networks. TCP, however, substantially unerutilizes network banwith over high-spee connections with long The work reporte in this paper is sponsore in part by NSF CAREER ANI , NSF ANI RTTs [, 2, 4, 7]. TCP increases its congestion winow by one packet in every roun trip time (RTT) an reuces it by half at a loss event. A loss event is efine to be a packet loss that causes TCP to reuce its congestion winow. Over 83,333/2 RTTs are require for TCP to increase its winow from half utilization to full utilization of 0Gbps with 500-byte packets approximately hour with 00ms RTT. This requires that no packet is lost within hour. That is, the loss rate cannot be more than loss event per 2,600,000,000 packets, which is less than the theoretical limit of the network s bit error rates [, 2]. Fine-tuning TCP parameters such as receiver winows an network interface buffers [9, 0,, 2] may mitigate this problem. One straightforwar solution is to increase the packet size by using the Jumbo packet option (up to 8KB) an use multiple TCP connections [3, 4, 5]. This scheme is similar to Aitive Increase an Multiplicative Decrease (AIMD) where the winow increases with a fixe increment per RTT an ecreases by a multiplicative factor uner losses. Although these approaches enhance utilization by having a larger increment per RTT than TCP, the banwith ratio between these flows an stanar TCP flows is always fixe. Therefore, even in the TCP s well-behaving operating range (between loss rates of 0-2 to 0-4 ), they always use proportionally a larger banwith share compare to stanar TCP. Guaranteeing both TCP frienliness an banwith scalability with one fixe increase rate of winow is challenging. It calls for aaptive schemes that vary the winow growth rate epening on network conitions. The research community has respone quickly to the nee to aress TCP s limitations. Several promising new protocols have been put forwar incluing XCP [5], SABUL [6], FAST [7], High Spee TCP (HSTCP) [, 2, 3], an Scalable TCP (STCP) [4]. Except XCP, these protocols aaptively ajust their increase rates base on the current winow size. These protocols are claime to be TCP frienly uner high loss rate environments as well as highly scalable uner low loss environments. The work presente here began as an attempt to assess the performance of high-spee congestion control protocols in uner realistic network operation: i.e., to assess their impact on overall network performance in the presence of heterogeneous, competing flows. We focus on HSTCP an STCP, which are winow-base, self-clocking protocols known for safer incremental eployment [6]. Other protocols such as

2 SABUL an FAST are not stuie because SABUL is a rate-base protocol an FAST has not yet been escribe in etail. Lakshman an Mahow[9] stuie RTT unfairness of TCP in networks with high banwith elay proucts. It was reporte that uner a FIFO queue (i.e., rop tail) that TCP throughput is α inversely proportional to RTT where α 2. This paper extens their work by investigating RTT fairness for new high-spee protocols. Our stuy reveals that notwithstaning their scalability an TCP frienliness properties, HSTCP an STCP have a serious RTT unfairness problem when multiple flows with ifferent RTT elays are competing for the same bottleneck banwith. We efine the RTT unfairness of two competing flows to be the throughput ratio in terms of their RTT ratio. Uner a completely synchronize loss moel, we show that two HSTCP flows with RTT an RTT 2 have RTT unfairness ( ) RTT 2 RTT an those of STCP have RTT unfairness even ( RTT ) 2 RTT. That is, the shorter RTT flow eventually starves off the longer RTT flow. Note that TCP an AIMD have RTT unfairness ( ) 2 RTT 2 / RTT [9]. We report that this problem commonly appears for rop tail routers while the severity reuces for RED, an therefore, greatly impairs the ability to incrementally eploy the protocols, without aequate support from active queue management (such as RED an XCP). The severe RTT unfairness of HSTCP an SCTP comes from their aaptability ironically the very reason that makes them more scalable to large banwith. In HSTCP an STCP, a larger winow increases faster than a smaller winow. Compoune with elay ifferences, RTT unfairness gets worse as the winow of a shorter RTT flow grows faster than that of a longer RTT flow. Another source of RTT unfairness is synchronize loss: simultaneous loss events for multiple competing flows. Uner the assumption that packet losses are uniformly istribute across all flows, a flow with a larger winow gets more loss events than a flow with a smaller winow. However, this assumption oes not hol true in practice, especially in networks with rop tail routers. A short-term loss rate uring the perio aroun the overflow of router buffers can be much higher than the average long-term loss rate. Therefore, it is very possible that both flows lose packets at the same time when the buffer overflows; otherwise none of them lose packets. Assuming a ranom packet loss probability p with the uniform istribution, the probability that a flow with winow size w loses at least one packet within an RTT is ( p) w, which increases exponentially as winow size w increases. That is, many high-spee connections with relatively large winows more likely lose packets simultaneously at the time of buffer overflow. Synchronize loss encourages RTT unfairness. Since loss events are more synchronize across ifferent winow sizes, larger winows with a short RTT can always grow faster than smaller winows with a long RTT. Furthermore, since synchronization can prolong convergence an cause a long-term oscillation of ata rates, it can hurt banwith fairness in short-term time scales. In our simulation stuy, we observe the short-term unfairness of STCP an HSTCP over both rop-tail an RED routers. Designing a high-spee congestion control that supports all three properties of RTT fairness, scalability, an TCP frienliness is challenging. For instance, AIMD, which provies square RTT unfairness an scales its banwith share by increasing its aitive increase factor, is not TCP frienly. HSTCP an STCP, which are extremely scalable uner low loss rates an TCP frienly uner high loss rates, are not RTT fair. Recognizing these challenges, we propose, in the secon part of this paper, a new protocol calle Binary Increase Congestion Control () satisfies all these criteria:. Scalability: can scale its banwith share to 0 Gbps aroun 3.5e-8 loss rates (comparable to HSTCP which reaches 0Gbps at e-7). 2. RTT fairness: for large winows, s RTT unfairness is proportional to the inverse square of the RTT ratio as in AIMD. 3. TCP frienliness: achieves boune TCP fairness for all winow sizes. Aroun high loss rates where TCP performs well, its TCP frienliness is comparable to STCP s. 4. Fairness an convergence: compare to HSTCP an STCP, achieves better banwith fairness over both short an long time scales, an faster convergence to a fair banwith share. The paper organize as follows: Section II escribes our simulation setup. Section III iscusses evience for synchronize loss. In Sections IV an V, we iscuss the behavior of HSTCP an STCP. In Section VI, we escribe an its properties. Section VII gives etails on simulation results. Relate work an Conclusion can be foun in Sections VIII an IX. II. SIMULATION SETUP Fig. shows the NS simulation setup that we use throughout the paper. Various bottleneck capacity an elays are teste. The buffer space at the bottleneck router is set to 00% of the banwith an elay prouct of the bottleneck link. All connections pass through the bottleneck link. Each link is configure to have ifferent RTTs an ifferent starting an finishing times to reuce the phase effect [7]. Three kins of backgroun traffic are simulate. Substantial web traffic consuming a minimum of 20% of bottleneck banwith, an up to 50% of bottleneck banwith when no other flows are present is generate in both irections to remove synchronization in TCP feeback. Twenty-five small TCP flows whose congestion winow sizes are restricte to be less than 64 are transmitte in each irection. Their starting an finishing times are set ranomly. An two to four long-live TCP flows (with no winow size restriction) are create for both irections. These three kins of simulate backgroun traffic totally consume a minimum of 20% of the backwar banwith.

3 S S2 S3 Sn x ms N forwar x backwar This simulation topology is roughly comparable to that of the real high-spee networks. High-spee networks of our interest are ifferent from the general Internet where a majority of bottlenecks are locate at the eges. High-spee networks with high-spee eges can create a bottleneck at locations where much high-spee traffic meets, such as Startlight in Chicago that connects CERN (Geneva) an Abilene. We make no claim about how realistic our backgroun traffic is. However, we believe that the amount of backgroun traffic in both irections, an ranomize RTTs an starting an finishing times are sufficient to reuce any simulation anomaly, e.g., the phase effect an synchronize feeback. We also pace TCP packets so that no more than two packets are sent in burst. A ranom elay between packet transmissions is inserte to avoi the phase effect. We test both RED an rop tail routers at the bottleneck link. For RED, we use aaptive RED with the bottom of max_p set to III. SYNCHRONIZED PACKET LOSS We use a synchronize loss moel for the theoretical analysis of RTT fairness. Before elving into the analysis, we provie some evience that synchronize loss is common in high-spee networks. To measure the extent of the synchronization, we ran a simulation involving 20 high-spee connections of HSTCP with RTTs varying from 40 ms to 50 ms in the network shown in Figure with bottleneck link banwith varying from 00 Mbps to 2.5 Gbps. To measure synchrony in packet losses, we first compute the average time interval between two consecutive loss events of a connection. We efine measurement interval to be the uration corresponing to one-tenth of the average interval. We consier all the flows having at least one loss event within the same synchronization interval unergo synchronize losses. To measure the egree of synchronize losses, we ivie the total simulation time by the unit of the measurement intervals, an then in each interval count the number of unique high-spee flows that have at least one packet loss. Synchronization ratio is the ratio of the number of measurement intervals containing at least x flows, where x is a preefine threshol, to the number of measurement intervals containing at least one flow. Figure 2 shows synchronization ratios of RED an rop tail routers when The use of a smaller bottom value is recommene by [] to reuce synchronize loss normally this value is set to 0.0. N2 Bottleneck Link X2 ms Fig.. Simulation network topology D D2 D3 Dn Synchronization Ratio DropTail, 50% threshol RED, 50% threshol DropTail, 25% threshol RED, 25% threshol Banwith (Mbps) Fig. 2: Synchronization Ratio uner RED an rop tail routers. the threshol is set to 25% an 50% of the total number of high-spee flows respectively. We observe that the synchronization ratio increases as the banwith increases, especially uner rop tail routers. As the banwith increases from 00 Mbps to 2.5 Gbps, the synchronization ratio with 50% threshol (i.e., 0 flows) increases from about 0.23 to about 0.6. That implies whenever a flow loses a packet in the rop tail router, the probability that at least half of the total flows experience packet losses uring a perio of measurement interval increases from 23% to 60%. On the other han, RED oes not incur as much synchronize loss. The synchronization ratio with 50% threshol is always close to zero. However, there still exists some amount of synchronization; the synchronization ratio for RED with 25% threshol uner 2.5 Gbps is about 23%, which means the probability that at least a quarter of the total flows have a synchronize loss event is aroun 23%. The result implies that the number of synchronize loss can be quite substantial in rop tail. Although it requires real network tests to confirm this fining, we believe that our simulation result is not ifficult to recreate in real networks. We leave that to future stuy. IV. RTT UNFAIRNESS OF HSTCP AND STCP In this section, we analyze the effect of synchronize loss on the RTT fairness of HSTCP an STCP. We use a synchronize loss moel where all high-spee flows competing on a bottleneck link experience loss events at the same time. By no means, we claim that this moel characterizes all the aspects of high-spee networks. We use this moel to gain insights into RTT unfairness that we observe in our simulation experiment escribe later. Let w i an RTT i enote the average winow size an the RTT of flow i (i=, 2) respectively. Let t enote the interval between two consecutive loss events of a flow uring steay state. In a network with a uniformly istribute packet loss, ifferent connections, regarless of their RTTs, get the same packet loss rate. But with a synchronize loss moel, ifferent connections may see ifferent packet loss rates. Suppose that the

4 loss events of flow i are uniformly istribute with rate p i. The total number of packets sent by flow i between its two consecutive loss events is /p i. The total number of RTTs between two consecutive loss events is t/rtt i. So, the average winow size can be obtaine as follows. / pi RTTi wi = = () t / RTTi t pi Let R(p) enote the response function of a protocol, which is the average sening rate of the protocol in the unit of packets per RTT, in terms of a loss event rate p. For HSTCP, STCP, an c AIMD, their response functions are of form R( p) =, RTT p where c an are protocol-epenent constants. The value of is between 0.5 an [2]. The value of for AIMD [8], HSTCP [], an STCP [4] is 0.5, 0.82, an, respectively. Therefore, the average sening rate of flow i is calculate as follows. wi c = p i = c / w i RTTi RTTi pi Substitute the above equation into (), we have RTTi RTT i w i = wi = t c / wi t c Therefore the RTT unfairness of the two flows, the ratio of their average throughputs, can be calculate as ( w / RTT )( p ) ( w / RTT )( p ) w / RTT RTT 2 = w RTT RTT / 2 where (-p i ) is approximate by, since p i is usually far less than. Substitute the value of for each protocol, we get the exponent /(-) for AIMD, HSTCP, an STCP, which is 2, 5.56, an, respectively. Table presents a simulation result that shows the banwith shares of two high-spee flows with ifferent ratios of RTTs running in rop tail (RED oes not show significant RTT unfairness). The inverse ratio of RTTs is varie to be from to 6 with base RTT 40 ms. AIMD shows a quaratic increase in the throughput ratio as the inverse RTT ratio increases. STCP shows almost 400 times throughput ratio for inverse RTT ratio 6. HSTCP shows about 00 times throughput ratio for inverse RTT ratio 6. Clearly the results inicate extremely unfair use of banwith by a short RTT flow in rop tail. The result shows better RTT unfairness than preicte by the analysis. This is because while the analysis is base on a completely synchronize loss moel, the simulation may Table : The throughput ratio of two high spee flows over various RTT ratios in 2.5Gbps networks. Inverse RTT Ratio 3 6 AIMD HSTCP STCP Winow Size (packets/rtt)) (Packets) Time (sec) involve loss events that are not synchronize. Also when the winow size is very small, the occurrence of synchronize loss is substantially low. Long RTT flows continue to reuce their winows up to a limit where synchronize loss oes not occur much. Thus, the winow ratio oes not get worse. Figure 3 shows a sample simulation run with ten STCP flows in rop tail. All flows are starte at ifferent times. Eight flows have 80 ms RTT, an two flows have 60 ms RTT. It exhibits a typical case of RTT unfairness. The two longer RTT flows slowly reuce their winow own to almost zero while the other flows merge into a single point. Note that STCP oes not converge in a completely synchronize moel because of Multiplicative Increase an Multiplicative Decrease (MIMD) winow control [20]. However, in this simulation, eight flows with the same RTT o converge (espite high oscillation). This inicates that there exists enough asynchrony in packet loss in our simulation to make the same RTT flows converge, but not enough to correct RTT unfairness. V. RESPONSE FUNCTION The response function of a congestion control protocol can show many characteristics of the protocol incluing RTT fairness, TCP frienliness, an scalability. This section briefly iscusses how these properties can be euce from response functions. Figure 4 raws the response functions of HSTCP, STCP, AIMD, an TCP in a log-log scale. AIMD uses the increase factor 32 an the ecrease factor From the previous section, we showe that for a protocol with a response function c / p where c an are constants an p is a loss event rate, its RTT unfairness coul be moele as /( ) ( RTT RTT ) Fig. 3: 0 scalable TCP flows; 2.4 Gbps, Drop tail. 2 / STCP0 STCP STCP2 STCP3 STCP4 STCP5 STCP6 STCP7 STCP8 STCP9. As increases, the slope of the response function an RTT unfairness increase. That is, the slope of a response function in a log-log scale etermines its RTT unfairness. Since TCP an AIMD have the same slope, the RTT unfairness of AIMD is the same as TCP square RTT unfairness. The RTT unfairness of STCP is infinite while that of HSTCP falls somewhere between TCP s an STCP s. The ability that a protocol utilizes the high amount of

5 Sening Rate R (Packet/RTT) e+6 e+5 e+4 e+3 e+2 More Scalable e+ e-7 e-6 e-5 e-4 e-3 e-2 Loss Event Rate Pevent More TCP Frienly Fig. 4: Response Functions of various protocols. Regular TCP HSTCP Scalable TCP AIMD(32, 0.25) banwith uner high-spee networks is etermine by its high sening rates uner low loss rates. Thus from Figure 4, a protocol becomes more scalable as its sening rate gets higher uner lower loss rates. STCP shows the best scalability followe by HSTCP an AIMD. The TCP frienliness of a protocol is inicate by the point where the response function of the protocol crosses that of TCP. This is because high-spee congestion control protocols act the same as stanar TCP over a higher loss rate than the cross point. Uner lower loss rates below this point, the protocols run their own scalable schemes. Uner this strategy, it becomes more TCP frienly if a protocol crosses TCP as low a loss rate as possible (to the left of the X axis) since the protocol follows TCP below that point. However, moving the cross point to the left while maintaining the scalability increases the slope of the response function, hurting RTT fairness. An ieal protocol woul be that () its response function crosses TCP s as low a rate as possible, an at the same time (2) uner lower loss rates, its slope is as close as that of AIMD. Note that the function nee not be a straight line, i.e., the slope can vary epening on loss rates. But its slope at any loss rates shoul not excee that of STCP although it can be much more forgiving uner a high loss rate where winow size is small enough to avoi frequent synchronize loss. VI. BINARY INCREASE CONGESTION CONTROL It is challenging to esign a protocol that can satisfy all three criteria: RTT fairness, TCP frienliness, an scalability. As allue earlier, these criteria nee not be satisfie simultaneously for all loss rates (for instance, it oes not make sense to be TCP frienly for low loss rates). A protocol shoul aapt its winow control epening on the size of winows. Below, we present such a protocol, calle Binary Increase Congestion Control (). consists of two parts: binary search increase an aitive increase. Binary search increase: We view congestion control as a searching problem in which the system gives yes/no feeback through packet loss as to whether the current sening rate (or winow) is larger than the network capacity. The starting points for this search are the current minimum winow size W min an maximum winow size W max. Usually, W max is the winow size just before the last fast recovery (i.e. where the last packet loss occurre), an W min is the winow size just after the fast recovery. The algorithm repeately computes the mipoint between W max an W min, sets the current winow size to the mipoint; an checks for feeback, in the form of packet losses. Base on this feeback, the mipoint is taken as the new W max if there is a packet loss, an as the new W min if not. The process repeats, until the ifference between W max an W min falls below a preset threshol, calle the minimum increment (S min ). This technique, which we call binary search increase, allows banwith probing to be more aggressive initially when the ifference from the current winow size to the target winow size is large, an become less aggressive as the current winow size gets closer to the target winow size. A unique feature of the protocol is that its increase function is logarithmic; it reuces its increase rate, as the winow size gets closer to the saturation point. The other scalable protocols ten to increase their rates at the saturation point so that the increment at the saturation point is the maximum in the current epoch (efine to be a perio between two consecutive loss events). Typically, the number of lost packets is proportional to the size of the last increment before the loss. Thus binary search increase can reuce packet loss. As we shall see, the main benefit of binary search is that it gives a concave response function, which meshes well with that of aitive increase escribe below. We iscuss the response function of in Section VI-B. Aitive Increase: In orer to ensure faster convergence an RTT-fairness, we combine binary search increase with an aitive increase strategy. When the istance to the mipoint from the current minimum is too large, increasing the winow size irectly to that mipoint may a too much stress to the network. When the istance from the current winow size to the target in binary search increase is larger than a prescribe maximum step, calle the maximum increment (S max ), we increase winow size by S max until the istance becomes less than S max, at which time winow increases irectly to the target. Thus, after a large winow reuction, the strategy initially increases the winow linearly, an then increases logarithmically. We call this combination of binary search increase an aitive increase binary increase. Combine with a multiplicative ecrease strategy, binary increase becomes close to pure aitive increase uner large winows. This is because a larger winow results in a larger reuction in multiplicative ecrease, an therefore, a longer aitive increase perio. When the winow size is small, it becomes close to pure binary search increase a shorter aitive increase perio. Slow Start: When the current winow size grows past the current maximum winow W max, the binary search algorithm switches to probing the new maximum winow, which is unknown (i.e., the winow size where loss can occur is unknown). We run a slow start strategy to probe for a new maximum, once the current winow size is greater than W max, but smaller than W max +S max. The congestion winow increases in each RTT

6 roun in steps W max +S min, W max +2*S min, W max +4*S min,..., W max +S max. The rationale is that the current winow size is likely to be at the saturation point because the last losses occur aroun the current winow size. probes for available banwith in a slow start, until it is safe to increase the winow by S max. After slow start, switches to aitive increase with fixe increment S max to probe a new W max. Fast convergence: Uner a completely synchronize loss moel, binary search increase combine with multiplicative ecrease converges to a fair share [8]. Suppose there are two flows with ifferent winow sizes, but with the same RTT. Since the larger winow reuces more in multiplicative ecrease (with a fixe factor β), the time to reach the target is longer for a larger winow. However, its convergence time can be very long. Assuming infinitely large S max, binary search increase takes log(βw max )-log(s min ) RTT rouns to reach the maximum winow after a winow reuction of βw max. Since winow size increases logarithmically, both larger an smaller winows can return to their respective maxima very fast almost at the same time (although the smaller winow flow gets to its maximum slightly faster). Thus, the smaller winow flow ens up taking away only a small amount of banwith from the larger flow before the next winow reuction, prolonging the convergence time. To remey this behavior, we moify the binary search increase as follows. In binary search increase, after a winow reuction, new maximum an minimum are set. Suppose these values are W max,i an W min,i for flow i (i=, 2). If the new W max,i is less than the previous, this winow is in a ownwar tren (so likely to have a winow larger than the fair share). Then, we reajust the new maximum to be the same as the new target winow which is the mipoint (i.e., W max,i =( W max,i + W min,i )/2), an then reajust the target an apply binary search. This has an effect of reucing the increase rate of the larger winow which allows the smaller winow to catch up. We call this strategy fast convergence. Figure 5 shows a sample run of two flows. Their operating moes are marke by circles an arrows. Winow Winow Size (packets/rtt) (Packets) Aitive Increase Binary Increase, Drop Tail Binary Search Increase Fast convergence Slow start Time (sec) Fig. 5: in working. BI-TCP0 BI-TCP flows A. Protocol Implementation Below, we present the pseuo-coe of implemente as a moification of TCP-SACK. The following preset parameters are use: low_winow: if the winow size is larger than this threshol, engages; otherwise normal TCP increase/ecrease. S max : the maximum increment. S min : the minimum increment. β: multiplicative winow ecrease factor. The following variables are use: W max : the maximum winow size cwn: congestion winow size; bic_inc: winow increment per RTT When entering faster recovery: if (cwn < low_winow){ //normal TCP cwn = cwn *0.5; return; } if (cwn < W max ) //fast convergence W max = cwn * (2-β) / 2; else W max = cwn; cwn = cwn * (-β);//multiplicative ecrease When not in fast recovery an an acknowlegment for a new packet arrives: if (cwn < low_winow){//normal TCP cwn = cwn + /cwn; return; } if (cwn < W max ) //binary search or aitive increase bic_inc = (W max cwn)/2; else //slow start or aitive increase bic_inc = cwn - W max ; if (bic_inc > S max ) //aitive increase bic_inc = S max ; else if (bic_inc < S min )//binary search increase //or slow start bic_inc = S min ; cwn = cwn + bic_inc/cwn; B. Characteristics of In this section, we analyze the response function an RTT fairness of. An analysis on the convergence an smoothness of the protocol can be foun in [8]. ) Response function of In this section, we present a eterministic analysis on the response function of.

7 We assume that a loss event happens at every /p packets. We efine a congestion epoch to be the time perio between two consecutive loss events. Let W max enote the winow size just before a loss event. After a loss event, the winow size ecreases to W max (-β). switches from aitive increase to binary search increase when the istance from the current winow size to the target winow is less than S max. Since the target winow is the mipoint between W max an the current winow size, it can be sai that switches to binary search increase when the istance from the current winow size to W max is less than 2S max. If the istance between the current winow an W max is less than 2S max, there is no aitive increase. Let N an N 2 be the numbers of RTT rouns of aitive increase an binary search increase, respectively. We have Wmax β N = max( 2, 0) S max Then the total amount of winow increase uring binary search increase can be expresse as W max β-n S max. Assuming that this quantity is ivisible by S min, then N 2 can be obtaine as follows. Wmax β NS max N 2 = log S min In the above equation, 2 correspons to the first an the last RTTs of the binary search increase. During aitive increase, the winow grows linearly with slope /S max. So, the total number of packets uring aitive increase, Y, can be obtaine as follows. Y = W β + W β + N S N (2) ( max ( ) max ( ) ( ) max ) 2 During binary search increase, the winow grows logarithmically. So, the total number of packets uring binary search increase, Y 2, can be expresse as follows. Y 2 = Wmax N 2 2( Wmax β NSmax ) + Smin (3) The total number of RTTs in an epoch is N = N + N 2, an the total number of packets in an epoch is Y = Y + Y 2. Since a loss event happens at every /p packets, Y can be expresse as Y = /p. Using (2) an (3), we may express W max as a function of p. Below, we give the close-form expression of W max for two special cases. First, we assume that W max β > 2S max, an W max β is ivisible by S max. Then N = W max β/s max -2. Now we can get 2 b + b + 4a c + p Wmax = (4) 2a where a = β (2-β)/(2S max ), b = log 2 (S max /S min )+(2-β)/2, an c = S max - S min. The average sening rate, R, is then, ( 2 β ) Y RTT p R = = (5) N RTT 2 β ( ) ( 2 β ) b + 4a c + + β b + p 2 In case that W max β >> 2S max, for a fixe S min, N >>N 2. Therefore, the sening rate of mainly epens on the linear increase part, an for small values of p, the sening rate can be approximate as follows: S max 2 β R when W max β >> S max (6) RTT 2 β p Note that for a very large winow, the sening rate becomes inepenent of S min. Eqn. (6) is very similar to the response function of AIMD [8] enote as follows. R α 2 β AIMD RTT 2 β p For a very large winow, the sening rate of is close to the sening rate of AIMD with increase parameter α = S max Next, we consier the case when W max β 2S max, then N =0, Assuming /p>>s min, we get W max as follows. Wmax W max β log + ( ) p 2 S 2 β min By solving the above equation using function LambertW(y) x [2], which is the only real solution of x e = y, we can get a close-form expression for W max. ln( 2) Wmax = 2 ln ( ) e ( 2) β LambertW 4 ln 2 β p ps min then, Y R = N RTT RTT = w log 2 S max min p Wmax β RTT + 2 log 2 2β Wmax β + 2 S min When 2β <<log(w max β / S min )+2, Wmax R (7) RTT Note that when W max β 2S max, the sening rate becomes inepenent of S max. In summary, the sening rate of is proportional to /p, with /2 < <. As the winow size increases, ecreases from to /2. For a fixe β, when the winow size is small, the sening rate is a function of S min an when the winow size is large, a function of S max. Our objective is that when winow is small, the protocol is TCP-frienly; an when the winow is large, it is more RTT fair an gives a higher sening rate than TCP. We can now achieve this objective by ajusting S min an S max. Before we give etails on how to set these parameters, let us examine the RTT fairness of. 2) RTT unfairness of As in Section IV, we consier the RTT unfairness of a protocol uner the synchronize loss moel. Recall that the RTT

8 unfairness of a protocol with a response function /( ) ( RTT RTT ) 2 / RTT From the previous section, we know that as the winow size increases, the value of of ecreases from to /2. That is, with a high banwith, has the same RTT unfairness as the TCP; while with a low banwith, has the same RTT unfairness as STCP. However, since uner small winows, synchronize loss is less frequent, we believe that this unfairness can be manage to be low. We verify this in Section VII. c p is Sening Rate R (Packet/RTT) e+7 e+6 e+5 e+4 e+3 e+2 S max Smin=0.0 beta=0.25 Linear Growth Regular TCP with Smax=8 with Smax=6 with Smax=32 with Smax=64 Logarithmic Growth 3) Setting the parameters In this section, we iscuss a guieline to etermine the preset parameters of : β, S min, S max an low_winow in Section VI-A. From (6) an (7), we observe that reucing β increases the sening rate. Reucing β also improves utilization. However it hurts convergence since larger winow flows give up their banwith slowly. From the equations, we can infer that β has a much less impact on the sening rate than S max. So it is easier to fix β an then ajust S min an S max. We choose 0.25 for β. Uner steay state, this can give approximately 94% utilization of the network. STCP chooses the same value for β. For a fixe β = 0.25, we plot the response function as we vary S max an S min. Figure 6 shows the response function for ifferent values of S max. As S max increases, the sening rate increases only for low loss rates using e-4 as a pivot. S max allows us to control the scalability of for large winows. We cannot increase S max arbitrarily high since it effectively increases the area of RTT unfairness (the area where the slope is larger than TCP s). Recall that when W max β 2S max, the protocol is less RTT fair. This area nees to be kept small to reuce the RTT unfairness of the protocol. Figure 7 plots the response function for various values of S min. As we reuce S min, the sening rate reuces aroun high loss rates (i.e., small winows) an the cross point between TCP an moves towar a lower loss rate. Since we can set low_winow to the winow size corresponing to the cross point, reucing S min improves TCP frienliness. However, we cannot reuce S min arbitrarily low because it makes the slope of the response function steeper before merging into the linear growth area, worsening RTT unfairness. HSTCP crosses TCP at winow size of 3, an STCP at 6. For a fixe β = 0.25, we plot the response function of with S max = 32 an S min = 0.0 in Figure 8. For comparison, we plot those of AIMD (α = 32, β = 0.25), HSTCP, STCP, an TCP. We observe that crosses TCP aroun p = e-2 an it also meets AIMD aroun p=e-5 an stays with AIMD. Clearly, sets an upper boun on TCP frienliness since the response functions of an TCP run in parallel after some point (at p=e-5). s TCP-frienliness uner high loss rates is comparable to STCP s, but less than HSTCP s. crosses TCP at the winow size of 4 (lower than STCP an HSTCP). The sening rate of over extremely low loss rates (e-8) is less than that of STCP an HSTCP. Sening Rate R (Packet/RTT) Sening Rate R (Packet/RTT) e+ e-8 e-7 e-6 e-5 e-4 e-3 e-2 e- e+7 e+6 e+5 e+4 e+3 e+2 e+ e+6 e+5 e+4 e+3 e+2 Loss Event Rate P Fig. 6: Response functions of for ifferent values of S max. Smax=32 beta=0.25 S min e-8 e-7 e-6 e-5 e-4 e-3 e-2 e- Loss Event Rate P Regular TCP with Smin= with Smin=0. with Smin=0.0 with Smin=0.00 Fig 7: Response functions of for ifferent values of S min. e+ e-7 e-6 e-5 e-4 e-3 e-2 Loss Event Rate Pevent Regular TCP HSTCP Scalable TCP AIMD(32, 0.25) (32, 0.25, 0.0) Fig. 8: Response functions of various protocols

9 VII. SIMULATION STUDY In this section, we compare the performance of using simulation with that of HSTCP, STCP, an AIMD. Every experiment uses the same simulation setup escribe in Section II. Unless explicitly state, the same amount of backgroun traffic is use for all the experiments. In orer to reuce noise in ata sampling, we take measurement only in the secon half of each run. We evaluate, AIMD, HSTCP, an STCP for the following properties: banwith utilization, TCP frienliness, RTT unfairness, an convergence to fairness. For, we use S max =32, S min =0.0, an β = 0.25, an for AIMD, α = 32 an β = All high-spee protocols are implemente by moifying TCP/SACK. Utilization: In orer to etermine whether a high-spee protocol uses available banwith effectively uner high-banwith environments, we measure banwith utilization for 2.5 Gbps bottleneck. Each test consists of two high-spee flows of the same type an two long-live TCP flows. In each test, we measure the total utilization of all the flows incluing backgroun traffic. In rop tail, all protocols give approximately 97% utilization, an in RED, AIMD an consume about 95% of the total banwith while HSTCP an STCP consume about 92%. The rop tail experiment consumes more banwith because rop tail allows flows to fill up the network buffers. Note buffers are part of the capacity. RTT Fairness: In this experiment, two high-spee flows with a ifferent RTT share the bottleneck. The RTT of flow is 40 ms. We vary the RTT of flow 2 among 40ms, 20ms, an 240ms. The bottleneck link elay is 0ms. We run two groups of simulation, each with ifferent values of bottleneck banwith: 2.5 Gbps, an 00 Mbps. This setup allows the protocols to be teste for RTT fairness for ifferent winow sizes. Accoring to our analysis in Section VI, aroun small winow sizes, shows the worst RTT unfairness. has winow sizes about 7000 (loss rate ) for 2.5Gbps an 300 (loss rate 0.004) for 00Mbps. Tables 2 an 3 show the results for the runs in 00Mbps an 2.5Gbps, respectively. We show the results of rop tail only. The RTT unfairness uner RED is close to the inverse of RTT ratio for every protocol. We omit the RED figures. It can be seen that the RTT unfairness of is relatively comparable to AIMD. This result verifies our analysis in Section VI. In Table 2, the performance of oes not eteriorate much while HSTCP an STCP have improve RTT unfairness. This is because in 00 Mbps, the winow size of all the flows is much smaller than in 2.5Gbps run. Therefore, the egree of synchronize loss is very low. Although the RTT unfairness of gets worse aroun this winow size by the analysis, this eficiency gets compensate by lack of synchronize loss so that it oes not have much performance egraation. Overall, its RTT unfairness is much better than HSTCP an STCP. HSTCP an STCP ten to starve long RTT flows uner high banwith environments. Table 2: The throughput ratio of protocols uner 00 Mbps Inverse RTT Ratio 3 6 AIMD HSTCP STCP Table 3: The throughput ratio of protocols uner 2.5Gbps Inverse RTT Ratio 3 6 AIMD HSTCP STCP TCP-frienliness: We run four groups of tests, each with ifferent bottleneck banwith. Each group consists of four inepenent runs, each with a ifferent type of high-spee flows. In every run, the same number of network flows (incluing backgroun) is use. Figure 9 shows the percentage of banwith share by each flow type uner rop tail. (RED gives approximately similar results as rop tail.) Three flow types are present: backgroun (web traffic an small TCP flows), long-live stanar TCP/SACK flows, an high-spee flows. Uner banwith below 500Mbps, the TCP frienliness of is comparable to that of STCP. At 20Mbps, long-live TCP an backgroun flows with flows consume slightly less banwith than those with HSTCP an STCP. However, in HSTCP an STCP simulations, the unuse banwith is slightly more than that in. That is, while consumes more banwith than HSTCP an STCP, it takes banwith not only from TCP, but also from unuse banwith. However, AIMD is too aggressive, an it takes too much banwith from TCP. For 500Mbps an 2.5Gbps, the amount of shares by backgroun an long-live TCP flows substantially reuce ue to TCP s limitation to scale its banwith usage in high-banwith. Uner 500Mbps, STCP,, an AIMD use approximately the same share of banwith. Uner 2.5Gbps, the banwith share of backgroun TCP traffic is very small. STCP becomes most aggressive. becomes frienlier to TCP. To sum, achieves goo TCP fairness for all winow sizes. Aroun high loss rates where TCP performs well, its TCP frienliness is comparable to STCP s. The result closely follows our analysis in Section VI-B. Fairness: Synchronize loss has impact also on banwith fairness an convergence time to the fair banwith share. In this experiment, we run 4 high-spee flows with RTT 00ms. Two flows start earlier ranomly in [0:60] secons, an the other two flows start later ranomly in [00:60] secons. The total simulation time is 600 secons. The bottleneck link banwith is 2.5Gbps. For this experiment, we measure the fairness inex [20] at each 50-secon interval, an we take samples only after the first 00 secons. This result gives an inication on () how

10 fast it converges to a fair share, an (2) even after convergence to a fair share, how much it oscillates aroun the fair share. Figure 0 shows the result for rop tail routers. At time aroun 50 secons, the secon set of two connections just start while the first two connections consuming almost all the banwith. Therefore, the ifference between these four connections is the largest, an hence the fairness inex is the lowest. As the latter two connections get more banwith, the fairness inex increases. The protocol whose fairness inex reaches the fastest has the fastest convergence spee. an AIMD give the best results, an they quickly converge to the fair share (where the fairness inex is close to ). STCP is the worst, an even after 600 secons, the fairness inex is still below We observe that the fairness inex of all protocols first increases fast, an then slowly reaches to. The reason is that the congestion winows of the latter two connections first increase exponentially in the slow start state; after the first loss event, they enter the congestion avoiance state, an increase slowly as specifie by each protocol. Figure shows the result for RED routers. We observe much better convergence for HSTCP an STCP than in rop tail. This is because RED oes not incur as much synchronize loss as rop tail. However, after convergence, HSTCP with RED shows a larger oscillation aroun the fair share than HSTCP with rop tail, an also than the other three protocols with RED. We note that the fairness inex of HSTCP increases earlier than others in Figure. The reason is the latter two connections in HSTCP simulation start earlier than those in other simulations, since the simulation script ranomly generates the starting time of each connection. VIII. RELATED WORK As HSTCP an STCP have been iscusse in etail in this paper, we examine other high-spee protocols that are not covere by this paper. Recent experiment [9] inicates that TCP can provie goo utilization even uner 0Gbps when the network is provisione with a large buffer an rop tail. However, the queue size of high-spee routers is very expensive an often limite to less than 20% of the banwith an elay proucts. Thus, generally, TCP is not suitable for applications requiring high banwith. FAST [7] moifies TCP-Vegas to provie a stable protocol for high-spee networks. It was proven that TCP can be instable as elay an network banwith increase. Using elay as an aitional cue to ajust the winow, the protocol is shown to give very high utilization of network banwith an stability. It fairness property is still uner investigation. XCP [5] is a router-assiste protocol. It gives excellent fairness, responsiveness, an high utilization. However, since it requires XCP routers to be eploye, it cannot be incrementally eploye. IX. CONCLUSION The significance of this paper is twofol. First, it presents RTT fairness as an important safety conition for high-spee congestion control an raise an issue that existing protocols may have a severe problem in eployment ue to lack of RTT Fairness Inex Fairness Inex 00% 80% 60% 40% 20% 0% Long-Live TCP Highspee Backgroun Unuse 20M 00M 500M 2500M AIM D H S T C P STC P AIM D H S T C P STC P AIM D H S T C P STC P AIM D H S T C P STC P Fig 9: TCP frienliness for various banwith networks (DROPTAIL) AIMD HSTCP STCP Time (secon) Fig. 0: Fairness inex over various time scales (DROP TAIL) 0.6 AIMD HSTCP STCP Time (secon) Fig. : Fairness inex over various time scales (RED)

11 fairness uner rop tail. RTT fairness has been largely ignore in esigning high-spee congestion control. Secon, this paper presents a new protocol that can support RTT fairness, TCP frienliness, an scalability. Our performance stuy inicates that it gives goo performance on all three metrics. We note that the response function of may not be the only one that can satisfy the three constraints. It is possible that there exists a better function that utilizes the traeoff among the three conitions. This is an area of more research. Another point of iscussion is that high-spee networks can greatly benefit from the eployment of AQM. Our work supports this case since a well-esigne AQM can relieve the protocol from the buren of various fairness constraints cause by synchronize loss. One possible limitation of is that as the loss rate reuces further below e-8, its sening rate oes not grow as fast as HSTCP an STCP. This is because of the lower slope of its response function. Hence, it may seem that there is a funamental traeoff between RTT fairness an scalability. We argue it is not the case. Uner such a low loss rate, most of loss is ue to signal or self-inflicte, i.e., loss is create because of the aggressiveness of the protocol. As a protocol gets more aggressive, it creates more loss an thus, nees to sen at a higher rate for a given loss rate. The utilization of the network capacity is more important uner such low loss rates, which is etermine mostly by the ecrease factor of congestion control. For TCP, the utilization is 75% an for STCP an, aroun 94% (these numbers are etermine from β). In aition, we believe that by the time that much higher banwith (in the orer of 00 Gbps) becomes available, the network must have more avance AQM schemes eploye so that we can use a higher slope response function. Then again, a less aggressive (i.e., lower slope) protocol, however, is less responsive to available banwith; so short file transfers will suffer. There are many facets to the problems, requiring more research efforts. We believe that fast convergence to efficiency requires a separate mechanism that etects the availability of unuse banwith which has been an active research area lately ([22, 23]). We foresee that avance in this fiel greatly benefits the congestion control research. [8] L. Xu, K. Harfoush, an I. Rhee, Binary increase congestion control for fast long-istance networks, Tech. Report, Computer Science Department, NC State University, 2004 [9] S. Ravot, "TCP transfers over high latency/banwith networks & Gri DT", Presente at First International Workshop on Protocols for Fast Long-Distance Networks (PFLDnet 2003), February 3-4, 2003, CERN, Geneva, Switzerlan [0] T. Dunigan, M. Mathis, an B. Tierney, A TCP tuning aemon, in Proceeings of SuperComputing: High-Performance Networking an Computing, Nov [] M. Jain, R. Prasa, C. Dovrolis, Socket buffer auto-sizing for maximum TCP throughput, Submitte for publication, 2003 [2] J. Semke, J. Mahavi, an M. Mathis, Automatic TCP buffer tuning, in Proceeings of ACM SIGCOMM, Aug. 998 [3] W. Allcock, J. Bester, J. Bresnahan, A. Chervenak, L. Liming, S. Meer, S. Tuecke., GriFTP protocol specification. GGF GriFTP Working Group Document, September [4] PFTP : [5] H. Sivakumar, S. Bailey, an R. L. Grossman, PSockets: The case for application-level network striping for ata intensive applications using high spee wie area networks, in Proceeings of SuperComputing: High-Performance Networking an Computing, Nov 2000 [6] D. Bansal, H. Balakrishnan, S. Floy, an S. Shenker, "Dynamic behavior of slowly responsive congestion controls", In Proceeings of SIGCOMM 200, San Diego, California. [7] S. Floy, an E. Kohler, Internet research nees better moels, October, 2002 [8] S. Floy, M. Hanley, an J. Pahye, A comparison of equation-base an AIMD congestion control, May 2000 [9] T. V. Lakshman, an U. Mahow, The performance of TCP/IP for networks with high banwith-elay proucts an ranom loss, IEEE/ACM Transactions on Networking, vol. 5 no 3, pp , July 997 [20] D. Chiu, an R. Jain, Analysis of the increase an ecrease algorithms for congestion avoiance in computer networks, Journal of Computer Networks an ISDN, Vol. 7, No., June 989, pp. -4 [2] R. M. Corless, G. H. Gonnet, D.E.G. Hare, D. J. Jeffrey, an Knuth, "On the LambertW function", Avances in Computational Mathematics 5(4): , 996 [22] C. Dovrolis, an M.Jain, ``En-to-en available banwith: measurement methoology, ynamics, an relation with TCP throughput''. In Proceeings of ACM SIGCOMM 2002, August [23] K. Harfoush, A. Bestavros, an J. Byers, "Measuring bottleneck banwith of targete path segments", In Proceeings of IEEE INFOCOM '03, April REFERENCES [] S. Floy, S. Ratnasamy, an S. Shenker, Moifying TCP s congestyion control for high spees, May 2002 [2] S. Floy, HighSpee TCP for large congestion winows, IETF, INTERNET DRAFT, raft-floy-tcp-highspee-0.txt, 2003 [3] S. Floy, Limite slow-start for TCP with large congestion winows, IETF, INTERNET DRAFT, raft-floy-tcp-slowstart-0.txt, 200 [4] T. Kelly, Scalable TCP: Improving performance in highspee wie area networks, Submitte for publication, December 2002 [5] Dina Katabi, M. Hanley, an C. Rohrs, "Internet congestion control for high banwith-elay prouct networks." ACM SIGCOMM 2002, Pittsburgh, August, 2002 [6] Y. Gu, X. Hong, M. Mazzucco, an R. L. Grossman, SABUL: A high performance ata transport protocol, Submitte for publication, 2002 [7] C. Jin, D. Wei, S. H. Low, G. Buhrmaster, J. Bunn, D. H. Choe, R. L. A. Cottrell, J. C. Doyle, H. Newman, F. Paganini, S. Ravot, an S. Singh, "FAST Kernel: Backgroun theory an experimental results". Presente at the First International Workshop on Protocols for Fast Long-Distance Networks (PFLDnet 2003), February 3-4, 2003, CERN, Geneva, Switzerlan