RTTometer: Measuring Path Minimum RTT with Confidence
Amgad Zeitoun
Zhiheng Wang
Sugih Jamin
Department of Electrical Engineering and Computer Science,
The University of Michigan, Ann Arbor
{azeitoun,zhihengw,jamin}@eecs.umich.edu
Abstract— Internet path delay is a substantial metric in determining path quality. Therefore, it is not surprising that RTT
plays a tangible role in several protocols and applications, such as
overlay network construction protocol, peer-to-peer services, and
proximity-based server redirection. Unfortunately, current RTT
measurement tools report delay without any further insight about
path condition. Therefore, applications usually estimate minimum
RTT by sending a large number of probes to gain confidence in
the measured RTT. Nevertheless, large number of probes does not
directly translate to better confidence. Usually, minimum RTT
can be measured using few probes. Based on observations of
path RTT presented in [1], we develop a set of techniques not
only to measure minimum path RTT but also to associate it with
confidence level that reveals the condition on the path during
measurement. Our tool, called RTTometer, is able to provide
confidence measure associated with path RTT. Besides, given a
required confidence level, RTTometer dynamically adjusts the
number of probes based on path’s condition. In this paper, we
describe our techniques implemented in RTTometer and present
our preliminary experiences using RTTometer to estimate RTT
on various representative paths of the Internet.
I. I NTRODUCTION
Internet path delay is one of the substantial metrics in
determining path quality. Path delay, either one-way time or
round-trip time (RTT), is often used to indicate “distance”
between hosts and estimate the proximity of Internet hosts. It
may also indicate the expected performance of an application
that uses network as a transport medium. For example, the
throughput of TCP (Transport Control Protocol) is partly
dependent on the round-trip time between communicating
hosts [2], [3]. Therefore, shorter RTT usually indicates smaller
service time, i.e., faster transfer rate. Furthermore, Internet
RTT is highly correlated with geographical distance; in fact,
GeoPing [4] exploits the correlation between geographic
distance and RTT to determine IP’s geographic location. It
maps an IP address into a geographic location by measuring
RTT from multiple points on the Internet.
Even though the primary goal of Internet architecture is
not to facilitate performance measurements between endhosts [5], several tools to measure RTT exploit features in
the IP protocol. One of the common tools to measure path
RTT is ping. The tool works by sending an ICMP (Internet
Control Message Protocol) packet, usually called a probe,
which forces end-host to elicit a reply message. The RTT,
then, is the elapsed time between the sending of the ICMP
packet and the recipient of the reply. The main goals of ping
are two folds. First, it is often used by operators to determine
host reachability. Second, it provides a way to observe the
dynamics of the RTTs along a path to determine any path
anomalies. Besides, ping reports the RTT of each probe and
it briefly provides statistics on probes (e.g., minimum, average,
and maximum RTTs) as well as probe losses. Due to the
preferential treatment of ICMP packets by different routers
and hosts, other flavors of ping-like tools have emerged. All
these tools, however, follow the same philosophy of the ping
tool in measuring and reporting delays with minimal statistics.
Services that use RTT probes as a regular routine to enhance
their performance [6], [7], [8], [9], [10], [4], such as, service
mirroring, distributed games, peer-to-peer applications, and
measurement infrastructures have been proliferated nowaday.
These services and applications usually require estimating
minimum path RTT between hosts. The accuracy, or, confidence, of the measured path RTT is critical to their operational
performance. To achieve certain level of confidence in their
measured path RTT, these services usually send a large number
of probes. For example, some services even send 220 probes
to estimate minimum RTT! These services, however, select
the value of a single probe among all these probes (e.g.,
minimum RTT) to reflect the property of the path. They often
do not make use of the information provided by the remaining
probes. Therefore, the main motivation to justify sending these
large number of probes is to add confidence to the captured
minimum RTT. Even with these large number of probes, no
service has shown the confidence in the captured minimum
RTT or the justification for the number of probes chosen.
In this paper, we present RTTometer, a tool to estimate
the minimum RTT of a path with an associated confidence
measure. RTTometer provides the same functionality of the
ping and other ping-like tools, but it differs in several ways.
First, RTTometer uses TCP probes instead of ICMP probes.
Second, other than the basic statistics for probes, it provides
a measure of accuracy, i.e., confidence, associated with the
minimum RTT observed along the path. The associated confidence takes into account the condition on the path during
RTT measurement (e.g., congestion and route change); therefore, it uses the information provided by all probes. Finally,
RTTometer dynamically determines the number of probes
required to capture the minimum RTT with a given accuracy.
An application can determine the tolerable error range and
the confidence measure according to its requirements. Our
philosophy comes from the belief that there is no general
solution that fits all applications, but rather, each application
can determine the best number of probes required to achieve
a given accuracy in the measured RTT. Even for the same
path the required number of probes needed to capture RTT
may change according to the path’s condition, e.g., congestion
level.
RTTometer is potentially useful for applications that rely
on measuring path RTT as a regular routine of their operation.
Specifically, these applications that depend on measuring minimum RTT of a path. The minimum path RTT is a property
related to the topology of the network rather than the load of
the network, hence it is of most interest to applications and
services that are sensitive to network topology. Potential use
of RTTometer is in the field of overlay network such as
Content-Addressable Network (CAN) [6], peer-to-peer applications such as Napster [17], various multiplayer game servers
such as Gamespy [18], end-to-end Internet distance estimation
services such as GNP [8] and IDMaps [9], geographical IP
mapping services such as GeoPing [4], or end-host multicast
applications such as Narada [10] and HMTP [7]. All these services and applications rely on measured RTT as a fundamental
tool to determine the location of an Internet host relative to
one or more measurement points. Inaccurate measured RTT
would result in performance degradation in these applications.
The rest of this paper is organized as follows. In Section II,
we present the set of measurement concepts and techniques
used in RTTometer. We, then, describe the design and
implementation of our tool in Section III. We present some
preliminary experiences studying a large number of paths
on the Internet and provide general observations in Section
IV. We briefly summarize several related work in Section V
Finally, the paper concludes in Section VI with final remarks.
II. RTTOMETER C ONCEPT
The concept of RTTometer tool is based on observations
developed in [1], where authors identified several challenges in
capturing the minimum RTT. This paper, however, addresses
the issue of implementing a tool for operational networks. It
provides a better approach in calculating the confidence of the
measured minimum RTT, and empirically tests this tool on a
large number of paths on the Internet. Estimating the minimum
RTT along a path requires sending a number of probes, K,
with inter-probe interval of τ time units. The probing period,
thus, lasts for L time units, where L = Kτ . There is a set
of challenges that face the process of measuring minimum
path RTT, namely, self-interference, persistent congestion, and
route changes. Self-interference occurs when the inter-probe
interval τ is too small, the rate at which probe packets are sent
may be high that they queue up behind each other. Persistent
congestion makes all probes sent during the measurement
period L experience queueing delays or some losses; the
captured RTT, then, would not represent the minimum RTT
of the path. Finally, route change is another factor that may
change the value of the measured RTT and, hence, can be
misinterpreted as persistent congestion.
In [1], authors studied the effect of each controllable parameter, K, τ , and L on the accuracy of the measured RTT.
For example, K determines the amount of traffic required by
II
III
e
I
II
e
Fig. 1.
An example of a phase-plot showing different congestion regions.
the measurement process, and the choice of τ can avoid selfinterference. In addition, they developed a technique to detect
inaccuracies in the measured RTT due to congestion and route
changes (uncontrollable factors). So, not only the minimum
RTT of a path is measured but also a confidence level, which
reflects the congestion condition along the path, is provided
to indicate how strong is the estimated minimum RTT.
A. Phase Plot
RTTometer’s main mechanism depends on the use of the
phase-plot technique described in [11]. Given a set of RTT
measurements, the phase-plot shows the correlation between
two adjacent measurements. In a phase-plot, the x-axis ticks
represent RT Tn and the y-axis ticks represent RT Tn+1 , where
n and n+1 are indices of consecutive probes. Fig. 1 shows
an example of a phase-plot. For probes that see an empty
queue, the experienced delay for each probe is RT Tn+1 =
RT Tn = p + ε, where p is the propagation delay of the
path and ε represents some random delay due to media access
contention, router processing overhead, ARP resolution, etc.
We discuss how to estimate ε in Section II-B. If there are
no queueing delays experienced by the probes then delays
are clustered around the point of value p1 at the bottom left
corner of the phase-plot. When a probe packet experiences
queueing delay, the corresponding data point in the phase-plot
deviates from p. Self-interference happens when inter-probe
interval is small such that probes start to queue behind each
other [11]. If two or more adjacent probes are queued behind
each other, then RT Tn+1 = RT Tn − τ , where τ is the interprobe interval. With the concept of laying points on the phaseplot, RTTometer uses techniques described in [1] to detect
self-interference.
B. Detecting Congestion and Route Change
Other than self-interference, RTTometer needs to detect
the condition of the network during measurement, mainly congestion and route changes. Minimum path RTT may change
due to congestion or route change conditions on the measured
path. Different applications may react differently when the
change in minimum RTT is due to one or the other of
1 Transmission
delay is negligible due to small probe size.
these causes. Therefore, RTTometer provides a confidence
level associated with the minimum RTT; based on that the
application decides according to its requirements.
The notion of congestion regions and congestion components are introduced in the phase-plot to capture network
dynamics during RTT measurement. In a phase-plot of a
given measurement period there are three congestion regions
as shown in Fig. 1:
a) Region I contains probe pairs that see empty queues and
experience minimum RTT plus minor random overheads
ε due to media access contention, router processing
overhead, etc.
b) Region III contains probe pairs that always see a queue.
This is the region of persistent congestion.
c) Region II contains probe pairs where one of the probes
experiences queueing delay but the other does not, i.e.,
there is a transition in congestion state between adjacent
probes. This is the region of transient congestion.
The congestion regions are determined based on two values,
the minimum RTT and ε. The minimum RTT determines the
bottom left corner of region I, while ε delimits the boundaries
of regions I, II, and III. The setting of ε accommodates for
a marginal error of the measured minimum RTT. In addition,
on a non-congested path, region I should contain most of the
points, and hence the most frequent value of the measured
RTT. Therefore, ε is equal to the size of a window around the
most frequent values of the RTT observed on a non-congested
path (i.e., the statistical mode). This also implies that the
window size is double the distance between the minimum RTT
and the mode RTT.
Given a trace of RTT measurements over a path, to detect
the congestion level of the measured path, RTTometer
computes the distribution of data points in the three regions of
the phase-plot. Assume that the value of minimum RTT on the
path is minRTT. Point pi = (RT Ti , RT Ti+1 ) is in region I
if RT Ti ≤ (minRT T + ε) and RT Ti+1 ≤ (minRT T +
ε). A point is in region II if M AX(RT Ti , RT Ti+1) >
(minRT T +ε) and M IN (RT Ti, RT Ti+1 ) ≤ (minRT T +ε).
Finally, a point is in region III if RT Ti > (minRT T +ε) and
RT Ti+1 > (minRT T + ε). Given N points in the phase-plot,
the computation of congestion component CI is:
CI =
N
1 X
1
1
×
, where
N p ,i=1 △(RT Ti ) △(RT Ti+1 )
(1)
i
△(x) =
1
T
⌈ x−minRT
⌉
ε
: x = minRT T
: x > minRT T
For all points in region II, the computation of congestion
region CII is:
X
1
1
],
CII = [K −
N
△(M AX(RT Ti, RT Ti+1 ))
pi ∈region II
(2)
where K is the number of points in region II. Finally, for
all points in region III, the computation of congestion region
CIII is:
CIII =
1
[M −
N
X
pi ∈region III
1
1
×
], (3)
△(RT Ti ) △(RT Ti+1)
where M is the number of points in region III.
Since each point in the phase-plot represents two consecutive measurements of path RTT, the calculation of CI , CII ,
and CIII depends on the value of both RTTs for each point.
The closer the RTT value to region I the more weight it has.
Different than the approach followed in [1] in calculating CI ,
instead of assigning weight of 1 to each point in region I
and 0 otherwise, in equation (1) each point in the phase-plot
contributes to the value of CI , the closer the point to region
I the more weight it adds. In the current approach, we divide
the phase-plot into squares, where each square has a height
and width of value ε, and the bottom left square is region I.
Given a value of RTT, function △(·) calculates the distance
between this RTT and minRT T in terms of the number of
squares, e.g., if RT T ≤ (minRT T + ε), the distance is 1.
For each point in the phase-plot, the distances of its RTTs are
used to calculate its weight. The maximum weight a point can
add is 1 (when the point is within region I). The weight of
a point is calculated by multiplying the inverse of its RTTs’
distances. Therefore, a point close to region I has more weight
than a point that is further away. Therefore, CI indicates the
confidence level in the measured RTT (e.g., if all points are
in region I, CI equals to 1) and CIII indicates the condition
on the path during measurements.
A reported minimum RTT along with a large value of CIII
may indicate either persistent congestion or route change.
To distinguish between the two cases, RTTometer provides
the number of lost probes in addition to the reported Cj ’s,
j = I, II, and III. Given the fact that some probes get
lost due to path congestion, an application can determine
whether the change is due to congestion or route change.
That is, applications should consider probe losses to signal
for congestion even if CI is high.
C. Summary
We briefly summarize some of the main conclusions in [1]
regarding the RTT measurement process.
• Inter-probe interval has minimum effect on the accuracy
of the captured minimum RTT. The inter-probe interval
τ can be set to a reasonable value that does not cause
self-interference among probes.
• Given a small error range, i.e., ε, minimum RTT of a
path can be captured for a stationary non-congested path
with a few number of probes.
• To minimize the effect of route changes on the measured
minimum RTT, the choice of the measurement period
L should be small enough that the probability of route
change occurring during this period is small (one minute
or less).
By leveraging the findings in [1], we develop the measurement tool, RTTometer, to estimate the minimum path RTT
with certain confidence level. In the subsequent sections we
present our implementation and experiences of RTTometer.
III. I MPLEMENTATION I SSUES
In this section we outline our implementation for
RTTometer. RTTometer does not require any changes on
the receiver’s side. Similar to ping and other ping-like tools,
it elicits replies from target hosts by exploiting protocol’s
semantics. Due to the limitations and disadvantages of ICMP
probes [12] (mainly, due to security concerns and preferential treatment at routers), RTTometer uses TCP probes to
measure RTT, often called TCP ping. The mechanism of TCP
ping was introduced in previous studies [13], [14], and it has
been evaluated in [1]. To measure RTT between a source and
a target host using TCP ping, the source sends a TCP SYN
packet to the target. The time elapsed until a TCP SYN-ACK
or RST packet returns from the target is the RTT. RTTometer
uses raw socket to send TCP probes and intercepts the replies
through packet sniffing using libpcap [15]. The timestamps of
packets captured by libpcap are used to indicate the send and
receive time of probes and probe replies, respectively. The RTT
is, then, the difference in timestamps between the probe reply
and the probe. In addition, we choose not to suppress the TCP
replies that emerge from the source host’s operating system.
For example, let’s imagine the scenario where RTTometer
sends a SYN request to the target host and the target host
replies with a SYN-ACK. RTTometer captures the reply
packet through libpcap and another copy is delivered to the
operating system’s TCP stack, which subsequently replies with
a RST packet because there are no states for the received
packet. We decided to leave this behavior as it helps to
guarantee that the connection at the target machine is not left
in a half-open state, just a matter of being courteous.
There are two main modes of operation for the
RTTometer. The first mode is used to estimate the value
of ε of a given path. In this mode, RTTometer sends a
number of probes and estimates the value of ε as double
the distance between the minimum and mode RTTs observed
from these measurements. Preferably, the number of probes
is large enough so that the mode RTT is statistically strong.
RTTometer reports the number of probes lost and the values
of Cj , j = I, II, and III, a user should take the lost probes
and the values of congestion regions into consideration when
estimating the value of ε. Estimating ε during congestion
periods may introduce errors in the estimated value. Should
a user experiences congestion while estimating ε (packet loss
and CIII are used to indicate congestion), a user is advised
to repeat the estimation some time later.
The second mode of operation is the minimum RTT estimation of a path. In this mode a user specifies the maximum
number of probes to be sent and may specify the value of ε
and the value of the confidence required (i.e., the threshold
on CI ). The default value of ε is 2, 4, or 6 ms based on
the path minimum RTT (see Section IV-B) with a confidence
level of 0.8. RTTometer starts sending probes and calculating
the value of congestion regions. The probing routine stops
when the required value of confidence has been reached for
a given value of ε, or when the maximum number of probes
is reached. In either case, RTTometer reports the minimum
RTT captured along with the values of CI , CII , and CIII .
In addition, the number of probes lost, 10-percentile, 25percentile, median, mode, 75-percentile, 90-percentile, and the
maximum of the observed RTTs are reported.
In both modes, user can specify the inter-probe interval
(minimum value is 10 ms and the default value is 500 ms).
Besides these parameters, a user can set the timeout value
for probes. If RTTometer does not receive a reply to a
probe within a timeout time units, the probe is considered
lost (default value is 3 seconds). In addition, a user can also
specify the port number of the TCP probes (default is port 80).
Finally, the value of the Time-To-Live (TTL) of the probes can
be specified. This allows RTTometer to measure the RTT to
a given hop along a path without addressing the probes directly
to that node (useful for probing routers along a path).
IV. E XPERIENCES
In this section, we present some preliminary observations
from a broad set of experiments to measure the delay on
different paths. First, we study the characteristics of the
distance between the mode RTT and the minimum RTT of
a path, which is used to determine ε. We then discuss the
issue of estimating ε from the operational point-of-view. Our
main motivation is to understand whether the estimation of
ε needs to be done quite frequently, or there is a reasonable
default value for ε. Finally, we look at the dynamics of the
number of probes required to capture the minimum RTT along
a path with a specific setting of ε and at a given confidence
level.
A. Observations on Mode RTT
In this experiment we study the behavior of path RTT.
Specifically, we are interested in studying the distribution of
the RTT and how close the mode RTT to the minimum RTT
on a given path. We run RTTometer from the University of
Michigan to a set of 5,937 hosts, mostly Web servers. Due
to the diversity of connectivities on the Internet, choosing a
representative set of paths is not a simple task. However, we
make sure that our set of target hosts contains hosts at different continents, connected through different links (high speed
links, DSL, cable modem, etc.), and belonging to different
organizations (educational, commercial, governmental, etc.).
For each target host, we send 100 probes with inter-probe
interval of 500 ms. To better understand and quantify the
behavior of the path RTT, we classify the set of paths into three
groups based on their minimum RTT. The first group (G1)
contains all paths with minimum RTT less than or equal to 50
ms. The second group (G2) contains paths with minimum RTT
between 50 ms and 150 ms. The third group (G3) contains
the remaining set of paths, i.e., larger than 150 ms. Table I
shows the number of paths in each group. Probe losses are
not uniformly distributed among all paths. Instead, some paths
suffer from severe probe losses while others are not, i.e., the
TABLE II
S TATISTICS OF ( MODE RTT - MINIMUM RTT) FOR DIFFERENT GROUPS .
Delay range
≤ 50 ms
50 - 150 ms
> 150 ms
Paths
2403 (40.5%)
2594 (43.7%)
940 (15.8%)
Avg. loss
2.3
2.4
4.2
Percentile
G1
G2
G3
1
10000
0.8
1000
Mode - Minimum (ms)
Cumulative Distribution Function
Group
G1
G2
G3
TABLE I
C LASSIFICATION OF TARGET PATHS .
0.6
0.4
0.2
G1
G2
G3
0
0
5
10
15
20
25
30
35
40
10
0.5
0.5
0.6
25
0.7
0.7
0.9
50
0.9
1
1.4
75
1.3
1.6
5.7
90
2.1
6.3
21.5
100
92
2170
1655
G1
G2
G3
100
10
1
0.1
45
50
0
(Mode - Minimum) (ms)
Fig. 2. CDF of the difference between minimum and mode RTTs on each
path at different groups.
loss distribution has a long tail. For example, for G1, 1,724
paths did not suffer any probe loss while more than 70 probes
were lost on few paths. For G2, 1,823 did not suffer any probe
loss, and similarly 555 paths in G3.
Recall that on the phase-plot ε delimits the set of congestion
regions, we, thus, look at the setting of ε for different paths.
The main question we investigate is whether there is a reasonable default value of ε for different paths. In [1], authors
found that for different paths the third quartile of the difference
between mode RTT and minimum RTT is 1 ms. For each
group, we look at the distance between the minimum RTT and
the mode RTT along each path. Fig. 2 shows the cumulative
fraction of the distance between minimum and mode RTTs
for paths that did not suffer any probe losses in G1, G2,
and G3. The distribution of the difference between mode and
minimum RTTs is long-tail, we only display the distribution
up to 50 ms in the figure. Table II shows statistics about the
distribution in each group. We can see that the difference
between mode and minimum RTTs slightly depends on the
path RTT itself, e.g., even for paths with large RTT (G3) we
can see that 50% of the paths have differences less than 1.4
ms. However, the tail of the distribution gets longer for paths
with large RTT. Since we do not have access to the set of
target hosts, we could not check if the route did not change
during the course of our RTT measurement. 2 Therefore, some
of the large differences between mode RTT and minimum RTT
may be due to route changes. Besides, some of the target hosts
are connected through relatively slow-speed links which have
weird behavior (See IV-D for discussion on behavior of DSL
and cable modem hosts). The result seems intuitive as on paths
2 We could identify some route changes by observing changes in the RTT
values, e.g., on one path, RTT changed from 100 ms to 2 sec after the 20th
probe and stayed on that value till the end of our probing period.
0.05
0.1
0.15
0.2
0.25
Mode Strength
Fig. 3. The distance between minimum RTT and mode RTT and its relation
to the strength of the observed mode RTT.
with large RTT the delay fluctuates more when crossing long
haul links or satellite connections.
We study the relationship between the distance between the
minimum RTT and the mode RTT and the strength of the
mode. The strength of the mode is defined as the fraction of
observations that consist the mode of a path. Fig. 3 depicts
this relationship. The y-axis represents the distance between
minimum RTT and mode RTT in log scale, while x-axis
represents the mode strength. We first note that most of
the samples have less than 5 ms between their minimum
RTTs and mode RTTs (Fig. 2). This suggests a small ε
value. Furthermore, while such distance of some of the paths
falls beyond 5 ms, all these paths have weak mode strength.
This observation suggests that during ε estimation mode, the
stronger the mode RTT is the more accurate is the estimation
of ε.
B. Dynamics of ε
As pointed out earlier, ε of a path is important in delimiting
different congestion regions. We have shown in the previous
subsection that the distance between mode and minimum RTT
on a given path may slightly depend on the minimum RTT of
the path. Since ε is double the difference between mode and
minimum RTTs, we investigate the stability of ε over time. The
main motivation is to study whether an estimated ε at a given
measurement period can be used at subsequent measurements
as well. Can we choose a default value for ε that would suite
different paths?
We conduct an experiment where we pick 5 representative
paths from each group. The minimum RTTs for the paths
selected from G1 are 5 ms (path 1), 15 ms (path 2), 25
ms (path 3), 35 ms (path 4), and 45 ms (path 5). For G2,
we pick paths with minimum RTTs of 55 ms (path 6), 75
4
G1
G2
G3
Mode - Minimun (ms)
3.5
3
2.5
2
1.5
1
0.5
0
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Path
Fig. 4. The dynamics of (mode RTT - minimum RTT) on 5 paths in each
group. Paths 1-5 are from G1, paths 6-10 are from G2, and paths 11-15 are
from G3. The solid line represents the interquartile range where the point
is the median, and the dotted line represents the range from 10%-tile to
90%-tile.
ms (path 7), 95 ms (path 8), 115 ms (path 9), and 135 ms
(path 10). Finally, from G3 we pick paths with minimum RTTs
of 160 ms (path 11), 215 ms (path 12), 240 ms (path 13), 300
ms (path 14), and 670 ms (path 15). For each path we send
100 probes every one hour over a period of one day.
For each path we calculate the distance between mode and
minimum RTTs. Fig. 4 shows the range of values for the
difference between mode and minimum RTTs on each path.
We can see that the ranges for path 6 and path 10 in G2
are quite large compared to other paths. By looking into their
measurements, we find that path 6 experienced some probe
losses (between 10% to 20%) during 4 measurement periods.
For path 10, we do not observe many probe losses, but the
minimum RTT is 135 ms and the distance between the mode
and the minimum, although larger than other paths, still less
than 4% of the minimum RTT. On path 12 in G3 we observe
that the minimum RTT shifts from 215 ms to 200 ms and
then to 230 ms. We conjecture that it may be due to some
route changes (probe losses are less than 2%). Still the mode
is close to the minimum RTT, i.e., the distance between mode
and minimum is less than 10% of the minimum RTT for the
worst case.
From the figure, we can see that the variation in ε is
relatively small compared to path’s minimum RTT. Therefore,
the estimated ε at a given time can still be used to delimit
the boundaries of the congestion regions at other times. In
fact, given the diversity of the minimum RTT on our observed
paths, we believe that a default value of ε can be chosen for
each group. For example, we can see that the variation in G1
is less than that in G2 and G3. Currently, RTTometer sets
the default values of ε as 2, 4, or 6 ms for paths in G1, G2,
or G3, respectively. RTTometer can set the default value of
ε, given that the user did not specify it on the command line,
by observing the value of the RTTs for the first few probes.
C. Number of Probes
Given that the estimated ε at a given measurement period
can be used during other times, in this subsection we study
Fig. 5. Distribution of the required number of probes to estimate minimum
RTT for different groups.
the required number of probes to estimate the minimum RTT
of a path. To elaborate, given a value of ε and a required
confidence level, what is the required number of probes needed
to estimate the minimum RTT? RTTometer stops probing
once it reaches the required confidence level or after it sends
the maximum number of probes. We study how the required
number of probes changes over time. Again, we use the 15
paths chosen previously, where we estimate the minimum RTT
with confidence level of 0.8. The setting of ε is 2 ms for paths
in G1, 4 ms for paths in G2, and 6 ms for paths in G3. We
repeat our measurements every hour for a period of one day.
We set the maximum number of probes sent for a given path
to 30 probes. RTTometer starts calculating the confidence
once a minimum number of probes have been sent. In our
experiment, we set this minimum threshold to 5 probes.
Fig. 5 shows the distribution of the number of probes. Note
that the minimum number of probes sent by RTTometer is
5, therefore the value on the x-axis starts at 5 in the figure.
As we see, for the majority of measurements, 6 or less probes
are enough to estimate minimum path RTT with confidence
0.8. During some measurements, 30 probes were not enough
to estimate the minimum RTT at the required confidence level.
Specially, for paths in G1 and G3. This may happen due to
the small value of ε in G1 or due to the large delays for paths
in G3.
Fig. 6 shows the estimation of minimum RTT for path
15 from G3 along with the confidence level. The left y-axis
represents the minimum delay in milliseconds, while the right
y-axis represents the confidence level. From the figure we can
see that for a couple of measurement periods, the confidence
level dropped severely. By looking at the measurements for
these periods, we find that the first dip occurred due to a delay
shift from 210 ms to 440 ms. The first two probes experienced
the low delay while the remaining 28 probes experienced the
high delay. RTTometer correctly signaled for low accuracy
level due to route changes. In the second dip, we find that
the delay did not change, but the confidence level was low.
During this measurement period, probes experienced high
loss rate (about 40%). Even though the minimum RTT still
1200
500
RTT
Confidence
1
400
0.6
600
0.4
400
0.2
350
RTT (ms)
0.8
800
Confidence
Minimum RTT (ms)
1000
End-host
Next-to-last hop
450
300
250
200
150
100
200
0
0
5
10
15
20
25
Measurement Index
50
0
10
20
30
40
50
60
70
80
90 100
Probe Sequence
Fig. 6.
The estimated minimum RTT on path 15 from G3 and the
corresponding confidence level.
Fig. 7. Behavior of RTT measured to end host connected through cable
modem and RTT measured to the next-to-last hop.
conforms with previous measurements, RTTometer signaled
for congestion behavior on the path.
used in previous studies [13], e.g., HTTP ping measures
the service time by a Web server to serve a GET request.
However, similar to ping, these tools only provide some
reachability test and simple statistics on the measured RTT.
Other measurement tools leverage the existent of services
to measure path delay. For example, King [16] estimates
the delay between any pair of arbitrary hosts by measuring
the latency between their authoritative DNS (Domain Name
Service) servers.
In all these tools, the main goal is to measure the delay
between end hosts. Users of these tools usually specify the
number of probes and the tools report the delay of each
probe. There is no other measure associated with the captured
delay. On the other hand, RTTometer not only measures the
delay but also associates measure of confidence that reveals
the condition of the path during measurements. In addition,
RTTometer dynamically adapts the number of probes required for a given confidence level.
D. DSL and Cable Modem End-hosts
During our course of data collection, we observed some
weird behavior on some paths. Closely investigating these
paths we find that they are paths where the last hop is a
broadband link (DSL or cable). On these paths the RTT
fluctuates more often. For example, RTT may fluctuate more
than 100 ms. We have observed that the fluctuation is more
prevalent for hosts connected with cable modem. In one case,
RTT measured to the end host fluctuates between values from
126 ms up to 400 ms! When we measure the RTT on the path
up to the next-to-last hop, the RTT is 67 ms and without much
fluctuation (Fig. 7). This indicates that the delay on the last
hop is the main cause for these fluctuations. This fluctuation
of RTT on the last hop may be due to routing mechanism done
inside the cable network. In addition, cable networks tend to
over-subscribe their network and rate limit user’s traffic; it is
common to have asymmetrical bandwidth on these links. Due
to the prevalence of broadband links technology as connection
media for users to the Internet, understanding the dynamics of
RTT on these links is the main goal in our future works.
V. R ELATED W ORK
Given the administratively decentralized nature of the Internet, the ping utility has turned out to be the common
primitive to all extant RTT measurement tools. To measure
the RTT between a source and a target host, ping sends an
Echo-Request ICMP message from the source host to the target
host, and measures the elapsed time until an Echo-Response
ICMP message returns from the target host. Due to the recent
heightened security awareness of network administrators [12],
increasingly large number of routers and firewalls do not forward ICMP packets. Furthermore, some system administrators
turn off support for ICMP “request-reply” in their operating
system kernel.
Other ping-like tools have emerged due to the limitations
and disadvantages of ping. TCP ping, for example, was
introduced and used in [14], [13]. Application-level ping was
VI. C ONCLUSION
In this paper, we have described techniques based on the use
of phase-plot presented in [11] to associate a confidence level
to a measured path RTT. By laying out subsequent probes in
different congestion regions in the phase-plot, we can assess
the condition on a path during RTT measurement. Factors
that may affect the accuracy of the measured RTT are selfinterference, congestion, and route changes. In addition, these
techniques make use of the information provided by all probes,
which reveals insight about path conditions. Using these
techniques the RTTometer tool can associate an accuracy
level with a measured RTT. RTTometer dynamically adjusts
the number of probes required to measure the RTT with a
given confidence level. Consequently, applications can make
better decision not only based on the measured RTT but also
on the revealed path condition.
R EFERENCES
[1] Z. Wang, A. Zeitoun, and S. Jamin, “Challenges and Lessons Learned
in Measuring Path RTT for Proximity-based Applications,” Proceedings
of Passive & Active Measurement Workshop (PAM’03), San Diego, CA,
April 2003.
[2] T. Lakshman and U. Madhow, “The Performance of TCP/IP for
Networks with High Bandwidth-Delay Products and Random Loss,”
ACM/IEEE Transactions on Networking, June 1997.
[3] J. Padhye, V. Firoiu, D. Towsley, and J. Kurose, “Modeling TCP
Throughput: A Simple Model and its Empirical Validation,” Proc. of
ACM SIGCOMM ’98, pp. 30 –314, 1998.
[4] V. N. Padmanabhan and L. Subramanian, “An Investigation of Geographic Mapping Techniques for Internet Hosts,” Proc. of ACM SIGCOMM ’01, August 2001.
[5] D. Clark, “The Design Philosophy of the DARPA Internet Protocols,”
Proc. of ACM SIGCOMM ’88, pp. 106–114, 1988.
[6] S. Ratnasamy, P. Francis, M. Handley, R. Karp, and S. Shenker, “A
Scalable Content-Addressable Network,” Proc. of ACM SIGCOMM ’01,
August 2001.
[7] B. Zhang, S. Jamin, and L. Zhang, “Host Multicast: A Framework for
Delivering Multicast to End Users,” Proc. of IEEE INFOCOM ’02, June
2002.
[8] T. S. Eugene Ng and H. Zhang, “Towards Global Network Positioning,”
ACM Internet Measurement Workshop 2001, November 2001.
[9] P. Francis et al., “IDMaps: A Global Internet Host Distance Estimation Service,” ACM/IEEE Transactions on Networking, vol. 9, no. 5,
Oct. 2001.
[10] Y. Chu, S. Rao, and H. Zhang, “A Case for End System Multicast,”
Proc. of ACM SIGMETRICS ’00, 2000.
[11] J.-C. Bolot, “Characterizing End-to-End Packet Delay and Loss in the
Internet,” Proc. of ACM SIGCOMM ’93, pp. 289–298, Sep. 1993.
[12] S. Savage, “Sting: a TCP-based Network Measurement Tool,” USENIX
Symposium on Internet Technologies and Systems’99, Oct. 1999.
[13] S. G. Dykes, C. L. Jeffery, and K. A. Robbins, “An Empirical Evaluation
of Client-Side Server Selection Algorithms,” Proc. of IEEE INFOCOM
’00, 2000.
[14] M. Horneffer, “Assessing Internet Performance Metrics Using LargeScale TCP-syn Based Measurements,” Passive & Active Measurement
Workshop (PAM’00), 2000.
[15] “libpcap,” http://www.tcpdump.org.
[16] K. Gummadi, S. Saroiu, and S. Gribble, “King: Estimating Latency
Between Arbitrary Internet End Hosts,” Proc. of ACM Internet Measurement Workshop (IMW’02), 2002.
[17] “Napster,” http://www.napster.com.
[18] “GameSpy Arcade,” http://www.gamespy.com.