INTERNATIONAL WORKSHOP ON WIRELESS AD-HOC NETWORKS (IWWAN) 2004
1
Testing and Analyzing TCP Performance in a
Wireless-Wired Mobile Ad Hoc Test Bed∗
Andreas Hafslund1, Lars Landmark2, Paal Engelstad3 and Frank Y. Li2
1
2
Thales Communications AS, Norway
UniK – University Graduate Center, Norway
3
Telenor FoU, Norway
Abstract—TCP is the most used connection-oriented transport
protocol in the Internet today. However, TCP has a number of
problems when the connections are running over wireless links.
This paper addresses some of them for mobile ad hoc
networking. We have made a real-life testbed for experimenting
and analyzing TCP connections from a multi-homed ad hoc
network into the global Internet. Through a set of experiments,
we show how TCP behaves when switching gateways, and also
the capture and unfairness problems of TCP. The testbed and
our initial results are the first step for making an improvement to
TCP for ad hoc networking.
Index Terms — ad hoc networking, global connectivity, TCP,
wireless testbed.
M
I. INTRODUCTION
obile Ad-hoc Network (MANET) [1] is usually a
network consisting of mobile nodes with wireless
network interfaces. Each node may function as both an endhost and an intermediate router for other nodes in the network.
MANETs have usually no fixed infrastructure, and services
on the Internet might not be available in these networks. A
likely scenario is that nodes on an ad-hoc network in some
cases also want to connect to nodes on the Internet, using
services available there. For a widespread and successful
deployment of MANETs, the ability to provide easy access to
the Internet is therefore a prerequisite. A possible approach is
to let a node that is participating in a MANET operate as an
Internet gateway and provide other nodes within the MANET
with Internet access.
Since the Transmission Control Protocol (TCP) [2] is the
most commonly used connection-oriented transport protocol
in the Internet today, using TCP for global Internet
connectivity from a MANET is a particularly important
problem.
TCP was initially developed for wired networks. There are
certain characteristics of wireless ad hoc networks, however,
that cause problems for TCP. Unlike for a wired network, the
network topology of a MANET can be very dynamic, due to
mobility of the nodes and radio channel fading. Furthermore,
∗
This work is supported by THALES Communications AS, Telenor FoU,
UniK – University Graduate Center, and Norges Forskningsråd (NFR).
the network connections often have high bit error ratio and
high probability of packet loss.
When mobile ad hoc networks are going to interconnect
with the Internet, TCP has to be adjusted or altered to
overcome the problems it has when running over wireless
links. The research community has been investigating TCP
performance over wireless links for several years. There exist
some suggestions to improve TCP or to introduce new
connection-oriented transport protocols. However, most of
these studies are based upon simulation work using the
network simulator ns-2 with the wireless extensions
developed at CMU [4]. The results presented in this paper, on
the other hand, are obtained by testing TCP performance
experimentally in a real-life testbed.
The global connectivity problem for ad hoc networking is
well known. Apart from the problem of using TCP, it also
includes issues such as addressing schemes, gateway
technologies, autoconfiguration, security and authentication.
Another issue that is important for Internet connectivity from
ad hoc networks is site multi-homing (i.e. that there is more
than one MANET node operating as a gateway). Since
MANETs have usually no infrastructure, it is difficult to
eliminate the possibility of multi-homing. Coordination and
election/negotiation mechanism to suppress multi-homing
would face problems in partially connected MANETs and
MANETs with network partitions and mergers. Instead, all
solutions to the global connectivity problems should take the
possibility of multi-homing into consideration. Multi-homing
makes however the global connectivity problems considerably
more complex and challenging.
We have focused our study of the problem regarding TCP
connectivity between a wireless, multi-homed ad hoc network
and a wired, global Internet, since this issue not yet has been
fully investigated, and especially not when the MANET is
multi-homed.
We have tested how TCP connections behave when the ad
hoc network has two different gateways to the Internet, and
when a sender has to switch between gateways during a TCP
session. This situation could occur for instance if one of the
gateways becomes unavailable due to node mobility.
In this paper we will detail the TCP problems for wireless
networking. We will present our TCP testbed scenario and
INTERNATIONAL WORKSHOP ON WIRELESS AD-HOC NETWORKS (IWWAN) 2004
parameters, and we will provide experimental results for TCP
connections through global connectivity. We will investigate
the TCP unfairness problem for two or more TCP flows going
out from a multi-homed network.
The paper is divided into 6 sections. Section 2 gives a brief
look into related work on the subject, whereas Section 3 gives
an overview of TCP functionality, TCP problems related to
wireless links, and the global connectivity problem for ad hoc
networking and TCP. We present our testbed architecture in
Section 4, and our initial results in Section 5. We conclude our
work in Section 6, where we also present our plans for further
study on this subject.
II. RELATED WORK
A lot of research regarding TCP over wireless links has
been undertaken. One much cited work is [9], which studies
TCP performance related to the 802.11 MAC layer. This
simulation study shows that the relation between MAC and
TCP causes severe unfairness conditions between TCP flows.
Both the 802.11 MAC layer and TCP have back-off timers.
These timers will affect each other to give a very low
throughput for TCP when there are packet collisions due to
the hidden terminal problems. This work also demonstrate
conflicts between TCP data packets and TCP ACKs going in
different direction in a wireless network when the window
sizes are greater than one.
Some related problems are the focus in [10]. This paper
demonstrates instability problems of TCP connections over
wireless links. The work is based upon simulations in ns-2,
and shows that the TCP maximum window size has an effect
on the problem. Like [9], it also shows that there are problems
between the 802.11 MAC and TCP, and that this gives
unfairness between TCP flows.
Another paper that investigates TCP problems for mobile
ad hoc networks is [12]. Unlike previous work, this paper tries
to model a realistic scenario, by doing simulations with ns-2.
This includes better models for mobility, channel error, and
shared-channel contention. Their results show that network
disconnections and reconnections due to mobility have the
most significant impact on TCP’s performance in mobile ad
hoc networks.
A recent paper [13] shows that TCP’s throughput and loss
can be minimized for a given network topology and node
mobility, with an optimized TCP window size. TCP achieves
best throughput via improved spatial channel reuse
Several papers try to improve TCP’s performance over
wireless links. Most of these are related to wireless cells,
where there are either a base satellite station or an 802.11
access point. Recently, there has also been focus on the TCP
problem for wireless, multi-hop, ad hoc networks. The study
[11] gives an overview of the TCP problem, and also a survey
of possible solutions to the problem.
2
III. OVERVIEW OF TCP
A. TCP functionality
TCP [2] is an end-to-end transport protocol that provides
reliable, connection-oriented services to the application layer.
TCP detects packets that are lost in the intermediate network,
and trigger retransmissions until the data is correctly and
completely received. This requires that the receiver returns
ACKs (acknowledgements) for each data packet transmitted
by the sender.
TCP continually measures the time ACKs take to return
from the receiver. A missed ACK is interpreted by TCP as a
lost packet, which in turn makes TCP to retransmit the lost
packet. In order to make a decision for the time it should wait
before retransmitting the lost packets, TCP maintains a
running average of the Round Trip Time (RTT). If the unacknowledged packet is delayed more than four times the
calculated average RTT, TCP assumes the packet is lost.
In addition to retransmitting the lost packet, TCP interprets
lost (or very delayed) packets as a result of network
congestion. This assumptions stems from the fact that the
protocol is designed for wired networks where other types of
packet losses are rare. The resulting action is that TCP backs
off and adjusts its transmission rate.
TCP has three main components for rate adaptation:
• Slow Start
• Congestion Avoidance
• Fast Retransmit
The Slow Start mechanism makes a quick test of the
capacity of the network to find the optimum rate of
transmission. The sender starts with a congestion window size
set to 1. For each ACK received, the window size is increased
exponentially. When a maximum slow start threshold
(ssthresh) is reached, the sender goes into the congestion
avoidance phase. In this phase the window is first reduced to a
half, and thereafter the increase of the congestion window is
linear.
With the Fast Retransmit mechanism in use the sender goes
at immediately into the congestion avoidance phase after
having received three duplicate ACKs without waiting for the
retransmit timer. This leads to better network utilization, and
thus also to better TCP connection throughput.
In addition to the use of packet drop for indication of
network congestion, the new Explicit Congestion Notification
(ECN) mechanism has been added to improve congestion
control [3]. When there is congestion in the network,
intermediate routers will mark the Congestion Experience
(CE) code point in the IP header of the TCP traffic. This will
notify the end hosts that their connections go through a
congested network. The end hosts can adjust their
transmission rate without waiting for either a retransmit timer
or duplicate ACKs. Thus, ECN can prevent unnecessary
packet drops.
B. TCP problems due to wireless links
As outlined in Section 2, many studies show that TCP
http://www.unik.no/personer/paalee
INTERNATIONAL WORKSHOP ON WIRELESS AD-HOC NETWORKS (IWWAN) 2004
performs poorly over wireless links. In this section we will
give an overview of some of these problems.
The main problem is due to non-congestion packet loss.
TCP is designed for a wired network where packet loss
normally is a result of network congestion. This is necessarily
not true for mobile ad hoc networks. Wireless links can have
high bit error rates, and high probability of packet loss even
when there is no congestion. TCP, however, will assume that
the network is congested, and the throughput will be
degraded.
Another problem is related to mobility. When there are
disconnections and reconnections in the network, it can easily
lead to TCP timeouts. The congestion window will be
lowered, and TCP will take a long time to recover.
Mobility can also cause arrival of out-of-order packets. The
TCP receiver will generate ACK for the highest in-order
packet. This will result in duplicate ACKs, and when three
duplicate ACKs have been received, TCP goes into the fast
retransmit phase.
C. TCP interconnectivity for ad hoc networks
We have assumed that the ad hoc network is multi-homed.
This means that there are at least two gateways providing
connectivity to the Internet. The mobile nodes in the ad hoc
network have to be assigned one or two gateways to use for
their TCP connection.
Usually, the metrics of the routing protocol is based on
number of hops, as it is in our testbed. Then, the outgoing
traffic will be sent via the gateway that is located the fewest
number of hops away from the source node.
For incoming TCP traffic from the Internet, the destination
node will normally check the first packet it receives from the
source node, and send all return traffic to the source IP
address in the packet. Thus, if the source IP address has an
address prefix that belongs to one of the gateways, the return
packet will end up at this gateway.
If the gateway is equipped with a full networking prefix
(which is a likely scenario with IPv6), a MANET node that
wants to get return traffic over the gateway can get an IP
address with this prefix assigned from the gateway. This
solution was proposed in [14].
In other cases, the gateway has only one routable IP
address, which is a likely case with IPv4. All nodes
communicating over the gateway must share the same address.
To make this possible, the gateway may implement a Mobile
IPv4 Foreign Agent (FA) [15]. The source node discovers the
gateway, registers with the FA (i.e. the gateway), and registers
the globally routable IP care-of-address of the FA with its
own Mobile IP Home Agent (HA). This solution was
proposed in [6] and [7].
Alternatively, the gateway may implement Network
Address Translation (NAT) [8]. Then the source node sends
traffic out through one of the gateways. The source IP address
(and source port number if NAT port translation is
implemented) will be translated to the IP address of the NAT.
Hence, return traffic will be routed back to the same gateway,
3
which translates the packet and forwards the packet onto the
ad hoc network. This solution has been proposed in [16].
Normally outgoing traffic belonging to the same TCP
session should be consistently routed through the same
gateway as that of the incoming traffic. If a NAT-based
gateway is being used, for example, the source node cannot
start sending traffic out on another gateway. In that case the IP
source address of outgoing packets would not be translated
correctly. They would not be recognized by the destination,
and the TCP connection would break.
If, on the other and, one of the other gateway technologies
mentioned above is being used, the packets should be still be
sent out through the same gateway as that of the incoming
traffic. The reason is that the source IP address belongs to the
gateway of the incoming traffic, and not to that of the
outgoing traffic. Thus, the outgoing traffic will be an easy
target for ingress filtering [17], which nowadays is more and
more common to implement on routers in access networks.
In some special scenarios the return traffic can end up at
either of the two gateways. This happens if the MANET has a
given networking prefix, both gateways belong to the same
interior routing domain. Then both gateways can advertise
connectivity to the MANET prefix, and either of the gateways
can receive the return packets. Another example is when the
two gateways are on the same link.
Since we are mostly interested in the TCP performance in
this paper, the exact mechanisms for ensuring global
connectivity are of minor importance. As a simplification to
gain full control over the testing environment, we therefore
assume a scenario where the destination node, and the two
gateways are on the same link.
IV.
TESTBED ARCHITECTURE
A. Testbed description
Our testbed is based upon six laptop PCs running on Linux,
kernel 2.4.20, where two laptops act as gateways to the
Internet. The topology of our testbed is illustrated in Figure 1.
One node is the common destination for two sources, and it is
located in the wired domain. The other three nodes are
entirely in the wireless domain.
Each of the nodes has 802.11b WL110 Compaq Wireless
PC Card for the wireless interface, except the destination node
(D) in the wired domain. Each wireless interface has 11Mbps
as the theoretical maximum throughput. This node is running
only an Ethernet card (802.3), and it is connected through a
hub to the two gateways (GW1 and GW2). In the wireless
network, the gateways are connected to two intermediate
nodes (IN1 and IN2, respectively). The sending node (S)
roams between the nodes IN1 and IN2.
The wireless ad hoc network is running the Optimized Link
State Routing (OLSR) protocol [5], which is a proactive
protocol. All nodes exchange topology information with the
other nodes in the network regularly through routing
messages. The wired network runs static routing between D
and GW1 and GW2.
INTERNATIONAL WORKSHOP ON WIRELESS AD-HOC NETWORKS (IWWAN) 2004
We let GW2 has a low bandwidth link to D; the link has a
maximum throughput of 600Kbps. This is to simulate the
problem where a mobile node roams between gateways with
different bandwidth constrains to the Internet. An example of
such a roaming could be between a Wireless LAN hotspot and
an UMTS base station.
Each TCP flow will try to use the entire available
bandwidth over the link, and sends out packets with length of
1470 bytes.
4
the rerouting between two gateways takes maximum 6
seconds. This is due to the rerouting of the OLSR protocol.
This first scenario shows the rerouting of the TCP flow when
roaming between GW1 and GW2. We have maximum
throughput for the TCP flow, except the seconds the rerouting
takes. The results is shown in Figure 2:
Figure 2: S roams between GW1 and GW2.
Figure 1: Testbed scenario for wired-wireless connectivity.
B. TCP connection scenarios
As an initial study for the TCP problem related to multihoming and the capture effect described in [9], we have
specified several different scenarios.
Scenario I: Only one TCP connection is set up between S
and D in this scenario. The mobile node S roams between IN1
and IN2, and thus S roams between GW1 and GW2. For this
scenario, we let the link between GW2 and D have full
bandwidth. The main goal for this test is to find the time for
the rerouting of the TCP flow between GW1 and GW2.
Scenario II: In this scenario we have two TCP flows, one
from S and one from IN2. We set the bandwidth for the link
between GW2 and D to 600Kbps. S is initially connected to
GW1, and then roams to GW2. The TCP flow from IN2 starts
5 seconds before the TCP flow from S.
Scenario III: This is the opposite scenario of Scenario II, S
is initially connected to IN2, and then S roams over to IN1
and GW1. Again we have two TCP flows, one from IN2 and
one from S. The flow from S starts 5 seconds after the flow
from IN2.
Scenario II: When the TCP flow from S starts, it has full
utilization of the bandwidth, but when S roams over to GW2
after 5 seconds, the TCP flow has zero throughput for more
than the 6 seconds the OLSR rerouting needs. After that, the
TCP flows gradually gets access to the bandwidth between
IN2 and GW2. We can see that the capture effect is such that
IN2 takes nearly all resources for the link, and the flow from S
will gradually get better throughput. This is shown in Figure
3.
V. TEST RESULTS
In this section we will present our results from the testing of
the different scenarios. Scenario 1 is an initial test to verify the
correctness and functionality of the OLSR protocol, and also
to find the time TCP needed when being rerouted between two
different gateways. The next two scenarios are for identifying
the capture effect when there are more TCP flows in the
wireless network.
Scenario I: In our earlier study [16], we have shown that
Figure 3: S roams from GW1 to GW2.
Scenario III: This is the opposite scenario of Scenario II, S
is initially connected to IN2, and then S roams over to IN1
and GW1 after 10 seconds. Here we can see that when S
roams over to IN1, S gets full access to the available
bandwidth immediately. Here we do not see any capture
effects, as expected. However, when the flow from S starts,
INTERNATIONAL WORKSHOP ON WIRELESS AD-HOC NETWORKS (IWWAN) 2004
we do see the capture effect. Since the flow from IN2 starts 5
seconds before the flow from S, this flow takes almost all
resources for the link between IN2 and GW2. When the flow
from S starts, it takes 5 more seconds before this flow acquires
a reasonably throughput. This is shown in Figure 4.
Figure 4: S roams from GW2 to GW1.
As was shown in [9] through simulations, we have shown
with our wireless-wired testbed architecture. The TCP capture
effect will degrade the TCP performance when more than one
flow tries to use the same wireless link. This is a problem that
should be investigated further. Also, we have shown that the
multi-homing of an ad hoc network causes delay because of
the rerouting, but this causes no problem for TCP. However,
when one of the gateways has a lower bandwidth link to the
Internet, TCP will have to adjust to this without timing off due
to congestion.
VI. CONCLUSION AND FURTHER STUDY
The work presented in this paper is the initial part of our
project in evaluating TCP performance, and further improving
and implementing the protocol for real-life ad hoc networks.
Our main interest is for multi-homed wireless, ad hoc
networks. As an initial step, we have constructed two different
scenarios for analysing TCP performance in a Linux testbed.
Through a set of experiments, we have shown how rerouting
in ad hoc networks affects TCP’s performance, and how
unfairness problem exists when multiple TCP connections coexists.
For further study, we shall focus on the following three
aspects:
• Further study of multi-homed ad hoc networks.
• Implementing and testing of an improved TCP for
wireless networks.
• Investigating how a gateway can be used for
controlling the TCP traffic and performance to and
from the ad hoc network.
ACKNOWLEDGMENT
We would also like to thank Andreas Tønnesen for help in
5
preparing the OLSR protocol, which is used in the ad hoc
network.
REFERENCES
[1] J. Macker, ”Mobile Ad hoc Networking (MANET): Routing Protocol
Performance Issues and Evaluation Considerations”. IETF RFC 2501,
January 1999.
[2] J. Postel, ”Transmission Control Protocol”. IETF RFC 793, September
1981.
[3] K. Ramakrishnan, S. Floyd, D. Black, ”The Addition of Explicit
Congestion Notification (ECN) to IP”. IETF RFC 3168, September 2001.
[4] http://www.isi.edu/nsnam/ns/index.html.
[5] T. Clausen, P. Jacquet, A Laoiti, P Minet, P. Muhlethaler, A Qayyum, L
Viennot, ”Optimized Link State Routing Protocol”. IETF RFC 3626
October 2003.
[6] C. Perkins, E. Belding-Royer, Y. Sun, ”Internet Connectivity for Ad Hoc
Mobile Networks”, International Journal of Wireless Information Networks,
April 2002.
[7] E. Belding-Royer, Y. Sun, C. Perkins, ”Global Connectivity for IPv4
Mobile Ad Hoc Networks ”, IETF Internet Draft, work in progress, November
2001.
[8] P. Srisuresh, M. Holdrege, "IP Network Address Translator (NAT)
Terminology and Considerations", IETF RFC 2663, August 1999.
[9] M. Gerla, K. Tang, R. Bagrodia, "TCP Performance in Wireless, Multihop
Networks", IEEE WMCSA’99, February 1999.
[10] S. Xu, T. Saadawi, "Does the IEEE 802.11 MAC Protocol Work Well in
Multihop Wireless Ad Hoc Networks", IEEE Communication Magazine, June
2001.
[11] H. Elaarag, "Improving TCP Performance over Mobile Networks", ACM
Computing surveys, Septemeber 2002.
[12] Z. Fu, X. Meng, S. Lu, "How Bad TCP Can Perform In Mobile Ad Hoc
Networks", IEEE ISCC, July 2002.
[13] Z. Fu, P. Zerfos, H. Luo, S. Lu, L. Zhang, M. Gerla, "The Impact of
Multihop Wireless Channel on TCP Throughput and Loss", IEEE INFOCOM
2003.
[14] R. Wakikawa, J Malinen, C. Perkins, A. Nilsson, A. Tuominen ”Global
Connectivity for IPv6 Mobile Ad Hoc Networks ”, IETF Internet Draft, work
in progress, November 2001.
[15] C. Perkins (editor), ”IP Mobility Support for IPv4”. IETF RFC 3220,
January 2002.
[16] P. Engelstad, A. Tønnesen, A. Hafslund, G. Egeland ”Internet
Connectivity for Multi-Homed Proactive Ad Hoc Networks” To appear in
Proceedings of the 2004 International Conference on Communication (ICC
2004), Paris, June 20-24, 2004.
[17] P. Ferguson and S. Senie, ”Ingress Address Filtering: Defeating Denial
of Service Attacks which employs IP Source Address Spoofing”. IETF RFC
2827, May 2000.