Abdul Jabbar Mohammad, Graduate Research Assistant
Dr.Victor Frost, Dan F. Servey Distinguished Professor
(September 24, 2003)
University of Kansas

Motivation

Polar Radar for Ice Sheet Measurements (PRISM)

The communication requirements of PRISM field experiments in Greenland and Antarctica include

Data telemetry from the field to the University

Access to University and web resources from field

Public outreach to increase the interest of student community (K-12) in scientific research and enable the science community to virtually participate in polar expeditions
 Generic data communication for Remote field research

Mainstream communication system for polar science expeditions, field camps in
Arctic/Antarctic and other research purposes

Government and security use
2
University of Kansas

Introduction – Commercial Satellite Systems

Polar regions do not have conventional communication facilities (dial-up, DSL, Cable
Modem, etc) and are not serviced by most of the major broadband satellite systems.
Intelsat
Inmarsat
Globalstar
University of Kansas
3

Introduction – Special Purpose Satellite Systems




NASA satellites like ATS3, LES9,
GOES, TDRS 1,and MARISAT2 provide broadband access to Polar
Regions
Geo-synchronous, they have a limited visibility window at Poles – typically
10-13 hrs/day.
High satellite altitude and low elevation angles (1-2 0 ) result in extremely large field equipment.
May not be readily available
20 m diameter Marisat and GOES antenna at South Pole
University of Kansas
4

Introduction - Iridium Satellite System

The only satellite system with true pole-to-pole coverage

66 low earth orbiting (LEO) satellites with 14 spares

It has onboard satellite switching technology which allows it to service large areas with fewer gateways

Since it was originally designed as a voice only system, it provides a low data rate of 2.4Kbps

Not practical to be used as a main stream/ life-line communication system
University of Kansas
5

Introduction – Problem Statement

Problem Statement
A reliable, lightweight, portable and easily scalable data/Internet access system providing true Polar coverage.

Solution
Implement a multi-link point-to-point Iridium communication system to combine multiple satellite links to obtain a single logical channel of aggregate bandwidth.
University of Kansas
6

Background - Iridium
Satellite Type
Satellite altitude
Minimum elevation angle
Average satellite view time
Access scheme
Maximum number of located users
Theoretical throughput
Type of data services
LEO
780 km
8.2
0
9-10 minutes
FDMA and TDMA
80 users in a radius of 318 km
2.4
– 3.45 Kbps
Iridium-to-Iridium, Iridium-to-PSTN
University of Kansas
7

Background - Inverse Multiplexing






Traditional Multiplexing Data from a multiple applications/users sent over a single high bandwidth link.
Inverse Multiplexing - Data from a single application is fragmented and or distributed over multiple low bandwidth links.
Increases the available bandwidth per application significantly
Multi-link point-to-point protocol (MLPPP), an extension to the PPP is a packet based inverse multiplexing solution
Overhead of 12 bytes
Fragmentation of network layer protocol data units
(PDU’s) into smaller segments depending upon the PDU size, link MTUs and the number of available links.
App 1
App 2
App 3
App 1
High Bandwidth Link
Multiplexing
Low Bandwidth Links
Inverse multiplexing
8
University of Kansas

Background - Multi-link point-to-point protocol



Layer 2 protocol that implements inverse multiplexing on a packet by packet basis
IP packets are encapsulated in to PPP frames with segment numbers – 12 byte overhead
Packet fragmentation depends on the number of available links and their capacity, packet size and MTU size
Not Fragmented
IP Packet
Fragmented
IP Packet
SH IP Packet SH PDF SH PDF SH PDF SH PDF
Modem i
Modem 1 Modem 2 Modem 3 Modem 4
SH-Segment header; PDF- packet data fragment
University of Kansas
9

Multi-channel Iridium System – Design
The design requirements of the system are as follows.

The multi-channel implementation should maximize the throughput.


To support real-time interactions, the system should minimize the end-to-end delay.
The overall system should be reliable and have autonomous operation so as to handle call drops and system/power failures in remote field deployment.
University of Kansas
10

Multi-channel Iridium System – Protocol Stack
Remote System
Application
HTTP, FTP, SSH
TCP
IP
PPP/MLPPP
Physical Modems point-to-point satellite links
Local System
Application
HTTP, FTP, SSH
TCP
IP
PPP/MLPPP
Physical Modems
University of Kansas
11

Multi-channel Iridium System – Network Architecture
World Wide Web
ITTC Default Router eth0
(Default gateway)
PPP Server ppp0
P-T-P Satellite link ppp0
NGRIP Camp, Greenland
100 Mbps
Ethernet
(Default gateway)
PPP Client
Guser 4 eth0
User 2 100 Mbps
Ethernet
User 3
User 1
Guser 3
Camp
WI-FI
Guser 2
G`user 1
ITTC Network, University of Kansas
University of Kansas
12

4-Channel Iridium System Implementation
Remote
System
PPP client
I. Modem 1
I. Modem 2
I. Modem 3
I. Modem 4
Remote Subsystem
Iridium
Gateway
Local
System
PPP Server
Local Subsystem
Modem
Pool
PSTN
University of Kansas
13

4-Channel System – Implemented at KU
University of Kansas
14

4-Channel System – Implemented at KU
University of Kansas
15

4-Channel System – Software Overview



Linux based system
Open source PPP package
Custom software developed at University of Kansas

Chat scripts for Iridium modems

PPP script to tune link parameters for Iridium satellite system

Client-Server configuration

Autonomous operation-connection setup, user authentication, detecting failures, reconnections, handling power failures/system resets, generating status information (text)
University of Kansas
16

4-Channel System – Modem Flow Control
START
Check if any modem is alive
NO
Connect
Modem 1
OK
FAIL
YES
Terminate
All
Monitor
Status
OK
FAIL
Connect
Modem 2
OK
FAILED
Monitor
Status
FAIL
OK
FAILED
Connect
Modem 3
OK
Monitor
Status
FAIL
Connect
Modem 4
OK
OK
FAILED
Monitor
Status
OK
FAIL
University of Kansas
17

Field Tests and Results – Field implementation
System with control software
USB In
USB Out
Antenna
Connectors
Dimensions: 24x19x5
University of Kansas
18

Field Tests and Results – Antenna Setup
3 FT
10 FT
University of Kansas
19

Results – Delay and Loss Measurement





Ping tests between the two machines at the end of the of satellite link
One way propagation delay = (800+8000+6000)Km / (3e6)Km/sec= 49msec
Transmission time for 64 bytes@2.4Kbps = 64*8/2400=213msec
Theoretically, the RTT delay =2*(49+213)= 524msec+delay at the gateway
Test results show an average RTT delay of 1.8 sec, which may be attributed to the intersatellite switching and delay at the gateway
Packet s sent
Packets received
%
Loss
Avg
RTT (sec)
Min Max mdev
50 50 0 1.835
1.347
4.127
0.798
100
100
200
100
100
200
0
0
0
1.785
1.448
4.056
0.573
2.067
1.313
6.255
1.272
1.815
1.333
6.228
0.809
University of Kansas
20

Results – Delay Measurement

Random variation of delay



High mean deviation
Delay increases linearly with packet size
Normalized delay is almost constant for MTU sizes > 800 bytes

Changing distance between the user and satellite as well as between satellites themselves
(ISLs)

Non-uniform traffic distribution, varying delays on different routes
RTT Vs Time of the day
6
5
4
3
8
7
2
1
0
1:00 3:00 9:00 13:00 15:00 20:00
Time of the day in hh:mm
Min
Avg
Max

Normalized RTT Vs Packet Size
35
30
32.05
26.50
25
20
15
10
13.83
12.35
9.24
7.51
7.52
7.04
7.13
5
0
56 100 200 300 500 800 1000 1200 1500
Packet Size (Bytes)
University of Kansas
21

Results – Throughput

Tools used
– TTCP, IPERF

Throughput varies to some extend due to RTT variation

Efficiency > 90%
Effective throughputs during large file transfers
File Size (MB)
2.3
1.5
2.5
0.75
3.2
1.6
Upload Time
(min)
45
28
35
11
60
23
Throughput
(bits/sec)
9091
7111
9275
6815
7143
9524
10
7
6
9
8
5
4
3
2
1
0
Method 1 Modem 2 Modems 3 Modems 4 Modems
Iperf
Iperf
Iperf
Ttcp
Ttcp
Average
2.1
1.9
1.7
2.29
2.48
2.1
4.0
3.9
4.5
4.43
4.40
4.25
7.0
7.0
6.8
6.6
7.0
6.88
9.6
9.3
9.7
8.9
8.78
9.26
Throughput Vs Number of Modems
2.1
4.25
6.88
9.26
1 2
Number of modems
3
University of Kansas
4
22

Results – Reliability: 10 th July 24 hr test
Total :
13 Call drops
4
3
2
1
0
Time
Modem up times during 24 hour test
Uptime %
80.6
91.8
94.7
96.8
Time
Time interval between call drops
(minutes)
146 106 114 50 25 84 89 8 7 7 17 11 137 618
University of Kansas
23

Results – Reliability: 12 th July 24 hr test
Total :
16 Call drops
Time
Modem up times during second 24 hour test
4
3
2
1
0
Uptime %
80
92
95
96
Time
Time interval between call drops
(minutes)
135 248 93 40 26 16 8 211 108 91 8 5 6 5 8 7 386
University of Kansas
24

Results – TCP performance of a single link





TCP Version – TCP SACK
Throughput of a segment is defined as the size of the segment seen divided by the time since the last segment was seen (in this direction).
RTT is defined as the time difference between the instance a TCP segment is transmitted (by the TCP layer) and the instance an acknowledgement of that segment is received.
The average throughput of the connection is
2.45Kbps.
The average RTT was found to be 20 seconds


Bandwidth Delay Product (BDP) = 2400*20/8 =
6000 bytes = 4 segments.
Due to low throughput, the BDP is small.
Throughput vs. Time
-------avg. of last 10 segments
-------- avg. of all the segments up to that point
RTT vs. Time
25
University of Kansas

Results – TCP performance of a 4-channel system



The average throughput obtained - 9.4 Kbps
The average RTT observed - 16.6 seconds
Factors affecting throughput and RTT

TCP Slow start
 MLPPP fragmentation

Random delay variation

TCP cumulative acknowledgments
Time Sequence Graph
Throughput vs. Time
-------avg. of last 10 segments
-------- avg. of all the segments up to that point
RTT vs. Time
------Received Acknowledgements
------TCP segments transmitted
------Received window advertisement
University of Kansas
26

Results – TCP performance of a 4-channel system
Time Sequence Graph

Outstanding Unacknowledged data and Congestion window
Time Sequence Graph
Outstanding Unacknowledged Data
------Received Acknowledgements
------TCP segments transmitted
------Received window advertisement
------Instantaneous outstanding data samples
------Average outstanding data
------Weighted average of outstanding data
University of Kansas
27

Results – TCP performance degradation due to packet loss

Low packet loss, long time experiments needed to determine the performance degradation

Two packet losses were observed in the FTP video upload resulting in packet retransmissions

Packet losses usually occur during call hand-offs

Retransmission time outs (RTO) is very large due to high RTT and high mean deviation
Time Sequence Graph Time Sequence Graph
Retransmitted packet
Retransmitted packet
University of Kansas
28

Results – TCP performance degradation due to packet loss
RTT vs. Time

Effect on the TCP performance due to packet loss

Decrease in the throughput following the packet loss

RTT peaks around the packet loss
 The average throughput of the connection is observed to be 7.44 Kbps.
Outstanding Unacknowledged Data
Throughput vs. Time
University of Kansas
29

Results – TCP performance degradation due to call drop
Time Sequence Graph





Packet loss due to a call drop on one links of the multilink bundle
A finite amount of time for the data link layer realize the link has failed
Large RTO timer
The entire window of packets (12 in this case) and acknowledgements that are in flight on that particular link are dropped.
Throughput of the connection – 7.6 Kbps.
Time Sequence Graph
------Received Acknowledgements
------TCP segments transmitted
------Received window advertisement
Outstanding Unacknowledged Data
------Instantaneous outstanding data samples
------Average outstanding data
------Weighted average of outstanding data
University of Kansas
30

Results – Mobile tests
Test Location
University of Kansas
31

Results – Mobile Performance
START
Throughput vs. Time
STOP
Avg. Throughput = 9 Kbps
University of Kansas
32

Results – Mobile Performance
(cont.d)
Throughput vs. Time
Avg. Throughput = 9.34 Kbps
STOP
START
University of Kansas
33

Applications – Uploads and Downloads
4
5
6
2
3
7
The following files were downloaded from WEB and ITTC network. The size of these files, their importance (on a scale of 1-10, based on user survey) are shown
1
8
9
Title
Spectrum Analyzer programmers
Manual
Matlab Programs
Voltage regulator data sheet
GPS software
Proposal submission
Downloaded/uploaded
Download from Agilent.com
Download from ITTC
Download from Fairchild.com
Download
Upload
Access point manager software Download from Orinoco.com
Size
7.2MB
Drawing of machine spares to order
Video of core, datasheet
Upload to University of Copenhagen
Upload for press release
Pictures, press release of longest core in Greenland
Upload to Kangerlussauq for press release
1MB
2MB
500KB
Imp
9
500KB
226KB
800KB
600KB
4.66MB
7
9
8
7
9
9
8
6
University of Kansas
34

Applications – Internet at camp
University of Kansas
35

Applications –WI-FI setup integrated with Iridium
University of Kansas
36

Applications - Wireless Internet

Wireless Internet access was used by many NGRIP researchers
 Email access put the researchers in touch with their home institutions
 Scientist at NGRIP were able to send information back to their home institutions

The system was also useful for general camp purpose: sending drawings to order spares for a broken caterpillar, excel spreadsheet for food order, general press releases, downloading weather reports for planning C-130 landings
University of Kansas
37

Applications – Outreach
 Daily journal logs were uploaded to the University of Kansas and posted on www.ku-prism.org
 Two way video conference was held twice to test the real time audio/video
 The system not only helped PRISM group to download critical software, but also made it possible to obtain faculty guidance and expertise on the field
Testing Video conference between the camp and the University
 It also helped the researchers back at the
University of Kansas to virtually view the field experiments since the video clips of experiments being carried out were also uploaded to the
University and posted on project website
Uploading daily field logs
University of Kansas
38

Lessons Learned

Multi-link PPP is relatively efficient over Iridium system. With 4-modems efficiency was observed to be >90%

The RTT the system is significant ~ 1.8 seconds, which is an impairment to real time interactions
 There is a large (random) variation in delay of the system

The average up-time of the overall connection is good > 95%

Mobile tests showed performance very similar to that of stationary system up to speeds of 20mph

A call drop on the first modem, results in a complete loss of connection, a potential bug in the PPP networking code; could be fixed in newer version of the PPP software

Strong interference from a nearby source on the same frequency (1.6GHz) could cause little disturbance in the system leading to either connection failures or call drops. This was observed when a radar sweeping between 500MHz-2GHz with an output of 20dBm was placed at distance less than
15 meters
University of Kansas
39

Conclusions

In order to provide data and Internet access to Polar Regions, we have developed a reliable, easily scalable, lightweight, and readily available multi-channel data communication system based on Iridium satellites that provide round the clock, pole-topole coverage.

A link management software is developed that ensures fully autonomous and reliable operation

An end-to-end network architecture providing Internet access to science expeditions in
Polar Regions is demonstrated.

This system provided for the first time, Internet access to NGRIP camp at Greenland and obtained the call drop pattern.

The TCP performance and the reliability of the system is characterized
University of Kansas
40

Future Work





Modify the MLPPP code so that the interface is attached to the bundle and not to the primary link
Additional research to determine the cause of delay and develop methods to overcome it.

Evaluate the new data-after-voice (DAV) service from Iridium
Evaluate different versions of TCP to determine the enhancements that can handle the random variation and value of RTT
Improve the user friendliness of the system

GUI for connection setup

Self-test / diagnostic tools for troubleshooting
Research into the spacing and sharing of antennas to reduce the antenna footprint
University of Kansas
41