Integration of Internet and
real time control systems in
telerobotics
Introduction
Many devices are today accessible through
Internet:
Web cams
Washing machines, coffee pots and other appliances
Robots
The term “control” is referred to the
possibility of sending a list of commands to
the remote device
Introduction
Note:
The control loop is not closed across the network
The remote equipment executes the commands under the
supervision of its own local controller
Why?
The unpredictable performance does not allow the
realization of an Internet-based, real-time control system,
in which the control loop is closed across the network
Introduction
Question:
If the characteristics of the Internet connection in use are
known, is it possible to implement a real-time, closed-loop
control system, integrating that connection?
Answer:
JBIT – Java Based Interface for Telerobotics: a telerobotics
equipment for real-time teleoperation over Internet, with
visual and force feedback
Introduction
Closing the loop across Internet…
Internet
Local
system
Internet
Internet
Remote
system
Summary
Internet modeling
Delays, losses and bandwidth
Control of dynamic systems with variable
time-delay
Solutions available in literature
Packet loss handling
The JBIT project
Internet modeling
Internet can be considered as a strongly
connected network of computers,
communicating with each other using
packetswitched protocols
To B
To B
B
To B
To B
To D
A
C
D
Internet modeling
From source to destination, each packet
traverses several nodes
Each node has different throughput, routing
policy, and buffering and queues
management
Each node handles many other data flows,
this resulting in random service time
Internet modeling
When the data flow between end users
exceeds the available bandwidth, congestion
occurs
The available (or marginal) bandwidth for a
single user depends on the data flow
generated by other users, sharing the same
physical connection
Internet modeling
End-to-end connection modeled by:
Average delay
Available throughput
Delay statistics/variations
Packet loss statistics
Simple experiments to get the model
ICMP packet injection and Round Trip Time (RTT)
measurement
Internet modeling
RTT vs. Time:
Short term
Long Term (1 week)
Internet modeling
RTT distribution:
Depends on the number of nodes traversed:
10000 km connection
30 km connection
5
hops
1
2
3
4
5
6
arianna.dei.unipd.it
dei-cisc.dei.unipd.it
147.162.6.254
pd4000.unipd.it
147.162.5.254
berica2.gest.unipd.it
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
arianna.dei.unipd.it
dei-cisc.dei.unipd.it
147.162.6.254
pd4000.unipd.it
cnaf-gw-2.infn.it
cnaf-gw1.infn.it
131.154.8.1
mix-serial3-4.Washington.mci.net
core1-fddi-0.Washington.mci.net
border1-fddi-0.Bloomington.mci.net
border1-fddi-0.Bloomington.mci.net
csunet-losnettos.Bloomington.mci.net
ISI-SWRL-GW.LN.NET
acg-isi.ln.net
jpl-acg.ln.net
192.138.85.65
b198-fddi.jpl.nasa.gov
helios.jpl.nasa.gov
17
hops
Internet modeling
RTT distribution:
30 km
Exponential-like
10000 km
Gaussian-like
Internet modeling
RTT phase plots (RTTn+1 vs. RTTn)
rttn+k+1 = rttn+k + P/ - 
Non-congested
Congested
Note: P is the packet length,  is the channel throughput,  is the period of the probing packet
Internet modeling
Available bandwidth can be obtained by
injecting a data flow that brings the
connection close to saturation.
The closer to saturation, the higher the
packet loss rate
Host
name

Average
RTT
[msec]
Berica 10 8.44
Helios 10 319.0
Berica 100 8.10
Helios 100 326.3
Std.
Dev.
[msec]
6.25
16.70
5.35
27.20
Packet
Loss
rate
5.97
51.13
0.08
41.36
Internet modeling - Summary
Available bandwidth, average delay, delay
deviation and loss rate can be easily obtained
with simple injection of probing packets
Saturation can be avoided by setting the
upper limit of the application throughput
below the available BW
Control over Internet
Closing the loop across Internet
Internet
Local
system
Variable
delay 1
Variable
delay 2
Remote
system
Control over Internet
Few solutions available in literature for timevarying delay
All algorithms are designed on the basis of
maximum delay and its max. derivative
Structural constraints:
With Internet, short interruptions may occur
Decentralized control is preferred
In case of interruptions, it must be allowed to
have reduced perfomance
Control over Internet
Other solution: provide a pair of queues to
even out the delay jitter
In this case, all constant-delay techniques can
be applied (e.g. Smith predictor)
Control over Internet
Which protocol is best suited for Internetbased control?
TCP: Acknowledge mechanism interrupts control
loop
RTP: Oriented to audio-video streaming
UDP: No detection of missing packets
Enhanced UDP:
Packet loss detection and time stamping
Control over Internet
Other problem: random packet losses
Losses may be single or burst (more than one
packet lost sequentially)
Burst losses occur if the data rate is over the
maximum available BW
Data flow must be adapted to operating
conditions
Monitoring of available bandwidth through RTT
on-line measurement
Control over Internet
Single losses can be handled with a predictor
The prediction is used instead of the missing
data
This works in conjunction with the buffering
mechanism and a packet loss detection
procedure
Based on E-UDP features
Control over Internet - Summary
Not all applications are suitable for IP-based
control
The application must “survive” even in case of
long interruptions:
Decentralized control must be allowed
Monitoring of connection characteristics and
adaptation of the controller to operating
conditions
Data recovery must be provided
Case study: Telerobotics
Bi-directional
data exchange
Case study: Telerobotics
This application is well suited for IP-based
control:
Decentralized structure
Several control algorithms for constant time delay
are available
s
Telerobotics at the Industrial Electronics Lab.
Research started in 1995
Biorobotics Lab. – UW – Seattle
Telerobotics Group – Jet Propulsion Laboratory
LVR - Università di Orleans-Bourges
Design and realization of an Internet-based
telerobotic equipment with force feedback
Enanched-UDP
JBIT
P@dus - OTELO
Web-based telerobotic systems
Several systems are available
None allow real time feedback and control
CGI interface and script-like commands
Connection is open when the query is sent and
closed upon reply arrival
no continuous video streaming
e-mail or updated environment picture as
feedback
JBIT: Java-Based Interface
for Telerobotics
Targets:
to enable any Internet user to access a
remote robot, with real time video, VR and
(possibly) force feedback
low cost (overall!)
no special tools (SW & HW)
JBIT solutions - 1
Cost:
Freeware
Low cost camera (Quickcam)
Commercial force display (Microsoft Side
Winder Pro) as master
Direct-drive, low cost robot as slave
(cannibalized from HDDs)
Sensorless force feedback
JBIT solutions - 2
Real-time visual and force feedback:
Java servlets for fast server response
compressed video streaming (H.263) for video
VR-based interface to improve the visual
feedback performance
coordinating force feedback + queuing + data
recovery to cope with IP delays and losses
Structure of JBIT system
JBIT: Client-server structure
JBIT: Client-server structure
Server (implemented with Java servlets)
Waiting for client requests
Access control, timeout
Data exchange between client and robot (no direct
access form client to robot)
Video encoding (H.263) and streaming
Monitoring of available bandwidth
Adaptation of video and control output rate
JBIT: Client-server structure
Client (implements user interface)
Communication with servlets
Selection of operating mode
Radio button and active joystick interface
VR interface
Video decoding and playing (Java Media
Framework - JMF)
Client interface
JBIT - Force feedback
•Up to 200 Hz sampling rate on a LAN, 50 Hz with a 14.4 kbps modem
JBIT - Force feedback
Perception of the remote environment
Depends on the connection delay
Remote tracking of
a square object
Reference
Conclusions
The realization of an IP-based closed loop
control system is not a trivial task.
Precise knowledge of
characteristics is needed
Such characteristics
monitored
the
must
be
connection’s
constantly
Reconstruction of lost data should be ensured
Conclusions
Need for some smart strategies to deal with
long term interruptions
Few applications are suitable for IP-based
control
Several issues still to be addressed in order to
achieve a general solution for IP-based
control
Conclusions
Some help may arrive from new network
technologies:
Hard real-time connections (e.g. Tenet)
New version of Internet (IPv6)
RSVP (Resource Reservation Protocol)
New video compression (MPEG4)
It will be possible to have connections with
prescribed Quality of Service (QoS), in terms
of throughput, losses and delays
Reliable control of remote equipment using
Internet could be implemented more easily.
On-going project: P@dus
PATIENT STATION (SLAVE)
ECHO PROBE
PATIENT
SLAVE
ECHOGRAPH
PC
EXPERT STATION (MASTER)
COMMUNICATION
CHANNEL
EXPERT
HAPTIC MASTER
PC
Tele-echographies