Real-Time Collaborative Environments
Colin Perkins
http://csperkins.org/
Talk Outline
• Real-time Collaborative Environments
–
–
–
–
–
Definition and Requirements
Example: AccessGrid
History and Current Status
Underlying Technologies and Standards
Capture and Analysis of Real-time Sessions
• Research Issues and Future Directions
Copyright © 2005 University of Glasgow
…from the perspective of a developer of those applications, and of
the relevant technical standards
Real-Time Collaborative Environments
• Support collaborative work by scientists,
industry and others
– Both one-to-one and group communication
– Widely distributed participants; heterogeneity
• Provide high-quality audio-visual media
• Provide sense of community and presence
Original Picture ©1998 UCL
– Venues as a rendezvous point; familiar virtual
meeting places
• Integrate with other computing resources
Copyright © 2005 University of Glasgow
– Visualization, shared state & data repositories,
computational resources, applications
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
• Secure and trusted infrastructure
Picture from http://www.accessgrid.org/
The AccessGrid
• Example: A widely used real-time collaborative environment
– Supports many-to-many interactive meetings
– Typically room based, although not required
by the underlying technology
• Components:
– Venue client
– Media tools
– User experience
Copyright © 2005 University of Glasgow
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
• A framework for initiating real-time
collaborative work sessions
– Default media tools provided, can be
replaced
Picture from http://www.accessgrid.org/
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
The AccessGrid: Venue Client
• Venue client provides the
rendezvous point to join a
session
• Maintains state, participant
list, links to other venues,
etc.
Copyright © 2005 University of Glasgow
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Picture from http://www.accessgrid.org/
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
The AccessGrid: Typical Media Tools
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Copyright © 2005 University of Glasgow
• Joining a venue starts multicast audio and
video tools on the controller display
– Audio level and activity indicator
– Video thumbnails and statistics
• A separate display shows the main video
windows…
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
The AccessGrid: User Experience
Copyright © 2005 University of Glasgow
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Tool development ongoing - the
user experience will change with
time…
Picture from http://www.accessgrid.org/
History and Standards Development
• Long development history
• Strong open standards
• Commercial interest
Grid Computing
vat
vi
c
NVP-II
nte
Mbone tools
IP Multicast
NVP
rat
AccessGrid
sdr
wb
3G phones
RTP
RFC 741
ST
ST-II
SIP
Copyright © 2005 University of Glasgow
H.320
1975
1985
H.323
1995
Video
conferencing
2005
Current Deployment Status
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
• Hundreds of nodes at sites worldwide
• Large scale multicast video conferences
• Standard, if primitive, media formats
• Non-standard web service control infrastructure
Quick Time™a nd a
TIFF ( Unco mpre ssed ) dec ompr esso r
ar e nee ded to see this pictur e.
• Tens of millions of mobile phones
• Voice telephony and multicast streaming
• Standard, advanced, media formats
• Standard SIP-based control infrastructure
Copyright © 2005 University of Glasgow
Others…
• Dedicated video conferencing
• Wi-Fi and wired SIP-based VoIP
• PC-based video streaming
• Mixture of standard and proprietary
• Largely successful deployment
of open standard media coding
and transport protocols
– With notable exceptions, and
continuing patent issues…
• Largely successful deployment
of open standard session setup
and control protocols
– AccessGrid, although open, is
the main exception to the use
of standards-based protocols
• Independent and interoperable
implementations available and
widely used
Underlying Technologies
Session and call control
Web/Grid
Services
SDP, SDPng
RTSP
Light weight
sessions
SIP
NAT Traversal
ICE/TURN
SAP STUN
TCP
UDP/UDPlite
IP
Copyright © 2005 University of Glasgow
• Complex protocol stack, many options
• Two phase operation: control then data
SIP+SDP
Web services
RTP + media codecs
Media
codecs
RTP
DCCP
Control Protocols: Concepts
• Many control protocols exist
– For telephony and integration with the PSTN
– For streaming video, television and radio
– For collaborative work in a Grid environment
• Some common features:
Copyright © 2005 University of Glasgow
–
–
–
–
Initial rendezvous point and user location
Media format negotiation
v=0
o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5
Address and port negotiation
s=SDP Seminar
i=A Seminar on the session description
Session metadata
protocol
•
•
•
•
Dates and times
Additional information
Participant scheduling
Shared state
u=http://www.example.com/seminars/sdp.pdf
e=j.doe@example.com (Jane Doe)
c=IN IP4 224.2.17.12/127
t=2873397496 2873404696
a=recvonly
m=audio 49170 RTP/AVP 0
m=video 51372 RTP/AVP 99
a=rtpmap:99 h263-1998/90000
– Negotiation of security parameters
Control Protocols: Implementations
• Session Initiation Protocol (SIP)
– Commercial video conferencing products
– Commercial telephony and voice-over-IP
• Real-Time Streaming Protocol
– Video streaming (e.g. QuickTime)
• Web services
– AccessGrid
All evolutions of HTTP designed
to support negotiation and control
of real-time media sessions
Copyright © 2005 University of Glasgow
Complex, flexible
Large toolkits
Initial negotiation
Ongoing control
Data Transfer Protocols
• Once negotiation completed, media data transferred
Application programs
• Complex due to nature of IP networks
– A “best effort” packet delivery service
– Performance not guaranteed
HTML
MIME
HTTP
SMTP
RTP
FTP SIP RTSP
• Loss, delay, reordering, corruption, duplication
TCP UDP
IP
– Higher level protocols must compensate
Copyright © 2005 University of Glasgow
• Application complexity, simple network
• An application must correct errors, recover
media timing, etc., in addition to processing
audio/visual data
• Cannot use TCP, since require accurate timing
Media codecs
ADSL
Ethernet
PPP
Twisted Pair
Optical Fibre
• Main standard is RTP: Real-time Transport Protocol
Wireless
RTP: Real-time Transport Protocol
• A standard protocol for real-time collaborative environments on
IP networks
• Provides features for:
–
–
–
–
Timing recovery and synchronisation
Loss detection and recovery
Media format identification
Participant identification and
presence
– Reception quality statistics
Copyright © 2005 University of Glasgow
but must be implemented in
an application!
• A toolkit for real-time data transfer
V PX
CC
M
PT
Sequence Number
Timestamp
Synchronization source (SSRC) identifier
Contributing source (CSRC) identifiers
Payload data
Padding
Capture and Analysis of Media Sessions
• Media capture and decoding is complex
– One cannot simply dump packets to a file!
– Need to understand transport protocol
• Reconstruct timing, correct errors, log metadata
– Need to understand the signalling protocol
• Metadata needed to find and understand the media
• Data storage format:
Copyright © 2005 University of Glasgow
– Standard file formats exist, but often lacking in metadata and context
– Formats based on the ISO multimedia file format better?
• Relevant? “RTP Payload Format for ETSI ES 201 108
Distributed Speech Recognition Encoding”, RFC 3557
Underlying Technologies
• Flexible and sophisticated control protocols
– Initial rendezvous and negotiation
– Ongoing control during a collaborative work session
• Capable data transfer protocols; range of media formats
• “Some assembly required”
– Protocols are expected to be tightly coupled to applications
– The design rejects much of the traditional protocol layering
Copyright © 2005 University of Glasgow
– Collaborative work environments use a wide range of protocols;
often in different ways to commercial products
Research Issues and Future Directions
• Continual evolution of protocols
– New media formats
– Extension of the control frameworks
• Emergency calls (E.911)
• Presence and messaging
• NAT traversal, seamless operation over present-day Internet
• Integration of AccessGrid with standard control protocols
– SIP-based architecture? Desirable to move away from “Grid” protocols, to
a more standard architecture, to leverage massive commercial development
Copyright © 2005 University of Glasgow
• Evolution of IP multicast  peer-to-peer overlays
Summary
• Real-time Collaborative Environments
–
–
–
–
–
Definition and Requirements
Example: AccessGrid
History and Current Status
Underlying Technologies and Standards
Capture and Analysis of Real-time Sessions
• Research Issues and Future Directions
Copyright © 2005 University of Glasgow
Any Questions?