RTP: Multimedia Streaming over IP
Colin Perkins
USC Information Sciences Institute
Internet Multimedia
• Internet Multimedia has long history:
– RFC 741, "Network Voice Protocol", 1977
– First video experiments in the early 1980s
• Modern standards development began in 1992:
– Developing from teleconferencing systems
– Audiocast of IETF meetings
• 20 sites on 3 continents
• Precursors to RTP and the present standards
– Standardized RTP in 1996
• Widespread availability of suitable networks in the last
couple of years
• Unusual characteristics, worth exploring
Copyright © 2002 Colin Perkins
Talk Outline
• Use of IP for real-time traffic
• Protocols for real-time multimedia over IP
– Outline of signalling protocols
– Outline of media transport protocols
• RTP: Real-time Transport Protocol
– RTP data transfer protocol
– RTP control protocol
• Robustness
–
–
–
–
Playout and timing correction
Error correction
Security
Congestion control
• Security
• Conclusions
Copyright © 2002 Colin Perkins
Talk Outline
• Use of IP for real-time traffic
• Protocols for real-time multimedia over IP
– Outline of signalling protocols
– Outline of media transport protocols
• RTP: Real-time Transport Protocol
– RTP data transfer protocol
– RTP control protocol
• Robustness
–
–
–
–
Playout and timing correction
Error correction
Security
Congestion control
• Security
• Conclusions
Copyright © 2002 Colin Perkins
The IP Protocol Stack
Application programs
HTML
MIME
Media codecs
HTTP SMTP
RTP
FTP SIP RTSP
TCP
UDP
IP
ADSL
Ethernet
PPP
Twisted Pair
Optical Fibre
Wireless
• IP forms an abstraction layer
– Applications and transport
protocols above
– Assorted link technologies
below
Copyright © 2002 Colin Perkins
• Applications can't see the link
layers
– Just see the performance of
the IP layer
– Must assume lowest common
denominator behaviour
• Link layer can't tell the needs
of the application
– Just see a series of packets
– Optimisations for particular
traffic classes are risky
– Is the traffic really what you
think?
• Decoupling applications from
the network
IP Service Model
The IP service model is limited
• Best effort packet transport
• Fragmentation
• Routing and addressing
Copyright © 2002 Colin Perkins
Best Effort Packet Transport
• Performance not guaranteed
• Packets can be…
–
–
–
–
–
lost
delayed
reordered
duplicated
corrupted
…and the transport protocol
must compensate
• Checksum to catch bit errors
• Many causes of problems:
–
–
–
–
–
–
–
Congestion may cause loss
Packet corruption may cause loss
Route changeover may cause loss
Queuing delay
Multi-path IP routing may reorder
Link-layer striping may reorder
Spurious retransmission and
router bugs cause duplicates
Assumption: Significant packet loss
Copyright © 2002 Colin Perkins
Packet Loss Patterns
Packet Loss Rate (percent)
45
40
35
30
25
20
15
10
5
0
8:00
10:00
12:00
14:00
Time of day
Copyright © 2002 Colin Perkins
16:00
18:00
Timing Disruption
45º line: clocks
are synchronized
Reception time
Discontinuity due
to route change.
Queuing jitter causes
variation in inter-packet
arrival time.
Non-45º slope shows
presence of clock skew
Transmission time
Copyright © 2002 Colin Perkins
Best Effort Transport
• Performance can be bad
– Applications should be prepared to compensate
– Doesn't have to be!
• Loss and jitter can be made arbitrarily low through
careful engineering
– Most backbone networks have very good performance
– Interconnects and customer LANs are currently the main
trouble spots
– Explicit QoS doesn't appear necessary
Copyright © 2002 Colin Perkins
Fragmentation
• IP fragments packets that exceed the MTU
– Often >1500 octets
• This causes an undesirable loss multiplier effect
– Loss of one fragment means the others must also be
discarded
– Better for the application to fragment, and generate small
IP packets
Original
Fragments
Packet loss
Reconstructed
Copyright © 2002 Colin Perkins
Routing and Addressing
• Most communication is unicast
– Point-to-point
• Inefficient for many receivers
– The server must generate n
copies of the data, for n clients
– Significant scaling bottleneck
• Multicast allows the server to
scale
– Adds network complexity
Copyright © 2002 Colin Perkins
• The use of IP with best
effort transport implies
heterogeneity
• Multicast makes this an
order of magnitude worse
– Each receiver sees different
loss characteristics
– Hard to get timely feedback
in a scalable manner
Properties of IP: Summary
• Best effort, unreliable, packet delivery service
– Much of the network performs well
– Peering points and customer premises often lacking
• Provides a fragmentation service
– Typically best avoided
• Provides a multicast service
– In some parts of the network
• 10% of broadband connections
• At best: simple, effective, service
Copyright © 2002 Colin Perkins
Transport Protocols
• The IP service, by itself, is very limited
– Just (tries to) deliver packets
• Always augmented by a transport protocol
– TCP
– UDP
– (others in development)
Copyright © 2002 Colin Perkins
TCP
• Enhances the raw IP service
–
–
–
–
Point-to-point and connection oriented
Service selection through ports
Reliable, in-order, delivery
Rate adaptation to match network capacity
Copyright © 2002 Colin Perkins
TCP
– High link utilization
– Fair share between flows
• Retransmission ensures
that no data is lost
Congestion Window Size
• Rate adaptation matches
throughput to capacity
– Reliable, in-order, delivery
Time
Slow start
Copyright © 2002 Colin Perkins
Slow start Congestion
avoidance
Congestion
avoidance
UDP
• Uses the services of IP
–
–
–
–
best effort (unreliable) but timely
fragmentation
routing and addressing
unicast and multicast
• Provides ports, in addition to IP addressing, but no
other services
Copyright © 2002 Colin Perkins
Reliability/Timeliness Tradeoff
Reliable
Unreliable
Not timely
Timely
TCP
UDP
RTP
• Protocols built on unreliable
packet networks must make
a fundamental trade-off:
– Timely
– Reliable
• TCP is at one extreme, UDP
the other
Copyright © 2002 Colin Perkins
• Multimedia systems choose
their transport carefully:
– TCP for signalling
– UDP for media data
• Application level protocols
can blur the boundary
– E.g. RTP for multimedia data
– Essential for performance
Talk Outline
• Use of IP for real-time traffic
• Protocols for real-time multimedia over IP
– Outline of signalling protocols
– Outline of media transport protocols
• RTP: Real-time Transport Protocol
– RTP data transfer protocol
– RTP control protocol
• Robustness
–
–
–
–
Playout and timing correction
Error correction
Security
Congestion control
• Security
• Conclusions
Copyright © 2002 Colin Perkins
The Multimedia Protocol Framework
Two fundamental components of the framework:
• Signalling protocols
• Media transport protocols
Copyright © 2002 Colin Perkins
Signalling Protocols
The first stage of any multimedia session is signalling
• User location
• Session initiation, call setup and teardown
• Media negotiation
• Conference control
Many signalling protocols exist
• Teleconferencing: H.323
• Telephony:
SIP
• Streaming:
RTSP, SAP
Copyright © 2002 Colin Perkins
Signalling Protocols: H.323
• Original signalling protocol for IP-based multimedia
– Extension of H.320 ISDN conferencing to IP
– Tightly coupled, small group, video conferencing
• Flexible media negotiation and call control
• Reputation for complexity
– ASN.1 encoding
– Many RTT call setup
• (mostly fixed in later versions)
Copyright © 2002 Colin Perkins
Signalling Protocols: SIP
• RFC 2543 (recently updated)
• SIP is used to invite someone to join a session
– Media negotiation
– User location
– Call setup and teardown
• Considerable overlap with H.323
– More flexible integration with other Internet services
• Email, Web, streaming media, recording, agents, etc.
– More limited media negotiation and call control
• Extensions underway
– Very different style
• Protocol operation is based on HTTP
• Reuses much existing infrastructure
Copyright © 2002 Colin Perkins
Signalling Protocols: RTSP
• RFC 2326
• Designed for control of a media on demand server
– Point-to-point VCR-style remote control
– Record/play/rewind/fast-forward
• Widespread commercial use…
– RealAudio, QuickTime
• May also be useful for controlling other devices
– Voice mail
– Interactive voice response
• Leverages HTTP and SIP infrastructure
Copyright © 2002 Colin Perkins
Signalling Protocols: SAP
• RFC 2974
• A multicast announce-listen protocol for wide area
announcement of multimedia sessions
– Announcers periodically multicast SDP descriptions to a
well known group
– Inter-announcement interval is 10+ minutes
– Listeners slowly build up a cache of sessions
• Suitable for announcing long-lived public sessions
– E.g. radio/TV station, event coverage
• Mostly used with IP multicast
– Talk of use with cable networks
Copyright © 2002 Colin Perkins
Media Transport Protocols
Once the session has been setup, media flows
Convergence on a single media transport protocol for:
• Voice over IP
• Teleconferencing
• Streaming media
Real-time Transport Protocol, RTP
• Flexible, supports many codecs and media types
• Extensible to new scenarios
Copyright © 2002 Colin Perkins
The Multimedia Protocol Framework
Call control
Media negotiation
RTSP
Light weight
sessions
SIP
Media
Media
codecs
codecs
RTP
RTP
SAP
TCP
UDP
IP
IETF Multimedia Protocol Stack
Copyright © 2002 Colin Perkins
Call control
RAS
Media negotiation
H225.0
UDP
H.245
TCP
IP
ITU Teleconferencing Protocols
Design Choices
• Depending on the scenario, implement:
– An appropriate signalling protocol
• RTSP
• SIP/H.323
• SAP
– Media transport using RTP
• One or more codecs
– MPEG
– H.263
• Error correction and concealment
• Congestion Control
Copyright © 2002 Colin Perkins
Talk Outline
• Use of IP for real-time traffic
• Protocols for real-time multimedia over IP
– Outline of signalling protocols
– Outline of media transport protocols
• RTP: Real-time Transport Protocol
– RTP data transfer protocol
– RTP control protocol
• Robustness
– Playout and timing correction
– Error correction
– Congestion control
• Security
• Conclusions
Copyright © 2002 Colin Perkins
RTP: Real-time Transport Protocol
• The standard for real-time transport over IP networks
– Streaming audio and video
– Voice over IP
• Published as an IETF proposed standard
–
–
–
–
RFCs 1889 and 1890 in January 1986
Adopted by ITU as part of H.323
Adopted by 3GPP for next generation cellular telephony
Widespread use in streaming: QuickTime, Real, Microsoft
• (HTTP streaming still common)
• Recently revised for draft standard status
– Work complete, awaiting publication
– Changes include:
• Clarifications and bug-fixes based on experience
• Scalability improvements
• Support for new codecs
– 100% backwards compatible
Copyright © 2002 Colin Perkins
Philosophy of RTP
• The challenge:
– build a mechanism for robust, real-time media delivery
above an unreliable and unpredictable transport layer
– without changing the transport layer
Push responsibility for media
Make the system robust to
delivery onto the end-points
network problems; media
where possible
data should be loss tolerant
The end-to-end argument
Copyright © 2002 Colin Perkins
Application level framing
The End-to-end Argument
• Two options for ensuring reliability
– Pass responsibility hop-by-hop, along with the data
• Email
– Responsibility remains with the end points, which ensure
delivery even if the intermediate steps are unreliable
• Both TCP and RTP take the second approach
• Consequences:
– Intelligence tends to "bubble-up" the protocol stack to the
end points
– The intermediate systems can be simple, and need not be
robust
• They can simply discard data they cannot deliver, since it will
be recovered end-to-end
• The network is dumb, but end-points are smart
Copyright © 2002 Colin Perkins
Application Level Framing
• Only the application has sufficient knowledge of its data
to make an informed decision about how that data should
be transported
• Implications:
– The transport protocol should accept data in application
meaningful chunks ("ADUs")
• The application must understand the data,
• The application must be able to process ADUs independently, in
arbitrary order, and in the presence of loss
– The transport protocol should expose details of delivery,
allowing the applications to react intelligently if there are
problems
•
•
•
•
Blind retransmission is not always appropriate
Maybe the data is stable, and an updated version can be sent
Maybe the data is obsolete, and doesn't need to be resent
Maybe an alternate representation of the data can be sent
Copyright © 2002 Colin Perkins
Philosophy of RTP
• The philosophy of RTP implies smart, network-aware,
applications that are capable of reacting to problems
end-to-end.
– Both sender and receiver are intelligent
– The network is dumb and can be unreliable
• Similar principles apply to the signalling protocols
– Mostly end-to-end operation, limited support for network
state
• Fits well with the IP service
• Contrast with traditional applications:
– Telephone network is smart, end-points are dumb
– MPEG sender is smart, receiver relatively dumb
Copyright © 2002 Colin Perkins
Protocol Components
Payload
Payload
Payload
Payload
Format
Format
Format
Format
RTP Profile
RTP Data Transfer Protocol
UDP
IP
Copyright © 2002 Colin Perkins
RTP Control Protocol
Protocol Components
RTP Data Transfer Protocol
• Transports application data
units
– Audio
– Video
• One RTP stream transports
each media type
Copyright © 2002 Colin Perkins
• Provides:
–
–
–
–
Source identification
Payload identification
Media sequencing
Timing recovery
• Extensions allow for error
correction
Protocol Components
RTP Control Protocol
• Reception quality feedback
– Packet loss fraction
– Average timing jitter
• Optional source description
– Name, location, email
address, phone number
Copyright © 2002 Colin Perkins
• Mapping from media clock to
external time-base
– E.g. for lip synchronization
• Loosely coupled membership
management
Protocol Components
Payload
Payload
Payload
Payload
Format
Format
Format
Format
• Provide the adaptation layer
between a particular codec
and RTP
• Optimised for robustness to
packet loss
Copyright © 2002 Colin Perkins
• Many payload formats exist,
with more being developed:
– H.261, H.263, M-JPEG,
MPEG-2, MPEG-4, BT.656,
SMPTE-292
Protocol Components
RTP Profile
• Define use of RTP in particular
application scenarios
– "Reasonable defaults"
– Adaptation to unusual conditions
• Single source multicast
• Operation without back channel
• Authenticated and secure operation
Copyright © 2002 Colin Perkins
• Provides a namespace
for payload formats
Combining the Pieces
• A multimedia session comprises several RTP sessions
– One for each media type
• Each RTP session:
– Implements a particular RTP profile
– Includes an RTP data flow
• Transporting a single media type according to one or more
payload formats
– E.g. Audio switching between G.729 and Fax
– E.g. Video using MPEG
– Includes an RTP control protocol flow
• Providing reception quality feedback, user information, etc.
– Is defined by:
• Source and destination IP addresses
• A pair of UDP ports: one for RTP, one for RTCP
Copyright © 2002 Colin Perkins
RTP Data Transfer Protocol
• The RTP Data Transfer
Protocol delivers a single
media stream from sender
to one, or more, receivers
– Few assumptions about the
underlying transport
– Usually runs over UDP/IP
• Typically implemented in an
application or as a library
– User level, not part of the
kernel
Copyright © 2002 Colin Perkins
RTP provides:
• Source identification
• Media identification
• Media transport
– Padding, if necessary
– Marking of significant
events
• Sequencing
• Timing recovery
Packet Format
V PX
CC
M
PT
Sequence Number
Timestamp
Synchronization source (SSRC) identifier
Contributing source (CSRC) identifiers
Payload data
Padding
Copyright © 2002 Colin Perkins
Source Identification
• Each packet carries a 32 bit
synchronization source
– Randomly chosen at startup, with collision detection
• Provides a transport layer
independent identifier
– Supports gateways
– IPv4, IPv6, ATM
• Identifies a single timesynchronized media flow
– Mapped to a persistent
identifier using RTCP
Copyright © 2002 Colin Perkins
V
P X
CC
M
PT
Sequence Number
Timestamp
Synchronization source (SSRC) identifier
Contributing source (CSRC) identifiers
Payload data
Padding
Source Identification
• Each packet may include a
list of contributing sources
– Allows data from up to 16
sources to be identified
– Each CSRC is the SSRC of a
mixed participant
V
P X
CC
M
PT
Sequence Number
Timestamp
Synchronization source (SSRC) identifier
Contributing source (CSRC) identifiers
Payload data
• Allows RTP to support mixers
and translators
– Mixers combine several flows
into one
• E.g. Conferencing MCU
– Translators change the
format of a flow, or gateway
between different networks
• Transcode to a lower bit-rate
• Gateway between unicast
and multicast
Copyright © 2002 Colin Perkins
Padding
Communication Models
• Mixers and translators greatly expand
the range of communications models
available to RTP
RTP end point
RTP Translator or mixer
Multicast group, the
network replicates
data as necessary, with
no translation or mixing.
Point-to-point
communication
via unicast
Four participants
communicating via
a multicast group
Replicated unicast: a
group of three using an
RTP translator/mixer to
mediate communications.
Copyright © 2002 Colin Perkins
Translated: multicast to
unicast. Two participants
communicating via a
multicast group, with a
third linked to the session
by an RTP translator.
Media Identification
• Each packet carries a 7 bit
payload type field
• Mapped to a payload format
during session setup
– Allows flexible signalling of
codec type and parameters
– Mapping can be static, if the
profile allows
• Each flow carries only one
type of media
– Only audio, or only video
• The payload type allows the
sender to switch between a
set of payload formats
– E.g. a flow carries only audio
but may switch between fax
and voice at any time
Copyright © 2002 Colin Perkins
V
P X
CC
M
PT
Sequence Number
Timestamp
Synchronization source (SSRC) identifier
Contributing source (CSRC) identifiers
Payload data
Padding
Media Transport and Payload Formats
• Packets contain a block of
payload data, described by
a payload format
• Payload formats describe the
mapping between codec output
and RTP packets
– Chosen so that each packet is
independently decodable
– Application level framing
• The payload data typically
includes a payload header to
ease parsing
– E.g. The H.261 payload format
copies some information from
the GOB header so each set of
macro-blocks can be decoded
independently
Copyright © 2002 Colin Perkins
V
P X
CC
M
PT
Sequence Number
Timestamp
Synchronization source (SSRC) identifier
Contributing source (CSRC) identifiers
Payload data
Padding
Media Transport: Marker
• Each packet includes a bit to
mark significant events
– Start of talk spurt for audio
– Last packet of frame for video
• A hint that special processing
may be required
V
P X
CC
M
PT
Sequence Number
Timestamp
Synchronization source (SSRC) identifier
Contributing source (CSRC) identifiers
Payload data
Padding
Copyright © 2002 Colin Perkins
Media Transport: Padding
• Each packet may be padded
beyond its natural size
• Rarely used, but needed by
some encryption algorithms
– DES in CBC modes operates
on 64 bit blocks
• The SRTP profile provides a
better security solution
Copyright © 2002 Colin Perkins
V
P X
CC
M
PT
Sequence Number
Timestamp
Synchronization source (SSRC) identifier
Contributing source (CSRC) identifiers
Payload data
Padding
Sequencing
• Each packet contains a 16
bit sequence number
– Random initial value
– Increments monotonically
with each packet sent
– Wraps around to zero when
the limit is reached
• Used to detect packet loss
– Is not used to determine
playout order
• Basic RTP does not provide
error correction
– The receiver is expected to
conceal the error, and to
continue processing
– Extensions provide forward
error correction and limited
retransmission
Copyright © 2002 Colin Perkins
V
P X
CC
M
PT
Sequence Number
Timestamp
Synchronization source (SSRC) identifier
Contributing source (CSRC) identifiers
Payload data
Padding
Timing Recovery
• Each packet contains a 32
bit timestamp
• Indicates the sampling
instant of the oldest
payload data
V
P X
CC
M
PT
Sequence Number
Timestamp
Synchronization source (SSRC) identifier
Contributing source (CSRC) identifiers
– Determines playout order
Payload data
• The clock rate is defined by
the payload format:
– Audio clock is sampling rate
– Video clock is 90kHz,
indicating the frame time
– Mapping to codec time-base
is also defined
• No requirements on stability
or accuracy of clock
– Implies receiver adaptation
Copyright © 2002 Colin Perkins
Padding
• Time code not carried
directly, but mapping to
wall clock time via RTCP
sender reports
RTP Control Protocol (RTCP)
• Each RTP data flow has an associated control flow
• The control flow provides:
– Time-base management
– Quality of service feedback
– Member identification and management
Copyright © 2002 Colin Perkins
Time-base Management
• Timestamps map between the RTP timeline and NTP
“wall-clock” time
– If a common NTP clock is used for multiple streams, a
receiver can synchronize them
• No explicit transport of SMPTE (or similar) time-codes
– Can be derived from NTP timestamps
• Accuracy limited by NTP resolution, unless external clock
provided
• RTSP provides a mapping function
• Also allows receivers to estimate data/packet rate and
possibly clock skew
Copyright © 2002 Colin Perkins
Reception Quality Reporting
• Quality of service feedback from each receiver:
–
–
–
–
–
Loss fraction
Cumulative number of packets lost
Highest sequence number received
Inter-arrival jitter
Round-trip time
• Many uses:
– Loss rate can be used to select amount of FEC to employ
– Jitter gives estimate of playout buffer delay at receiver
Copyright © 2002 Colin Perkins
Membership Management
• RTCP provides a canonical
name, mapping SSRC to a
persistent identifier
• Augments the membership
management provided by
the signalling protocol
– Used to associate streams
for synchronisation
– Primarily using the explicit
leave indication
• RTCP can optionally deliver
source description data:
–
–
–
–
–
Name
Email address
Phone number
Location
(extend with metadata)
• Provides loosely coupled
presence information
– Explicit leave message
Copyright © 2002 Colin Perkins
RTCP reporting interval
• RTCP is a low-rate reporting protocol
– Not intended for uses that require instant feedback
– Scalable to very large sessions
• Statistical summary of group conditions
• Packets are sent periodically
– The interval between packets is adjusted to limit RTCP to
once per 5 seconds, or 5% of the data rate
– Randomized to avoid synchronization
Copyright © 2002 Colin Perkins
RTP: Summary
• Flexible and extensible media transfer protocol
– Supports a range of codecs
– Allows detection of network problems
– Allows recovery of media timing
• Associated, low rate, reporting of reception quality,
time-base, and presence information
Copyright © 2002 Colin Perkins
Talk Outline
• Use of IP for real-time traffic
• Protocols for real-time multimedia over IP
– Outline of signalling protocols
– Outline of media transport protocols
• RTP: Real-time Transport Protocol
– RTP data transfer protocol
– RTP control protocol
• Robustness
– Playout and timing correction
– Error correction
– Congestion control
• Security
• Conclusions
Copyright © 2002 Colin Perkins
Robustness
• RTP operates over UDP/IP
– Best effort delivery
– Packets can be lost, delayed, reordered, duplicated, etc.
• Applications are responsible for correct playout
– Timing recovery
– Error concealment
– Congestion control
Copyright © 2002 Colin Perkins
Timing Recovery
• The network can seriously
disrupt media timing
• Receivers must include a
jitter compensation buffer
to reconstruct the media for
playout
Original media stream
Sender
Router
Constant inter-packet spacing
Internet
Network induces
timing jitter into
the media stream
Received media stream
Router
Receiver
Variable inter-packet spacing
Copyright © 2002 Colin Perkins
Playout and Timing Correction
1st talk spurt
2nd talk spurt
Transmission
Jitter affects inter-packet timing
Network
Transit
Reception
Playout
Buffer
Network transit delay
Playout
Playout buffering delay added
to compensate for jitter.
Copyright © 2002 Colin Perkins
Packet discarded
due to late arrival
Playout and Timing Correction
• RTP does not specify a
standard playout buffer
or timing reconstruction
algorithm
– Provides the necessary
information; allowing
product differentiation
• Compare with MPEG, where
the buffer model is closely
defined
Copyright © 2002 Colin Perkins
• Many trade-offs to consider:
– latency versus quality
– speed of reaction to change
– buffering ability
• Typical design:
– Streaming applications use
large delay (10+ seconds)
– Interactive applications try
to keep delay low (tens of
milliseconds)
Error Correction
• Limited Retransmission
– RTCP feedback profile
• Unequal error protection
• Interleaving
• Forward Error Correction
– Media specific
– Media independent
Reliable
Unreliable
Not timely
Timely
TCP
UDP
RTP
⇒ Add limited reliability to RTP
Copyright © 2002 Colin Perkins
Retransmission in RTP
• Why must retransmission be limited?
– Timely versus reliable
– We don't want to re-invent TCP
• How to implement retransmission?
– RTCP provides a back channel
– Modify the RTCP timing rules to allow early feedback
• Keep the fundamental scaling rules to avoid potential for
implosion
Copyright © 2002 Colin Perkins
RTCP Feedback Profile
• RTCP reports sent as usual
• Feedback can be sent early
– Ignore 5 second rule
– Borrow bandwidth from the
next reporting interval
– Delays next report
– Send minimal report packet
Immediate
FB mode
2
Report every
relevant event
immediately
Early RTCP
mode
Report many
of the events
but not all
Send feedback + regular RTCP packets
Copyright © 2002 Colin Perkins
• Transport layer feedback
– NACK, ACK sequence number
• Payload specific feedback
– Reference picture selection
– MPEG-4 NEWPRED
• Under development in IETF
Regular
RTCP mode
Just regular
RTCP packets
Group
size
Forward Error Correction
• Retransmission relies on
feedback from receivers
• Alternative:
– Sender adds redundant data
to the media stream
– Receivers use this to correct
errors, without contacting
sender
– Must request that lost
packets are resent
• Works well in many cases
• Two scenarios where it is
inefficient:
• Forward Error Correction
– Well known technique at the
link layer
– Also be applicable at the IP
level and above
– Round-trip time is large
– Many receivers, independent
loss events
RTP
Copyright © 2002 Colin Perkins
RTP
RTP
FEC
RTP
RTP
RTP
FEC
RTP
Media Specific FEC
• Some codecs may naturally
be loss tolerant
• Design payload format to
take advantage of this
Copyright © 2002 Colin Perkins
Media Specific FEC
• Some codecs may naturally
be loss tolerant
• Design payload format to
take advantage of this
– Audio/video redundancy
– RFC 2198, AMR, H.263+
1
2
3
4
Original Stream
1
2
3
4
Redundant data added
1
2
4
Packet lost in transport
1
2
4
Reconstructed stream
Copyright © 2002 Colin Perkins
3
Media Independent FEC
• Media specific FEC needs to
be produced by the encoder
• Better if the FEC can be derived
from pre-compressed media
– Part of the payload format
– Either off-line or on-line
– Less load on sender
– Perhaps less network efficient
⇒ Compression performed at
time of transmission
⇒ To pick appropriate FEC
⇒ Undesirable
⇒ Too much load on sender to
support many streams
Compress
Source image
Packetize
RTP
RTP
RTP
FEC
Copyright © 2002 Colin Perkins
RTP
FEC
Parity FEC
• Parity codes to protect serial
communication well known
• Can apply same technique to
packet networks
– Generate parity packet
Bit stream A
Parity code: A XOR B
Bit stream B
Copyright © 2002 Colin Perkins
1
1
1
0
0
0
1
• Standard for parity FEC:
– RFC 2733
– Flexible parity operation
• Standards for Reed-Solomon
coding under development
1
1
1
0
0
0
1
0
0
1
1
0
1
0
1
0
1
0
1
1
1
1
0
1
0
1
1
1
0
0
1
1
0
1
0
Transmission loses B
Calculate parity to recover
B = A XOR (A XOR B)
Unequal Error Protection
• Not all data in the packets
is equally important
– Headers and codec state
updates are vital
– Media data is of variable
importance
• Data used for predication
• Data used in a single frame
• Some links have high bit
error rate
– Causes packet corruption
– Detected by UDP checksum
– Packet discarded
• Seriously impacts wireless
link performance
– Cellular, especially
Partial Checksum
at the UDP level
Copyright © 2002 Colin Perkins
Interleaving
• Can interleave, to make
bursts of loss appear as
random loss
• Packet loss concealment
and correction work best
when loss is isolated
– Single packet losses
– Adds considerable delay
• Packet loss on the Internet
is bursty
1
2
3
4
5
6
7
8
9
1
5
9
13
2
6
10
14
3
1
5
9
13
2
6
10
14
1
2
4
5
6
Copyright © 2002 Colin Perkins
8
9
• Popular with streaming apps
– Part of many audio payload
formats
10 11 12 13 14
7
10
11 15
15 16
Original stream
4
8
12 16
Interleaved stream
4
8
12 16
Packet loss
12 13 14
16
Reconstructed stream
Error Correction: Summary
• Extensive, and ongoing,
research and standards
work developing error
correction for RTP
–
–
–
–
–
Retransmission
Media/codec specific FEC
Media independent FEC
Partial checksum
Interleaving
⇒ Network doesn't have to be
perfect
• If it's okay for data traffic,
it's probably okay for video
• Unless you have very strict
requirements
⇒ Acceptable to overprovision
for QoS
• No real need for RSVP or
differentiated services
⇒ Well designed applications
can tolerate significant loss
• Often, 5% loss acceptable
Copyright © 2002 Colin Perkins
Congestion Control
But…
… the preceding assumed
traffic is well behaved
… assumed flows respond to
congestion in the network
… or, QoS and flow admission
control employed
Copyright © 2002 Colin Perkins
Congestion Control
– No admission control
– The network accepts all
packets and tries to deliver
them.
• However, no guarantee of
delivery provided
– Excess packets discarded if
links become congested.
• The transport protocol must
detect loss, and reduce its
rate to allow the congestion
to clear
– TCP does this automatically
– RTP does not
Copyright © 2002 Colin Perkins
Normal operation
Congestion Collapse
Packets delivered
• An IP network provides a
best-effort packet-switched
service.
Packets sent
Congestion Control
– Possible rate changes
depend on the codec
– Complex feedback loop
between codec and network
• For RTP, implies senders
should observe receiver
feedback
– If loss fraction is non-zero,
consider sending less
– As loss decreases, consider
sending faster
Copyright © 2002 Colin Perkins
Normal operation
Congestion Collapse
Packets delivered
• Adaptation must be done by
the application
Packets sent
TCP Friendly Rate Control
• Possible to predict the longterm average throughput of
TCP
s
T=
R
3p
2p
+ 3 p(1 + 32 p 2 ) ⋅ Trto
8
3
• Derive throughput in terms
of observed loss rate, RTT
and packet size
– Measurable qualities in RTP
• Adapt sending rate to match
– Driven by reception of RTCP
– Drives codec operation
Copyright © 2002 Colin Perkins
• The "best current practice"
congestion control scheme
for multimedia flows
• With appropriate parameters:
– Fair to TCP on average
– Slowly changing rate
Limitations of Congestion Control
• TCP friendly algorithms are
new, and evolving
– Very limited deployment
– Not clear that they have
reached their final form
• Interactions between codec
and network are not well
defined
– Unclear how slow response,
or limited adaptability, will
impact fairness
Copyright © 2002 Colin Perkins
• Human factors aspects play a
key role
– Congestion control implies
variable quality
– Subjectively very annoying,
unless the rate of change is
slow
– Can have a significant impact
on congestion control
Talk Outline
• Use of IP for real-time traffic
• Protocols for real-time multimedia over IP
– Outline of signalling protocols
– Outline of media transport protocols
• RTP: Real-time Transport Protocol
– RTP data transfer protocol
– RTP control protocol
• Robustness
– Playout and timing correction
– Error correction
– Congestion control
• Security
• Conclusions
Copyright © 2002 Colin Perkins
Security
Several aspects to multimedia security
• Confidentiality of the media
RTP can help here
• Authentication of the sender
• Watermarking
• Storage
Copyright © 2002 Colin Perkins
Security in RTP
• Basic RTP provides limited
security
– Packets may be encrypted
• DES is specified; algorithm
may be negotiated during
session setup
– Does not support sender
authentication
• A secure RTP profile is under
development
– Encryption for confidentiality
• Encrypts only the payload data,
not the headers
• AES in counter or F8 mode
• Robust to bit errors, allows
header compression, suitable
for cellular wireless
– Sender authentication
• Adds a trailer to each packet,
containing authentication code
• HMAC-SHA1
Copyright © 2002 Colin Perkins
Watermarking, Storage and DRM
• RTP is concerned only with
the transmission of media
• Does not consider:
– The contents of the video
image
– How the image is captured,
generated and stored
Copyright © 2002 Colin Perkins
• Implications:
– Receivers may store the payload
• RTP cannot influence the process
• The copy may not be perfect
– Watermarks or other embedded
data can be inserted at source
• But must be robust to packet loss
Talk Outline
• Use of IP for real-time traffic
• Protocols for real-time multimedia over IP
– Outline of signalling protocols
– Outline of media transport protocols
• RTP: Real-time Transport Protocol
– RTP data transfer protocol
– RTP control protocol
• Robustness
– Playout and timing correction
– Error correction
– Congestion control
• Security
• Conclusions
Copyright © 2002 Colin Perkins
Conclusions
• IP provides a non-optimal service for video transport
– Careful network engineering can solve many problems
• Multimedia protocol framework comprises:
– Signalling: H.323, SIP, RTSP, SAP
– Media Transport: RTP + codecs
• RTP provides:
– Robust, flexible and extensible media transport
– Range of error correction schemes
– Range of security solutions
• Limitations:
– Congestion control
Copyright © 2002 Colin Perkins
For More Information
IETF Audio/Video Transport Working Group
http://www.ietf.org/html.charters/avt-charter.html
Colin Perkins
http://www.east.isi.edu/~csp/
Copyright © 2002 Colin Perkins