Lecture 9: Multimedia Transmission
Protocol
Hongli Luo, CEIT
Multimedia Transmission Protocol
 RTSP
 RTP
 RTCP
 SIP
 Socket Programming
User Control of Streaming Media: RTSP
HTTP
 does not target multimedia
content
 no commands for fast
forward, etc.
RTSP: RFC 2326
 Allow the media player and
server to exchange playback
control information
 Allows a media player to
control the transmission of a
media stream
 client-server application
layer protocol
 user control: rewind, fast
forward, pause, resume,
repositioning, etc…
What it doesn’t do:
 doesn’t define how
audio/video is encapsulated
for streaming over network
 doesn’t restrict how streamed
media is transported (UDP or
TCP possible)
 doesn’t specify how media
player buffers audio/video
RTSP: out of band control
FTP uses an “out-of-band”
control channel:
 file transferred over one
TCP connection.
 control info (directory
changes, file deletion,
rename) sent over
separate TCP
connection
 “out-of-band”, “in-band”
channels use different
port numbers
RTSP messages also sent
out-of-band:
 RTSP control
messages use different
port numbers than
media stream: out-ofband.
 port 554
 Over TCP or UDP
 media stream is
considered “in-band”.
 Adopted by
RealNetworks
FTP: the file transfer protocol
user
at host
FTP
FTP
user
client
interface
local file
system
file transfer
FTP
server
remote file
system
 transfer file to/from remote host
 client/server model


client: side that initiates transfer (either to/from remote)
server: remote host
 ftp: RFC 959
 ftp server: port 21
FTP: separate control, data connections
TCP control connection
port 21
 FTP client contacts FTP server at





port 21, TCP is transport protocol
client authorized over control
connection
client browses remote directory by
sending commands over control
connection.
when server receives file transfer
command, server opens 2nd TCP
connection (for file) to client
after transferring one file, server
closes data connection.
server opens another TCP data
connection to transfer another file.
FTP
client
TCP data connection
port 20
FTP
server
 The control session remains open
throughout the duration of the user
session
 control connection: “out of band”
 FTP server maintains “state”:
current directory, earlier
authentication
RTSP Example
Scenario:
 metafile communicated to web browser
 browser launches player
 player sets up an RTSP control connection, data
connection to streaming server
Metafile Example
<title>Twister</title>
<session>
<group language=en lipsync>
<switch>
<track type=audio
e="PCMU/8000/1"
src = "rtsp://audio.example.com/twister/audio.en/lofi">
<track type=audio
e="DVI4/16000/2" pt="90 DVI4/8000/1"
src="rtsp://audio.example.com/twister/audio.en/hifi">
</switch>
<track type="video/jpeg"
src="rtsp://video.example.com/twister/video">
</group>
</session>
RTSP Operation
RTSP Exchange Example
C: SETUP rtsp://audio.example.com/twister/audio RTSP/1.0
Transport: rtp/udp; compression; port=3056; mode=PLAY
S: RTSP/1.0 200 1 OK
Session 4231
C: PLAY rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0
Session: 4231
Range: npt=0S: RTSP/1.0 200 2 OK
Session 4231
C: PAUSE rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0
Session: 4231
Range: npt=37
S: RTSP/1.0 200 3 OK
Session 4231
C: TEARDOWN rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0
Session: 4231
S: RTSP/1.0 200 4 OK
Session 4231
Real-Time Protocol (RTP)
 RTP specifies packet
structure for packets carrying
audio, video data



Audio: PCM, GSM, MP3
Video: MPEG and h.263
Proprietary audio and video
formats
 RTP packet provides



payload type
identification
packet sequence
numbering
time stamping
 RFC 3550
 RTP runs in end systems
 RTP packets encapsulated in
UDP segments
 interoperability: if two Internet
phone applications run RTP,
then they may be able to work
together
RTP runs on top of UDP
RTP libraries provide transport-layer interface
that extends UDP:
• port numbers, IP addresses
• payload type identification
• packet sequence numbering
• time-stamping
RTP Example
 consider sending 64
kbps PCM-encoded
voice over RTP.
 application collects
encoded data in
chunks, e.g., every 20
msec = 160 bytes in a
chunk.
 audio chunk + RTP
header form RTP
packet, which is
encapsulated in UDP
segment
 RTP header indicates
type of audio encoding
in each packet

sender can change
encoding during
conference.
 RTP header also
contains sequence
numbers, timestamps.
RTP and QoS
 RTP does not provide any mechanism to ensure
timely data delivery or other QoS guarantees.
 RTP does not provide
 Timely delivery of data
 QoS guarantees
 Guarantee delivery of packets
 Prevention of out-of-order delivery of packets
 RTP encapsulation is only seen at end systems (not)
by intermediate routers.
 routers providing best-effort service, making no
special effort to ensure that RTP packets arrive at
destination in timely matter.
RTP Header (12 bytes)
Payload Type (7 bits): Indicates type of encoding currently being
used. If sender changes encoding in middle of conference, sender
informs receiver via payload type field.
•Payload type 0: PCM mu-law, 64 kbps
•Payload type 3, GSM, 13 kbps
•Payload type 7, LPC, 2.4 kbps
•Payload type 26, Motion JPEG
•Payload type 31. H.261
•Payload type 33, MPEG2 video
Sequence Number (16 bits): Increments by one for each RTP packet
sent, and may be used to detect packet loss and to restore packet
sequence.
RTP Header (2)

Timestamp field (32 bytes long): sampling instant of
first byte in this RTP data packet



Receiver can use it to remove packet jitter and to provide
synchronous playout
for audio, timestamp clock typically increments by one for
each sampling period (for example, each 125 usecs for 8
KHz sampling clock)
if application generates chunks of 160 encoded samples,
then timestamp increases by 160 for each RTP packet when
source is active. Timestamp clock continues to increase at
constant rate when source is inactive.

SSRC field (32 bits long):

Miscellaneous fields (9 bits)
identifies source of RTP stream.
Each stream in RTP session should have distinct SSRC.
Developing Software Applications with RTP
 Two approaches to develop an RTP-based
networked applications


Incorporate RTP by hand –
• write the code that performs RTP encapsulation at the
sender side and RTP decapsulation at the client side
Use existing RTP libraries (for C programmers) and Java
classes (for Java programmers)
• The libraries and classes perform the encapsulation and
decapsulation for the application
Incorporate RTP by hand - example
 A server that encapsulates stored video frames into
RTP packets





grab video frame,
add RTP headers to frame and generate an RTP packet
create UDP segments, send segments to UDP socket
include seq numbers and time stamps
The API is the standard UDP socket API
 A client decapsulates the RTP packet and display the
video frame
 RTSP


Client: issue setup/play/pause/teardown commands
Server: accepts the requests and take actions
 RTP does not mandate a specific port number.
 The application developer specifies the port number
for the two sides of the application.
 Use existing Java RTP class to implement (or C RTP
library for C programmers) to implement the RTP.

The sender application provides
• media chunk, payload-type number, SSRC, timestamp,
destination port number, destination IP

Java Media Framework (JMF) includes a complete RTP
implementation
Real-Time Control Protocol (RTCP)
 works in conjunction with
RTP.
 each participant in RTP
session periodically
transmits RTCP control
packets to all other
participants.
 each RTCP packet
contains sender and/or
receiver reports

report statistics useful to
application: # packets
sent, # packets lost,
interarrival jitter, etc.
 feedback can be used
to control performance

sender may modify its
transmissions based on
feedback
RTCP - Continued
each RTP session: typically a single multicast address; all RTP /RTCP packets
belonging to session use multicast address.


RTP, RTCP packets distinguished from each other via distinct port numbers.
 RTCP port number is set to be equal to the RTP port number plus one

to limit traffic, each participant reduces RTCP traffic as number of
conference participants increases
RTCP Packets
Receiver report packets:
 Receiver aggregates its reception report into a single
RTCP packet
 The packet is sent into the multicast tree that connects
all the session’s participants.
 Fields in reception report:




SSRC of RTP stream
fraction of packets lost – the sender can switch to different
encoding rates
last sequence number
average interarrival jitter – a smoothed estimate of the variation
in the interarrival time between successive packets in the RTP
stream
RTCP Packets
Sender report packets:
 Sender creates and transmits RTCP sender report
packets
 The packets include information such as




SSRC of RTP stream,
Time stamp, wall clock time (current time) of the most recently
generated RTP packet in the stream
number of packets sent,
number of bytes sent
 Sender reports can be used to synchronize different
media streams within a RTP session.
RTCP Packets
Source description packets:
 Sender also creates and transmits source description
packets.
 Includes e-mail address of sender, sender's name,
SSRC of associated RTP stream, application that
generates the RTP stream
 provide mapping between the SSRC and the user/host
name
 RTCP packets are stackable


Receiver reception reports, sender reports, and source
descriptors can be concatenated into a single packet
The RTCP packet is then encapsulated into a UDP segment
Synchronization of Streams
 RTCP can synchronize
different media streams
within a RTP session
 consider videoconferencing
app for which each sender
generates one RTP stream
for video, one for audio.
 timestamps in RTP packets
tied to the video, audio
sampling clocks
 not tied to wall-clock time
(real time)
 each RTCP sender-report
packet contains (for most
recently generated packet in
associated RTP stream):


timestamp of RTP packet
wall-clock time for when
packet was created.
 receivers uses association to
synchronize playout of audio,
video
RTCP Bandwidth Scaling
 RTCP attempts to limit its
traffic to 5% of session
bandwidth.
Example
 Suppose one sender,
sending video at 2 Mbps.
Then RTCP attempts to limit
its traffic to 100 Kbps.
 RTCP gives 75% of rate to
receivers; remaining 25% to
sender
 75 kbps is equally shared
among receivers:

with R receivers, each receiver
gets to send RTCP traffic at
75/R kbps.
 sender gets to send RTCP traffic
at 25 kbps.
 participant determines RTCP
packet transmission period by
calculating avg RTCP packet
size (across entire session) and
dividing by allocated rate
RTCP Bandwidth Scaling (2)
 The period for transmitting RTCP packets for a sender is
T = (number of senders ) * (avg. RTCP packet size)
/ (.25 * .05 * session bandwidth)
 The period for transmitting RTCP packets for a receiver
is
T = (number of senders ) *(avg. RTCP packet size)
/ (.75 * .05 * session bandwidth)
SIP: Session Initiation Protocol [RFC 3261]
SIP long-term vision:
 all telephone calls, video conference calls take place
over Internet
 people are identified by names or e-mail addresses,
rather than by phone numbers
 you can reach callee, no matter where callee roams,
no matter what IP device callee is currently using –
computer or PDA
SIP Services
 Setting up a call
between caller and
callee, SIP provides
mechanisms ..
 for caller to let callee
know she wants to
establish a call
 so caller, callee can
agree on media type,
encoding
 to end call
 determine current IP
address of callee:


Callee has dynamic IP by
DHCP or has multiple IP
devices
maps mnemonic
identifier to current IP
address
 call management:
 add new media streams
during call
 change encoding during
call
 invite others
 transfer, hold calls
Setting up a call to known IP address
Bob
Alice
167.180.112.24
INVITE bob
@193.64.2
10.89
c=IN IP4 16
7.180.112.2
4
m=audio 38
060 RTP/A
VP 0
193.64.210.89
port 5060
port 5060
Bob's
terminal rings
200 OK
.210.89
c=IN IP4 193.64
RTP/AVP 3
3
m=audio 4875
ACK
port 5060
Bob’s 200 OK message
indicates his port number,
IP address, preferred
encoding (GSM)

SIP messages can be
sent over TCP or UDP;
here sent over RTP/UDP.

m Law audio
port 38060
GSM
Alice’s SIP invite
message indicates her
port number, IP address,
encoding she prefers to
receive (PCM ulaw)

port 48753
default
is 5060.
time
time
SIP port number
Setting up a call (more)
 SIP is an out-of-band protocol

SIP messages are sent and
received in sockets different
from those for media data
 SIP messages are ASCII-
readable and resemble HTTP
messages
 SIP requires all messages to
be acknowledged

It can run over UDP or TCP
 media can be sent over RTP or
some other protocol
 codec negotiation:
suppose Bob doesn’t
have PCM ulaw encoder.
 Bob will instead reply
with 606 Not Acceptable
Reply, listing his
encoders Alice can then
send new INVITE
message, advertising
different encoder
 rejecting a call
 Bob can reject with
replies “busy,” “gone,”
“payment required,”
“forbidden”

SIP Addresses
 Bob’s SIP address is sip:bob@193.64.210.89
 When Alice’s SIP device sends an INVITE message,
the message would include this email-like address
 The SIP infrastructure would then route the message
to the IP advice that Bob is currently using
 Other possible forms for SIP address


Phone number
First/last name
 SIP address can be included in Web page
Example of SIP message
INVITE sip:bob@domain.com SIP/2.0
Via: SIP/2.0/UDP 167.180.112.24
From: sip:alice@hereway.com
To: sip:bob@domain.com
Call-ID: a2e3a@pigeon.hereway.com
Content-Type: application/sdp
Content-Length: 885
c=IN IP4 167.180.112.24
m=audio 38060 RTP/AVP 0
Notes:
 HTTP message syntax
 sdp = session description protocol
 Call-ID is unique for every call.
Here we don’t know
Bob’s IP address.
Intermediate SIP
servers needed.

Alice sends, receives
SIP messages using
SIP default port 506

Alice specifies in
Via: IP address of the
device, header that
SIP client sends,
receives SIP messages
over UDP

Name translation and user locataion
 caller wants to call
callee, but only has
callee’s name or e-mail
address.
 need to get IP address
of callee’s current host:



user moves around
DHCP protocol
user has different IP
devices (PC, PDA, car
device)
 result can be based on:
 time of day (work, home)
 caller (don’t want boss to
call you at home)
 status of callee (calls sent
to voicemail when callee is
already talking to someone)
Service provided by SIP
servers:
 SIP proxy server
 SIP registrar server
SIP Proxy
 Alice sends invite message to her proxy server
 contains address sip:bob@domain.com
 proxy responsible for routing SIP messages to callee
 possibly through multiple proxies.
 callee sends response back through the same set of
proxies.
 proxy returns SIP response message to Alice

contains Bob’s IP address
 proxy analogous to local DNS server
SIP Registrar
 when Bob starts SIP client, client sends SIP REGISTER
message to Bob’s registrar server
(similar function needed by Instant Messaging)
 Often SIP registrars and SIP proxies are run on the
same host
Register Message:
REGISTER sip:domain.com SIP/2.0
Via: SIP/2.0/UDP 193.64.210.89
From: sip:bob@domain.com
To: sip:bob@domain.com
Expires: 3600
Example
Caller jim@umass.edu
with places a
call to keith@upenn.edu
SIP registrar
upenn.edu
SIP
registrar
eurecom.fr
2
(1) Jim sends INVITE
message to umass SIP
proxy. (2) Proxy forwards
request to upenn
registrar server.
(3) upenn server returns
redirect response,
indicating that it should
try keith@eurecom.fr
SIP proxy
umass.edu
1
3
4
5
7
8
6
9
SIP client
217.123.56.89
SIP client
197.87.54.21
(4) umass proxy sends INVITE to eurecom registrar. (5) eurecom
registrar forwards INVITE to 197.87.54.21, which is running keith’s SIP
client. (6-8) SIP response sent back (9) media sent directly
between clients.
Note: also a SIP ack message, which is not shown.
Comparison with H.323
 H.323 is another signaling
 H.323 comes from the ITU
protocol for real-time,
interactive audio and video
conferencing
 H.323 is a complete,
vertically integrated suite of
protocols for multimedia
conferencing: signaling,
registration, admission
control, transport, codecs
 SIP is a single component.
Works with RTP, but does
not mandate it. Can be
combined with other
protocols, services
(telephony).
 SIP comes from IETF:
Borrows much of its
concepts from HTTP
 SIP has Web flavor,
whereas H.323 has
telephony flavor.
 SIP uses the KISS principle:
Keep it simple stupid.
Socket programming
 Development of network applications
 Implementation of protocol standard defined in an RFC
• Client and server conform to the rules of RFC
• Use the port number associated with the protocol
• Allows interoperability
 Proprietary network application
• The application-layer protocol used by the client and the
server do not necessarily conform to any existing RFC
• Developer creates both client and server programs
• Not interoperable with other applications
• Not to use well-known port numbers defined in RFCs
• TCP or UDP at the transport layer?
Socket programming
Goal: learn how to build client/server application that communicate
using sockets
Socket API
 introduced in BSD4.1 UNIX, 1981
 explicitly created, used, released
by apps
 client/server paradigm
 two types of transport service via
socket API:
 unreliable datagram
 reliable, byte stream-oriented
socket
a host-local,
application-created,
OS-controlled interface
(a “door”) into which
application process can
both send and
receive messages to/from
another application
process
Socket-programming using TCP
Socket: a door between application process and end-end-transport
protocol (UCP or TCP)
TCP service: reliable transfer of bytes from one process to another
controlled by
application
developer
controlled by
operating
system
process
process
socket
TCP with
buffers,
variables
host or
server
internet
socket
TCP with
buffers,
variables
host or
server
controlled by
application
developer
controlled by
operating
system
Socket programming with TCP
Client must contact server
 server process must first be
running
 server must have created
socket (door) that welcomes
client’s contact
Client contacts server by:
 creating client-local TCP socket
 specifying IP address, port
number of server process
 The client choses a source port
number
 When client creates socket:
client TCP establishes
connection to server TCP
 When contacted by client, server
TCP creates new socket for
server process to communicate
with client


allows server to talk with multiple
clients
source port numbers used to
distinguish clients
 TCP socket is identified by a four-
tuple: (source IP address, source
port number, destination IP
address, destination port number)
application viewpoint
TCP provides reliable, in-order
transfer of bytes (“pipe”)
between client and server
Client/server socket interaction: TCP
Server (running on hostid)
Client
create socket,
port=x, for
incoming request:
welcomeSocket =
ServerSocket()
TCP
wait for incoming
connection request connection
connectionSocket =
welcomeSocket.accept()
read request from
connectionSocket
write reply to
connectionSocket
close
connectionSocket
setup
create socket,
connect to hostid, port=x
clientSocket =
Socket()
send request using
clientSocket
read reply from
clientSocket
close
clientSocket
Stream jargon
output
stream
monitor
inFromUser
Client
Process
process
input
stream
outToServer
characters that flow into or out
of a process.
 An input stream is attached to
some input source for the
process, e.g., keyboard or
socket.
 An output stream is attached to
an output source, e.g., monitor
or socket.
keyboard
inFromServer
 A stream is a sequence of
input
stream
client
TCP
clientSocket
socket
to network
TCP
socket
from network
Socket programming with TCP
Example client-server app:
1) client reads line from standard input
(inFromUser stream) , sends to
server via socket (outToServer
stream)
2) server reads line from socket
3) server converts line to uppercase,
sends back to client
4) client reads, prints modified line
from socket (inFromServer
stream)
Example: Java client (TCP)
import java.io.*;
import java.net.*;
class TCPClient {
public static void main(String argv[]) throws Exception
{
String sentence;
String modifiedSentence;
Create
input stream
Create
client socket,
connect to server
Create
output stream
attached to socket
BufferedReader inFromUser =
new BufferedReader(new InputStreamReader(System.in));
Socket clientSocket = new Socket("hostname", 6789);
System.out.println(“client port: " + clientSocket.getLocalPort());
DataOutputStream outToServer =
new DataOutputStream(clientSocket.getOutputStream());
Example: Java client (TCP), cont.
Create
input stream
attached to socket
BufferedReader inFromServer =
new BufferedReader(new
InputStreamReader(clientSocket.getInputStream()));
sentence = inFromUser.readLine();
Send line
to server
outToServer.writeBytes(sentence + '\n');
Read line
from server
modifiedSentence = inFromServer.readLine();
System.out.println("FROM SERVER: " + modifiedSentence);
clientSocket.close();
}
}
Example: Java server (TCP)
import java.io.*;
import java.net.*;
class TCPServer {
Create
welcoming socket
at port 6789
Wait, on welcoming
socket for contact
by client
Create input
stream, attached
to socket
public static void main(String argv[]) throws Exception
{
String clientSentence;
String capitalizedSentence;
ServerSocket welcomeSocket = new ServerSocket(6789);
while(true) {
Socket connectionSocket = welcomeSocket.accept();
BufferedReader inFromClient =
new BufferedReader(new
InputStreamReader(connectionSocket.getInputStream()));
Example: Java server (TCP), cont
Create output
stream, attached
to socket
DataOutputStream outToClient =
new DataOutputStream(connectionSocket.getOutputStream());
Read in line
from socket
clientSentence = inFromClient.readLine();
capitalizedSentence = clientSentence.toUpperCase() + '\n';
Write out line
to socket
outToClient.writeBytes(capitalizedSentence);
}
}
}
End of while loop,
loop back and wait for
another client connection
Socket programming with UDP
UDP: no “connection” between
client and server
 no handshaking
 sender explicitly attaches IP
address and port of destination
to each packet
 server must extract IP address,
port of sender from received
packet
UDP: transmitted data may be
received out of order, or lost
application viewpoint
UDP provides unreliable transfer
of groups of bytes (“datagrams”)
between client and server
Client/server socket interaction: UDP
Server (running on hostid)
create socket,
port=x, for
incoming request:
serverSocket =
DatagramSocket()
read request from
serverSocket
write reply to
serverSocket
specifying client
host address,
port number
Client
create socket,
clientSocket =
DatagramSocket()
Create, address (hostid, port=x,
send datagram request
using clientSocket
read reply from
clientSocket
close
clientSocket
Example: Java client (UDP)
input
stream
Client
process
monitor
inFromUser
keyboard
Process
Input: receives
packet (recall
thatTCP received
“byte stream”)
UDP
packet
receivePacket
packet (recall
that TCP sent
“byte stream”)
sendPacket
Output: sends
UDP
packet
client
UDP
clientSocket
socket
to network
UDP
socket
from network
Example: Java client (UDP)
import java.io.*;
import java.net.*;
Create
input stream
Create
client socket
Translate
hostname to IP
address using DNS
class UDPClient {
public static void main(String args[]) throws Exception
{
BufferedReader inFromUser =
new BufferedReader(new InputStreamReader(System.in));
DatagramSocket clientSocket = new DatagramSocket();
InetAddress IPAddress = InetAddress.getByName("hostname");
byte[] sendData = new byte[1024];
byte[] receiveData = new byte[1024];
String sentence = inFromUser.readLine();
sendData = sentence.getBytes();
Example: Java client (UDP), cont.
Create datagram
with data-to-send,
length, IP addr, port
DatagramPacket sendPacket =
new DatagramPacket(sendData, sendData.length, IPAddress, 9876);
Send datagram
to server
clientSocket.send(sendPacket);
Read datagram
from server
clientSocket.receive(receivePacket);
DatagramPacket receivePacket =
new DatagramPacket(receiveData, receiveData.length);
String modifiedSentence =
new String(receivePacket.getData());
System.out.println("FROM SERVER:" + modifiedSentence);
clientSocket.close();
}
}
Example: Java server (UDP)
import java.io.*;
import java.net.*;
Create
datagram socket
at port 9876
class UDPServer {
public static void main(String args[]) throws Exception
{
DatagramSocket serverSocket = new DatagramSocket(9876);
byte[] receiveData = new byte[1024];
byte[] sendData = new byte[1024];
while(true)
{
Create space for
received datagram
Receive
datagram
DatagramPacket receivePacket =
new DatagramPacket(receiveData, receiveData.length);
serverSocket.receive(receivePacket);
Example: Java server (UDP), cont
String sentence = new String(receivePacket.getData());
Get IP addr
port #, of
sender
InetAddress IPAddress = receivePacket.getAddress();
int port = receivePacket.getPort();
String capitalizedSentence = sentence.toUpperCase();
sendData = capitalizedSentence.getBytes();
Create datagram
to send to client
DatagramPacket sendPacket =
new DatagramPacket(sendData, sendData.length, IPAddress,
port);
Write out
datagram
to socket
serverSocket.send(sendPacket);
}
}
}
End of while loop,
loop back and wait for
another datagram