GStreamer
T-111.5350 Multimedia
programming
Mika Ristimäki, Jenny Lahti
History




The project was founded in 1999 by Erik Walthinsen.
Later it was joined by the maintainer of GStreamer,
Wim Taymans.
The first corporate sponsor was RidgeRun, that hired
Erik Walthinsen to develop methods for embedding
GStreamer in smaller devices
The GStreamer logo was designed by Brock A. Frazier,
who worked for RidgeRun.
Introduction






GStreamer was created to serve as a host for
multimedia applications.
It is written in the C programming language
It is a cross-platform product, so it works e.g. on Linux,
Solaris, Mac OS X and Microsoft Windows.
It provides a framework of plugins, for data flow and
media type handling.
It is created to handle both audio and video.
With GStreamer’s development framework you can
write any type of streaming multimedia application.
License – LGPL




GStreamer is released under LGPL (GNU Lesser
General Public License).
LGPL means that the GStremer software itself has
copyleft rules which means that you have to keep the
software free.
But you are allowed to create your own software with
your own copyright rules and link it with GStremer.
For companies like Nokia this license gives a
profitable basis for their own software. They can use
GStremer as a multimedia framework, but own the
copyright to their own software, because it is merely
linked to GStreamer.
Applications



GStreamer’s most obvious use is creating a media
player on top of it, because it supports many needed
formats for this.
GStreamer can also be used to create more complex
programs like video or audio editing applications.
We intend to use it to create an Internet radio.
N770, N800 and N810


Nokia’s open source product N770 Internet Tablet
and its successors N800 and N810, are palm-sized
tablet PC:s running Linux
The tablets use GStreamer as a multimedia
framework



This enables 3rd party developers to use on-board DSP
without extensive knowledge about its architecture
They support audio (MP3, MPEG4-AAC, WAV, AMR
and MP2) and video (MPEG-1, MPEG-4, Real Video,
H.263, AVI and 3GP).
They have a Real player and a Flash player
N770, N800 and N810
Lifetrak




A project at University of California, Department for
Electrical Engineering by Sasank Reddy and Jeff
Mascia.
Lifetrak uses Nokia 770 as a hardware platform for
creating a portable music player.
Lifetrak enables a context-aware playlist that
automatically selects music based upon the users
location, the pace of his movement and the
environment
Lifetrak has a simple learning mechanism that can
adjust the selection of the music based on the user’s
ratings.
Lifetrak
Lifetrak



Lifetrak’s playing functionality is built on top of
GStreamer
Lifetrak supports song files in the MP3 format, and in
the future also Ogg-Vorbis, WAV, AAC, and MP4 files.
GStreamer can handle each of these compression
standards
GStreamer is also used for setting the volume, getting
the position, and setting the position of the current
song.
Mad Hatter’s Cocktail Party




A project at Standford University, Palo Alto Research
Center and Robotics Institute in Carnegie Mellon
University
Project wants to enhance conference call type mobile
communication
The idea is to make communication possible for
several small groups in a single audio space and for
these groups to be able to regroup when desired.
The communication model is similar to a cocktail party.
Mad Hatter’s Cocktail Party



The idea is to analyse the audio, find these groups and
make each listener hear his own group with a higher
sound level and dimm out the voices of the rest.
This simulates a situation where you stand closer to the
ones you listen to.
This is done by analysing when people speak,
assuming that people in the same group mainly speek
in turns.
Mad Hatter’s Cocktail Party


The server, that analyses the voices, is a PC running
Linux using GStreamer.
The server receives audio streame from each client,
analyzes them, applyes audio effect and performs
mixing, and finally transmits audio back to the
listeners.
Usage

GStreamer is so called “pipelined” multimedia framework


The basic building block of a pipeline is called an element


Pipe-line consists of sequence of operations
Pipe-line consists of chain of these elements and data flows
through it.
For example, a basic mp3 audio decoder

Element 1


Element 2


Reads mp3 file from hard disk.
Decodes the mp3 data to raw audio
Element 3

Sends the raw audio to soundcard
Usage

Every element may also have a different number of
pads
Sink (input) pad and source (output) pad.
 Pad is an element’s link to outside world
 Pads can also be dynamic so that a pad can be randomly
created and/or destroyed.


Interconnected elements form a bin
Is also element
 Same operations that can be done to an element can be
done to a bin also
 Pipeline is a specialized type of bin so that the top level bin
has to be pipeline

Usage



A bus takes care of internal messaging in GStreamer.
A callback function can be defined that takes care of
EOS or other errors messages and acts upon them.
Every pipe-line has a bus by default


The developer should create a callback function for the bus.
When the main loop is running the bus is checked for messages and when
a new message is noticed the callback function will be called
Example
#include <gst/gst.h>
bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline));
gst_bus_add_watch (bus, bus_call, loop);
int
gst_object_unref (bus);
main (int argc,
char *argv[])
gst_bin_add_many (GST_BIN (pipeline),
{
source, parser, decoder, conv, sink, NULL);
pipeline = gst_pipeline_new ("audio-player");
source = gst_element_factory_make ("filesrc", "file-source");
gst_element_link (source, parser);
parser = gst_element_factory_make ("oggdemux", "oggparser");
gst_element_link_many (decoder, conv, sink, NULL);
decoder = gst_element_factory_make ("vorbisdec", "vorbisdecoder");
conv = gst_element_factory_make ("audioconvert", "converter");
sink = gst_element_factory_make ("alsasink", "alsa-output");
g_object_set (G_OBJECT (source), "location", argv[1], NULL);
g_signal_connect (parser, "pad-added", G_CALLBACK
(new_pad), NULL);
}
Example
#include <gst/gst.h>
bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline));
gst_bus_add_watch (bus, bus_call, loop);
int
gst_object_unref (bus);
main (int argc,
char *argv[])
Creating elements
gst_bin_add_many (GST_BIN (pipeline),
{
source, parser, decoder, conv, sink, NULL);
pipeline = gst_pipeline_new ("audio-player");
source = gst_element_factory_make ("filesrc", "file-source");
gst_element_link (source, parser);
parser = gst_element_factory_make ("oggdemux", "oggparser");
gst_element_link_many (decoder, conv, sink, NULL);
decoder = gst_element_factory_make ("vorbisdec", "vorbisdecoder");
conv = gst_element_factory_make ("audioconvert", "converter");
sink = gst_element_factory_make ("alsasink", "alsa-output");
g_object_set (G_OBJECT (source), "location", argv[1], NULL);
g_signal_connect (parser, "pad-added", G_CALLBACK
(new_pad), NULL);
}
Example
#include <gst/gst.h>
bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline));
gst_bus_add_watch (bus, bus_call, loop);
int
gst_object_unref (bus);
main (int argc,
char *argv[])
Creating elements
gst_bin_add_many (GST_BIN (pipeline),
{
source, parser, decoder, conv, sink, NULL);
pipeline = gst_pipeline_new ("audio-player");
source = gst_element_factory_make ("filesrc", "file-source");
gst_element_link (source, parser);
parser = gst_element_factory_make ("oggdemux", "oggparser");
gst_element_link_many (decoder, conv, sink, NULL);
decoder = gst_element_factory_make ("vorbisdec", "vorbisdecoder");
g_signal_connect (parser, "pad-added", G_CALLBACK
(new_pad), NULL);
}
conv = gst_element_factory_make ("audioconvert", "converter");
sink = gst_element_factory_make ("alsasink", "alsa-output");
g_object_set (G_OBJECT (source), "location", argv[1], NULL);
Setting filesrc
element properties
Example
#include <gst/gst.h>
Adding the callback
function for bus
bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline));
gst_bus_add_watch (bus, bus_call, loop);
int
gst_object_unref (bus);
main (int argc,
char *argv[])
Creating elements
gst_bin_add_many (GST_BIN (pipeline),
{
source, parser, decoder, conv, sink, NULL);
pipeline = gst_pipeline_new ("audio-player");
source = gst_element_factory_make ("filesrc", "file-source");
gst_element_link (source, parser);
parser = gst_element_factory_make ("oggdemux", "oggparser");
gst_element_link_many (decoder, conv, sink, NULL);
decoder = gst_element_factory_make ("vorbisdec", "vorbisdecoder");
g_signal_connect (parser, "pad-added", G_CALLBACK
(new_pad), NULL);
}
conv = gst_element_factory_make ("audioconvert", "converter");
sink = gst_element_factory_make ("alsasink", "alsa-output");
g_object_set (G_OBJECT (source), "location", argv[1], NULL);
Setting filesrc
element properties
Example
#include <gst/gst.h>
Adding the callback
function for bus
bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline));
gst_bus_add_watch (bus, bus_call, loop);
int
gst_object_unref (bus);
main (int argc,
char *argv[])
Creating elements
Putting elements to a bin
gst_bin_add_many (GST_BIN (pipeline),
{
source, parser, decoder, conv, sink, NULL);
pipeline = gst_pipeline_new ("audio-player");
source = gst_element_factory_make ("filesrc", "file-source");
gst_element_link (source, parser);
parser = gst_element_factory_make ("oggdemux", "oggparser");
gst_element_link_many (decoder, conv, sink, NULL);
decoder = gst_element_factory_make ("vorbisdec", "vorbisdecoder");
g_signal_connect (parser, "pad-added", G_CALLBACK
(new_pad), NULL);
}
conv = gst_element_factory_make ("audioconvert", "converter");
sink = gst_element_factory_make ("alsasink", "alsa-output");
g_object_set (G_OBJECT (source), "location", argv[1], NULL);
Setting filesrc
element properties
Example
#include <gst/gst.h>
Adding the callback
function for bus
bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline));
gst_bus_add_watch (bus, bus_call, loop);
int
gst_object_unref (bus);
main (int argc,
char *argv[])
Creating elements
Putting elements to a bin
gst_bin_add_many (GST_BIN (pipeline),
{
source, parser, decoder, conv, sink, NULL);
pipeline = gst_pipeline_new ("audio-player");
source = gst_element_factory_make ("filesrc", "file-source");
gst_element_link (source, parser);
parser = gst_element_factory_make ("oggdemux", "oggparser");
gst_element_link_many (decoder, conv, sink, NULL);
decoder = gst_element_factory_make ("vorbisdec", "vorbisdecoder");
g_signal_connect (parser, "pad-added", G_CALLBACK
(new_pad), NULL);
}
Linking elements
together
conv = gst_element_factory_make ("audioconvert", "converter");
sink = gst_element_factory_make ("alsasink", "alsa-output");
g_object_set (G_OBJECT (source), "location", argv[1], NULL);
Setting filesrc
element properties
Example
static gboolean bus_call (GstBus
gpointer
Callback funtion for
error handling
*bus, GstMessage *msg,
data)
{
GMainLoop *loop = (GMainLoop *) data;
switch (GST_MESSAGE_TYPE (msg)) {
case GST_MESSAGE_EOS:
g_print ("End-of-stream\n");
g_main_loop_quit (loop);
break;
case GST_MESSAGE_ERROR: {
gchar *debug;
GError *err;
static void
new_pad (GstElement *element,
GstPad *pad,
gpointer data)
{
GstPad *sinkpad;
/* We can now link this pad with the audio decoder */
g_print ("Dynamic pad created, linking parser/decoder\n");
sinkpad = gst_element_get_pad (decoder, "sink");
gst_pad_link (pad, sinkpad);
gst_object_unref (sinkpad);
gst_message_parse_error (msg, &err, &debug);
g_free (debug);
g_print ("Error: %s\n", err->message);
g_error_free (err);
g_main_loop_quit (loop);
break;
}
default:
break;
}
return TRUE;
}
Function for
dynamic pad
creation
}
References







[1] GStreamer Application Development Manual (0.10.12.1), Wim Taymans, Steve Baker, Andy Wingo,
Ronald S. Bultje, Stefan Kost
[2] DSP video processing via open-source APIs, EETimes OnLine, Rishi Bhatteracharya, 30.10.2006
http://www.eetimes.com/news/design/showArticle.jhtml?articleID=193401461
[3] The Mad Hatter’s Coktail Party: A Social Mobile Audio Space Supporting Multiple Simultaneous
Conversations, Paul M. Aoki, Matthew Romaine, Margaret H. Szymanski, James D. Thornton, Daniel Wilson
and Allison Woodruff, Palo Alto Research Center, CCRMA, Dept. of Music Stanford University, Robotics
Institute Carnegie Mellon University, Florida USA 5.-10.43.2003, Paper: Design for the Socially Mobile
[4] Lifetrak: Music In Tune With Your Life, Sasank Reddy, Jeff Mascia, NESL, Department of Electrical
Engineering University of California, Los Angeles
[5] Multimedia Abstraction Architecture of Cross-platform Applications Based on IMS, Jie Chen, Master
Thesis, Helsinki University of Technology / Department of Computer Science and Engineering, 2006
[6] Linux for suits: a first look at the Nokia 770, Doc Searls, 02/2006, Linux Journal, Volume 2006 Issue
142, Publisher: Specialized Systems Consultants, Inc.
[7] Wikipedia, The Free Encyclopedia, (GStreamer, LGPL), http://en.wikipedia.org/wiki/Gstreamer,
http://en.wikipedia.org/wiki/GNU_Lesser_General_Public_License