use and abuse of pipes with audio data
Debian Administration
Posted by Anonymous on Tue 24 May 2005 at 12:17
What ho! Get on board! Let me take you on a voyage of discovery on the vast ocean of
GNU/Linux. We will venture forth and study some nifty tools, including the curious thing
known as a fifo.
fifo file fun!
What is a fifo ? Well, it's a special sort of file. The name stands for "file in, file out",
indicating that the file goes in and out. It is sometimes also called a named pipe.
Why is that?
Well, suppose I type
mpg123 -s *.mp3 | rawplay
This is a nice simple one-liner. It gets the mp3 files converted into a stream of raw audio
and plays the raw audio with rawplay ("apt-get install rawrec" will install rawplay).
Looking at it a bit closer, I could say I am piping the output of mpg123 to a temporary file
which I do not see (symbolised by the | sign) and then running that temporary file past
rawplay. I could say the temporary file is hidden from sight. I could say all that, but I'd be
wrong actually, because it is not a temporary file - it is a buffer - and data continuously
streams via this buffer.
What's the difference between a file and a buffer?
Well, let me uglify the one-liner by bringing a temporary file into sight. Let me name the
temporary file explicitly. I could do:
mpg123 -s *.mp3 > tempfile
cat tempfile | rawplay
rm tempfile
and get the same result as our one-liner. But this is now a multistep process, and would
probably involve an awful lot of diskswapping as tempfile is created in its full glory, and
only then streamed into rawplay. Not at all nice and continuous compared to the one-liner
buffer version.
There is another way of making it a multistep process though. I can make an equivalent of
the one-liner buffer version, without the diskswapping overhead, but still using a temporary
handle, like this:
mkfifo myfifo
mpg123 -s *.mp3 > myfifo
cat myfifo | rawplay
rm myfifo
Oh yes. I sneaked the fifo thing in here. It is called a "named pipe" because it is a pipe type
of operation being done and the pipe has been explicitly named. In this case it has been
called myfifo. myfifo does the file in/file out bit, and it differs from a normal file (such as
tempfile earlier) because it has a file size of 0. Not surprisingly, perhaps, because it is a
buffer, sending data out as it gets it.
That's nifty, But what's it good for? Why should I even bother with a multistep fifo process
using this weirdo stuff when I have the one-liner?
Well, now I have an explicit handle on the step in the middle, a handle that takes up no
diskspace, and it lets me do things on-the-fly to the stream.
Fifo? A handle that lets me do things on the fly?
Yes. Some more examples are probably a good idea now:
I will use the netcat utility to illustrate things. netcat (or nc) is a must-have tool that allows
the machine to send and listen to stuff on ports. (We covered an introduction to netcat
previously - but if you don't have it "apt-get install netcat" will install it).
Netcat is like a telnet on steroids. Eg: suppose I have a smtp server, which I will
unimaginatively call smtpserver, allowing access on the standard port 25. Then:
nc smtpserver 25
will be pretty much the same as
telnet smtpserver 25
See the man page for netcat for more details.
Now, suppose I have this tinpot laptop with horrible shrill speakers, and a multimedia
desktop machine with StupendaSound speakers, (you know the kind I mean, the ones with
an eyepopping, earbleeding bass). They are connected via LAN. I don't want to hear the
stream on my tinny sounding laptop - I want to hear it in its full ghettoblasting glory on the
multimedia desktop.
So on tinpot I type:
tinpot:~$ mkfifo myfifo
tinpot:~$ mpg123 -s *.mp3 > myfifo
Ie: I make the fifo, and stream the raw audio to it.
On ghettoblaster I type:
ghettoblaster:~$ nc -l -p 2345 | rawplay
Here, netcat is listening on port 2345, and piping anything it hears to rawplay. There is no
sound being heard yet.
Then back on tinpot I type:
tinpot:~$ cat myfifo
| nc ghettoblaster 2345
Ie, I am taking the myfifo buffer on tinpot and forwarding it to netcat, which sends it out to
ghettoblaster's port 2345.
The moment I type in the last line, the sound starts up (think of it as a valve in a plumbing
pipe system - until the myfifo valve is released, nothing flows). Of course, this would be a
terrible way to do distribute the stream over the internet, since it is uncompressed. But on a
LAN with loads of bandwidth this is just dandy.
Why use fifo tricks at all? After all, I could use just these two lines, and do away with
myfifo entirely:
$ghettoblaster:~$ nc -l -p 2345 | rawplay
$tinpot:~$ mpg123 -s *.mp3 | nc ghettoblaster 2345
True, and that works splendidly for two machines if I want that only one of them play the
sound. But suppose I have a third machine in the lounge and I want to play the stream there
as well, at the same time? Try as I may, there is no way of getting sound out on more than
one machine on-the-fly without installing a full distributed sound application - unless I use
fifo, the magic handle in the middle. You see, once you have a handle, you can use all the
usual trickery with pipes, redirection, and, most importantly in our case, tee.
tee for two (or more)
Let's get really adventurous, and pipe the raw sound data not just to two, or even three
machines. No indeed. Let's send it to 6 machines all over the house. It makes the structure
of things a bit clearer actually. The set up then goes like this:
On the machine that is the source of the sound, (tinpot) I do:
tinpot:~$ mkfifo myfifo1 myfifo2 myfifo3 .... myfifo6
tinpot:~$ mpg123 -s *.mp3 > myfifo1
Then I do the following on the machines indicated.
machine1:~$ nc -l -p 2345 | rawplay
tinpot:~$ cat myfifo1 | tee myfifo2 | nc machine1 2345
machine2:~$ nc -l -p 2345 | rawplay
tinpot:~$ cat myfifo2 | tee myfifo3 | nc machine2 2345
...
machine6:~$ nc -l -p 2345 | rawplay
tinpot:~$ cat myfifo6 | nc machine6 2345
The moment I release the valve (ie type "cat myfifo8 | nc machine6 2345" on the last
machine), all 6 machines start playing. (Remember, tee pipes stuff to STDOUT as well as
to a file (in this case a fifo). It functions like an audio splitter in this case).
So there I am, with music blaring more or less in sync throughout the house, and a big
geeky grin on my face. For I am now a Master of fifo, tee, and netcat! With my newfound
nerd bravado, I venture forth to the next adventure in GNU/Linux geekdom:
natty new nettee net utility - tee for fifo
So what if the gobbledygook title of this subsection doesn't even sound like English? Think
of it as me being incoherent with excitement over David Mathog's rather spiffy nettee.
nettee is a clever new utility that sort of combines netcat with tee, and so does away with
fifos. Grab it and compile it off the site (there is no debian package for it to date (24 May
2005) that I am aware of, but that may have changed by the time you read this - it seems to
only have been around for about a month so far).
Install nettee on all the machines, and run it on each like this:
machine1:~$ ./nettee | rawplay
machine2:~$ ./nettee | rawplay
...
machine6:~$ ./nettee | rawplay
so now each of these is listening on the nettee port (9997 by default).
Then on the source machine you type in:
tinpot:~$ mpg123 -s *.mp3 | ./nettee -in - -next machine1,machine2,machine3, ... ,machine6
That's it!
The walls are a-quaking now as I blast out "WE WILL ROCK YOU!!!" at full annoy-myneighbours level. Not just on one machine, but simultaneously now on all the machines in
the house. The plaster is falling down, and the irritating thumps of my neighbour's broom
are drowned out by the boneshaking ear-crushing sound. I thump my desk frenziedly along
with the music in the primal excitement of the moment, feeling deliriously self-satisfied
and smug, I bellow out aloud: "Yep, nettee really, literally, ROCKS!!!"
Epilogue
And as this article at last comes back from its voyage of discovery on the endless ocean of
GNU/Linux, coming back to rest on terra firma once again, I say unto you have have so
bravely gone with me: go forth, young geek, and discover new ways to abuse these tools,
and spread the good word! Yea, for verily you are one of the Chosen ones!
PJ