i/o multiplexing
On adding a ‘poll()’ method to our
character-mode device-driver for
an 82573L network controller
An application idea
• We want to create an application program
that would allow two users to conduct an
on-line chat-session while working from
two separate nodes on our ‘anchor’ cluster
• Whatever either user decides to type will
be displayed on the both users’ screens
• To keep their two input-streams visually
separated we want to use two ‘windows’
‘netchat.cpp’
Whatever a user types will be displayed in the lower window on that
user’s screen -- and in the upper window on the other user’s screen
Hello, Susan
Hi, Paul
Hi, Paul
_
Hello, Susan
_
anchor01
anchor02
Multiplexed input-streams
• In order to implement the foregoing idea,
it’s necessary for the ‘netchat’ program to
accept input from TWO hardware devices:
– the keyboard (i.e., ‘standard input’ device)
– the ethernet controller (i.e., ‘/dev/nic’)
• Such a situation is a fairly common one in
UNIX/Linux application programming, and
is referred to as ‘i/o multiplexing’
Device-driver’s role
• Special software support is needed within
each of the device-drivers in order for any
application-program to do ‘i/o multiplexing’
in a way that avoids wasting of CPU time
• (For an excellent discussion of the various
approaches that can be taken to deal with
this basic programming issue, see Richard
Stevens: Advanced Programming in the
UNIX Environment, Chapter 12)
The basic problem
• Normally when an application ‘reads’ from
a device-file, that process will ‘sleep’ until
some data is available from that device
• So if data becomes available on another
device, it will not get processed because
the application is ‘blocked’ from being
given any CPU time by the OS scheduler
• This would spoil our ‘netchat’ application
‘read()’ causes ‘blocking’
read
terminal
write
‘netchat’
application
write
read
Ethernet
controller
Whichever device the application attempts to read from, it
will get ‘blocked’ until that device has some data to deliver
Do multiprocessing?
• One idea for getting around this ‘blocking’
problem would be to just use the ‘fork()’
system-call to create a separate process
for reading from the different device-files
• Each process can sleep, and whichever
process receives any new data will be
awakened and scheduled for execution
• No changes needed to device-driver code
Different processes do ‘read()’
read
‘netchat’
parent- process
write
Ethernet
controller
terminal
write
‘netchat’
child-process
read
Using multiple processes can overcome the ‘blocking-read’
problem, but complicates the code for program termination
Non-blocking ‘read’
• It is possible for the application to request
‘non-blocking’ read-operations – i.e., any
‘read()’ calls will immediately return with 0
as return-value in case no data is available
• The standard-input device-driver already
has support for this non-blocking option,
and it can be easily added to the ‘read()’
function in network controller’s driver
Code-modification
ssize_t my_read( struct file *file, char *buf, size_t len, loff_t *pos )
{
static int
rxhead = 0;
// in case no new data has been received, then either
// return immediately if non-blocking mode is in effect
// or else sleep until some new data arrives (or until
// the user hits <CONTROL>-C to cancel execution)
if ( rxhead == ioread32( io + E1000_RDH ) {
if ( file->f_flags & O_NONBLOCK ) return 0;
if ( wait_event_interruptible( wq_rx,
rxring != ioread32( io + E1000_RDH ) )
return –EINTR;
}
…
Uses ‘busy-waiting’ loop
read
terminal
write
‘netchat’
application
write
read
Ethernet
controller
Using the ‘nonblocking-read’ option overcomes the problem
of a sleeping task, but it wastefully consumes the CPU time
The ‘elegant’ solution
• The ‘select()’ system-call provides a very
general scheme for doing i/o-multiplexing
in a manner that avoids wasting CPU time
or making the program-code complicated
• But it does require adding an extra driver
‘method’ – the so-called ‘poll()’ function
The ‘select()’ arguments
• Using ‘select()’ requires an application to
setup an ‘fd_set’ object, which defines the
set of file-descriptors whose activity needs
to be monitored by the Linux kernel (in our
‘netchat’ application this would be just the
two device-files: the console keyboard and
the gigabit ethernet network controller)
• This ‘fd_set’ object becomes an argument
Using ‘select()’ in ‘netchat’
int
int
kbd = STDIN_FILENO;
aux = open( “/dev/nic”, O_RDWR );
fd_set
permset;
FD_ZERO( &permset );
FD_SET( kbd, &permset );
FD_SET( aux, &permset );
// keyboard ID
// device-file ID
// create an ‘fd_set’ object
// initialize it to ‘empty’
// add keyboard to set
// and add the nic to set
while (1) {
fd_set
readset = permset;
if ( select( 1+aux, &readset, NULL, NULL, NULL ) < 0 ) break;
if ( FD_ISSET( kbd, &readset ) ) { /* process keyboard input */ }
if ( FD_ISSET( aux, &readset ) ) { /* process network input */ }
}
How it works
• The ‘readset’ argument to the ‘select()’
system-call lets the kernel know which
device-drivers should have their ‘poll()’
method invoked
• Then each device-driver’s ‘poll()’ method
will perform a test to determine if any new
data is ready to be read from that device
• So the application calls ‘read()’ only when
a device is ready with data immediately
struct file_operations
• We need to include the function-pointer to
our implementation for the ‘poll()’ method:
struct file_operations
my_fops = {
owner:
read:
write:
ioctl:
poll:
};
THIS_MODULE,
my_read,
my_write,
my_ioctl,
my_poll,
Our driver’s ‘poll()’ method
• Linux provides helper-functions to do most
of the supporting work for use of ‘select()’
#include <linux/poll.h>
// for the ‘poll_wait()’ helper-function
unsigned int my_poll( struct file *file, struct poll_table_struct *wait )
{
unsigned int
mask = 0;
poll_wait( file, &wq_recv, wait );
if ( (ioread32( io + E1000_RDH ) != rxhead )
mask |= POLLIN | POLLRDNORM;
return
}
mask;
The ‘ncurses’ library
• Our ‘netchat’ application uses the Linux
implementation for the UNIX ‘curses’ API
• This function-library includes routines that
not only can manage the console’s cursor,
but also implements ‘raw’ keyboard input
(for instantaneous keystroke processing)
and allows drawing of window-borders
Compiling ‘netchat’
• The ‘g++’ compiler can compile-and-link in
a single command:
$ g++ netchat.cpp -l ncurses -o netchat
The C++ source-file
Name of the library
Lowercase letter ‘L’ (for library to link with)
Name for the compiler’s output-file
Try it out
• We can try running the ‘netchat’ program
on any pair of our anchor-cluster stations
• Of course, we will first need to install our
‘nicpoll.ko’ device-driver module on each
of those ‘anchor’ machines
• Then a user at either machine can type in
any messages, and every character typed
will be visible immediately on both screens
But…
• There is a slight problem with using the
present version of our ‘nicpoll.c’ module
• All of the network packets are ‘broadcast’
to every station on the ‘anchor’ cluster!
• So any third-party who is simultaneously
trying to use our network will be receiving
all the packets that are being broadcast
In-class exercise
• Based on what you have learned from our
past lessons, and from your reading of the
Intel “Open Source Software Developer’s
Manual” for its PCIe GbE Controllers, can
you propose any ideas for using the NIC’s
hardware capabilities so as to allow a pair
of anchor-stations to use ‘netchat’ without
any other anchor-stations being involved?