Multimodal Interaction
Dr. Mike Spann
m.spann@bham.ac.uk
http://www.eee.bham.ac.uk/spannm
Contents
 Introduction
 ImageJ introduction
 Image representation in ImageJ
 Displaying images in ImageJ
 ImageJ utility class
 Regions of interest
 Writing plug-ins
 OpenCV introduction
 Image handling and display in OpenCV
 Other libraries
 Key tips
 Key resources
Introduction
 Two free platforms for developing image processing
software are ImageJ and OpenCV
 ImageJ – Java
 OpenCV – C++, C#
 They both allow rapid development of image
processing algorithms
 ImageJ through the use of plugins
 Both have extensive libraries of well known image
processing functions for the user to build upon
ImageJ introduction
 All of this material is taken from the ImageJ
tutorial accessed from my web-site
 http://mtd.fh-hagenberg.at/depot/imaging/imagej/
 Also check out the ImageJ home page
 http://rsb.info.nih.gov/ij/
 ImageJ is a free image processing system
allowing easy development of Java based image
processing algorithms in the form of plug-ins
ImageJ introduction
 It comes with a user-friendly GUI and can run
either as an application or an applet
 http://rsb.info.nih.gov/ij/applet/
ImageJ introduction
 It can handle 8,16,24 and 32 bit images
 It can handle most standard image formats





TIFF
GIF
JPEG
BMP
DICOM
 It can handle stacks of images
 Also there are plug-ins allowing it to handle movie
files
 It can also handle regions of interest (ROI’s)
ImageJ introduction
 The key point about ImageJ is that it is simple to add
your own algorithms (written as plug-ins) callable
from the front-end GUI
 File I/O taken care of by ImageJ
 Pixel access easy from image handles defined within
ImageJ
Image representation in ImageJ
 ImageJ has 5 built-in image classes
 8 bit grayscale (byte)
 8 bit colour (byte)
 16 bit grayscale (short)
 RGB colour (int)
 32 bit image (float)
 It also supports image stacks consisting of images
(slices) of the same size
Image representation in ImageJ
 ImageJ uses 2 classes to represent and manipulate
images
 ImagePlus

An image is represented by an ImagePlus object
 ImageProcessor

This holds the pixel data and contains methods to access the
pixel data
Image representation in ImageJ
 Pixel access methods in ImageProcessor include
 Object getPixels() – returns a reference to the pixel array
(need to cast to appropriate type)
 int getHeight() – returns height of pixel array
 int getWidth() – returns width of pixel array
Image representation in ImageJ
 A subclass of ImageProcessor is passed to the run()
method of the plug-in filter (see later)
 ByteProcessor - 8 bit grayscale images
 ShortProcessor – 16 bit grayscale images
 ColorProcessor – RGB images
 FloatProcessor – 32 bit floating point images
Image representation in ImageJ
 Pixel representation in ImageJ uses the byte datatype
for grayscale and colour images and short for 16-bit
grayscale
 byte/short are signed data types


byte ranges from –128 to 127
short ranges from –32768 to 32767
 Obviously grayscale values are usually positive values
Image representation in ImageJ
 To cast a byte to an integer, we need to eliminate the
sign bit
byte[] pixels=(byte[]) ip.getPixels();
int grey=0xxff & pixels[j];
 Can cast back the other way easily enough
pixels[j]=(byte) grey;
Image representation in ImageJ
 The ColorProcessor return the pixels as int[] and the
RGB values are packed into the one int variable
31
0 bit
int[] pixels=(int[]) ip.getPixels();
int red=(0xxff0000 & pixels[j]) >> 16;
int green=(0xx00ff00 & pixels[j]) >> 8;
int blue=(0xx0000ff & pixels[j]);
Image representation in ImageJ
 Can reconstitute an RGB array by shifting the other
way :
pixels[j]=((red & 0xff)<<16)+((green & 0xff)<<8)+(blue & 0xff);
Displaying images in ImageJ
 A class ImageWindow is used to display ImagePlus
objects
 We don’t normally need to access methods of
ImageWindow
 These are automatically called from ImagePlus methods
show(), draw() and updateAndDraw()
ImageJ utility class
 ImageJ contains a class called IJ which contains a
number of useful static methods
 Error messages


static void error(String message) – displays an error
message in a dialog box
static boolean showMessageWithCancel(String title,
String message) – allows the user to cancel the plug in or
continue
ImageJ utility class
 Displaying text
 static void write(String s) - Outputs text in a window –
useful for displaying textual or numerical results of
algorithms
 Displaying text in a status bar
 static void showStatus(String s)
ImageJ utility class
 User input
 static double getNumber(String prompt, double default)
– Allows the user to input a number in a dialog box
 static String getString(String prompt, String default) –
Allows the user to input a string in a dialog box
 The GenericDialog class is a more sophisticated
way of inputting more than a single number or
string
Regions of interest (ROI’s)
 A plug in filter does not always have to work on the
whole image
 ImageJ supports ROI’s which can are usually
rectangular but can be other shapes as well.
 We set/get the ROI using the following method of
the ImageProcessor class
 void setROI(int x, int y, int w int h)
 Rectangle getROI()
Writing plug-ins
 To write a plug in requires developing a class which
implements either the PlugIn or PlugInFilter
interfaces
 The second is more usual as this is used when the filter
requires an input image
import ij.*;
import ij.plugin.filter.PlugInFilter;
import ij.process;
class myPlugin implements PlugInFilter
Writing plug-ins
 Methods setup() and run() must then be provided
 Method setup() sets up the plugin filter for use
int setup(String arg, ImagePlus imp)
 String arg allows input arguments to be passed to the
plugin
 Argument imp handled automatically by ImageJ

It’s the currently active image
Writing plug-ins
 Method setup() returns a flag word representing
the capabilities of the plug-in filter (for example
the types of images it can handle). For example :
 static int DOES_8G
 static int DOES_RGB
 static int DOES_ALL
 static int NO_CHANGES (plug in doesn’t change the
image data)
 static int ROI_REQUIRED
 etc
Writing plug-ins
 The run() method is called when the plugin is run
from the ImageJ menu
 It contains the code to process individual pixels
 If no input image is required
void run(String arg)
 If an input image is required
void run(ImageProcessor ip)
Writing plug-ins
 Once the Java program is written and compiled, the
.class file is stored in the plug-in directory which is a
sub-directory of the ImageJ home directory
 The class name should contain an underscore symbol
(eg MyAlgorithm_ )
 It then appears under the plug-in sub-menu of the
ImageJ gui
OpenCV introduction
 Based on Intel IP Library
 More than 300 functions
 Features
 Multi-platform
 Provides a simple window manager with sliders mouse call backs
etc
 Uses Intel Integrated Performance Primitives to enhance
performance
 Tutorials & Documentation
 Sample code
 Download from
 sourceforge.net/projects/opencvlibrary/
 Operates under
 Windows 95/98/NT/2000/XP
 POSIX Linux/BSD/OSX/Win2K/WinXP
Image handling and display in
OpenCV
 Still images from file
 Using cvLoadImage()
IplImage* cvLoadImage(const char* filename,int
iscolor=1);
Returns a pointer to an ipl image
Filename: the name of the image file
iscolor:
>0 image is loaded as 3 channel colour
0 image is loaded as grey-level
<0 No. of channels determined by file
 File formats






Windows bitmaps: bmp, DIB
JPEG files: jpeg, jpg, jpe
Portable Network Graphics: png
Portable image format: pbm, pgm, ppm
Sun raster: sr, ras
Tiff Files: tiff, tif
Image handling and display in
OpenCV
 Simple code to load and display an image
 Uses library HighGUI
 Need to register image window and display it
 int cvNamedWindow(const char* name, int flags);
name:
int:
id name and text on image banner
1, autosize to fit the image
 void cvShowImage(const char* name, const CvArr*
img);
name:
img:
id name and text on image banner
pointer to image to be displayed
 CvCapture* cvCaptureFromFile(const char* fname);
fname:
video file name
 CvCapture* cvCaptureFromCAM(int index);
index:
positive integer specifies video source
/* usage: prog <image_name> */
#include "cv.h“
/* OpenCV data types and prototypes */
#include "highgui.h“
/* Image read and display*/
int main( int argc, char** argv )
{IplImage* img;
if(argc==2 && (img=cvLoadImage(argv[1], 1)) !=0)
{cvNamedWindow("Image view", 1);
cvShowImage("Image view", img);
cvWaitKey(0); /*very important, contains event */
/*processing loop inside
*/
cvDestroyWindow("Image view");
cvReleaseImage(&img);
return 0;
}
return -1;
}
Other libraries:(CV)
 Image Processing
 Edges, filters, morphologic operators, pyramids,
histograms, boundary tracing
 Structural analysis
 Contour processing, computational geometry
 Motion Analysis and Object Tracking
 Object tracking, optical flow, motion templates
 Pattern Recognition
 Camera calibration and 3D Reconstruction
 Camera calibration, pose estimation, epipolar geometry
Other libraries:cxcore
 Operations on arrays
 Linear algebra, statistics, random number generator
 Dynamic structures
 Trees, graphs, sets, sequences
 Drawing
 Curves, text, point sets, contours
Other libraries:ML
 Machine Learning
 Bayes classifier
 K-nearest neighbour classifier
 Support Vector Machine
 Decision Trees
 Boosting
 Random Forest
 Expectation Maximisation
 Neural Networks
Other libraries:CvCam/Cvaux
 CvCam
 Video Input / Output





Select and set up a camera
Render a video stream
Control the Video
Process video frames
Multiple cameras under Linux and Windows
 Cvaux

Experimental functions
Key tips
 IplImage data Types








IPL_DEPTH_8U
IPL_DEPTH_8S
IPL_DEPTH_16U
IPL_DEPTH_16S
IPL_DEPTH_32S
IPL_DEPTH_32F
IPL_DEPTH_ 64F
Channels:
unsigned 8 bit
signed 8 bit
unsigned 16 bit
signed 16 bit
signed 32 bit
single precision floating point
double precision floating point
1, 2, 3 or 4
sequence is: b0, g0, r0, b1, g1, r1
Key tips
 Accessing image pixels
 Coordinates are indexed from 0
 Origin is top left or bottom left
 To access an element of an 8-bit 1-channel image,
I (IplImage* img):
I(x,y)~
((uchar*)(img->imageData + img->widthStep*y)[x]
Key tips
 To access an element of an 8-bit 3-channel image,
I (IplImage* img):
I(x,y)blue~
((uchar*)(img->imageData + img->widthStep*y)[x*3]
I(x,y)green~
((uchar*)(img->imageData + img->widthStep*y)[x*3+1]
I(x,y)red~
((uchar*)(img->imageData + img->widthStep*y)[x*3+2]
 To access an element of a 32-bit floating point 1-channel
image, I (IplImage* img):
I(x,y)~
((float*)(img->imageData + img->widthStep*y)[x]
Key tips
 In general to access an element of an N-channel image of type T
I(x,y)c~
((T*)(img->imageData + img->widthStep*y)[x*N+c]
 More simply using the provided macro
CV_IMAGE_ELEM(img, T, y, x*N+c)
 Efficient way to increment brightness of point (100, 100) by 30
CvPoint pt = {100, 100};
uchar *tmp_ptr=&((uchar*)(img->ImageData
+ img->widthStep*pt.y))[x*3];
tmp_ptr[0]+=30;
tmp_ptr[1]+=30;
tmp_ptr[2]+=30;
Key resources
 A brief intro to OpenCV:
www.cs.iit.edu/~agam/cs512/lect-notes/opencvintro/index.html
 A more detailed guide (also describes DirectX/DirectShow):
http://www.site.uottawa.ca/~laganier/tutorial/
opencv+directshow/cvision.htm
 Wikipedia
http://en.wikipedia.org/wiki/OpenCV
 Book
Learning OpenCV: Computer Vision with the OpenCV
Library (Paperback)
Gary Bradski and Adrian Kaehler
O'Reilly Media, Inc. (15 Jul 2008)
ISBN 0596516134