Paul Cockshott
Images In Java
ALMA TADEMA RIVALES INCONSCIENTES 1893
Introduction
• The AWT and how to put a simple image
onto the screen
• Layout managers
• The Jimage Class,
Summary of this section
• At the end of this lecture you should have
an idea of how to display a JPEG image
on the screen, and how to load it into the
Jimage class to carry out further image
processing.
Agenda
•
•
•
•
•
AWT Images
Image Producers and Consumers
Jimage class
Pixel Representations
JPEG files
Overview
• AWT abstract windows toolkit, supported
by JavaSoft
• Operating system independent layer for
windowing in Java
• Fiendishly obscure
• Designed around requirements of images
being streamed off the web
Connections
• Simple image display program to show
how to display a JPEG file
• Pipeline model of image production
• Jimages act as image consumers
• Jimages allow arithmetic on image
• Jimages provide output to AWT images
and JPEG
How to display a picture 1
import java.awt.*;
import java.awt.image.*;
import java.util.*;
class JPEGshow extends Frame {
...
static public void main(String[] args) {
if (args.length == 1) new JPEGshow(args[0]);
else System.err.println("usage: java JPEGshow <image file>");
}
}
This is a standard Java Program class with a public static void
main method
Constructor for JPEGshow
See slide on these
JPEGshow(String filename) {
super("JPEG show Example");
add(
new ImageCanvas(getToolkit().getImage(filename)
),
BorderLayout.CENTER);
setSize(700, 540);
show();
}
Read in a JPEG file
Size of a frame
The toolkit
• Each frame has associated with it a toolkit
object the provides an interface to OS
specific operations.
•
•
•
•
CreateImage
CreateMenu
CreateLabel
CreateMenuBar …. etc
Roll your own ImageCanvas
Constructor
class ImageCanvas extends Component
{
the image
Image image;
ImageCanvas(Image image)
{this.image = image;}
public void paint(Graphics g)
{ g.drawImage(image, 0, 0, this);}
}
just stores
Paint is called whenever a component must be shown,
the Graphics object does the actual drawing, it has to be
passed in because it is what knows about physically
drawing on the screen
Image Class
• Pipeline flow model of image processing
• Images are just tokens linking producers and
consumers
ImageProducer
Image
ImageConsumer
ImageProducer Methods
•
•
•
addConsumer(ImageConsumer ic) This method is used
to register an ImageConsumer with the ImageProducer
for access to the image data during a later
reconstruction of the Image.
removeConsumer(ImageConsumer ic) This method
removes the given ImageConsumer object from the list
of consumers currently registered to receive image data.
startProduction(ImageConsumer ic) This method starts
an immediate reconstruction of the image data
ImageConsumer methods
•
void setDimensions(int width, int height)
The dimensions of the source image are reported
using the setDimensions method call.
• Void setPixels(int x, int y, int w, int h, ColorModel model,
byte[] pixels, int off, int scansize)
The pixels of the image are delivered using one or
more calls to the setPixels method.
Image Class continued
ImageProducer
Image
ImageConsumer
Image.getSource
Images contain a pointer to their producer which holds the actual
data for the image. This can be recovered using the getSource
method. This allows a consumer to get at the pixel data of
an image by adding itself to the producer and starting production
Summary
• AWT is operating system independent
• Streaming image model
• Images as tokens
• Producer - consumer pipeline
• See chapters 6 of textbook
Buffered Image Class
• Standard AWT images are just tokens for
data streams.
• A BufferedImage actually contains the
data.
BufferedImage
Colour model
Raster
JPEGCodec class
• This class has factory methods to create
JPEG encoders and decoders:
• createJPEGDecoder(InputStream s)
• createJPEGEncoder(OutputStream d)
Read a BufferedImage
FileInputStream in = new
FileInputStream(“myfile.jpg”);
JPEGImageDecoder dec=
JPEGCodec.createJPEGDecoder(in);
BufferedImage im =
decoder.decodeAsBufferedImage();
getRGB
• You can access the pixels of a buffered
image using
int getRGB(int x, int y)
The default colour representation is:
alpha
Bit 31
red
green
blue
Bit 0
Writing pixels
• This can be done with the setRGB
method.
• This takes x, and y co-ordinates and a
pixel encoded as a 32 bit integer
•
im . setRGB(2, 5, 255);
• Would set pixel 2,5 to 255 = bright blue.
Creating sub images
• You can create a sub area within a
buffered image using the
• public BufferedImage getSubimage(
–
–
–
–
int
int
int
int
x,
y,
w,
h);
Method of BufferedImage
Jimage implements ImageConsumer
• Library of image processing classes
developed in the department
• Available for student practicals
• Algebraic rather than stream oriented
• Interfaces to MMX hardware under
windows
Algebraic orientation
By this we mean the it is structured around
algebraic expressions whose values are
images
Thus if A and B are images and  is some
operator then
AB
is also an image
Jimage operators
•

•
•
•

•
•
•
Arithmetic
I+J Universal plus(Universal)
I-J
Universal minus(Universal)
I×J Universal times(Universal)
I÷J Universal divide(Universal)
IUniversal abs()
Filtering
Jimage convolve(double[] k) convolve with symmetrical separable
kernel.
public abstract Jimage convolve(double[][] kernel)with non
separable kernel
Scaling
•
Jimage getScaledInstance(int nwidth, int
nheight)
This scales with bicubic interpolation.
•
Jimage getScaledInstance(int nwidth, int
nheight, int ndepth)
This method allows the depth as well as area of an
image to be altered if it is reduced the planes are
aggregated if increased they are interpolated.
More operations
Data access
int rgbpixel(int x,int y)
Converts the plane information into a pixel in the direct
color model of java.
public abstract int upixel(int x,
int y,
int plane)
- returns unsigned integer pixel
public abstract float fpixel(int x,
int y,
int plane)
Returns the pixel in the range -1 to +1.
Data Access
•
•
•
•
public abstract void setPixel(int x,
int y,
int plane,
double pix)
•
•
•
•
– Pixel information in range -1 to +1
public void setSubImage(int x,
int y,
int z,
Jimage im)
– Update an area of an image with another one. The other
one must not run off the edge of the one being written to.
The source of the copying is the 0th plane of the source
jimage.
Jimage input output
• public void putJPEGImage(
•
java.lang.String fileName,
•
int quality)
•
throws java.io.IOException
– Outputs the image to a jpeg file
• public boolean getImage(java.lang.String
fileName)
– Initialise the Jimage from the specified file. The file
must be jpeg or gif.
Jimage to AWT Image conversion
• public java.awt.Image getAWTImage()
• public java.awt.image.ImageProducer
getProducer()
Jimage implementations
JIMAGE CLASS HIERARCHY
COM.C3D.IMAGE
Jimage
abstract
COM.C3D.IMAGE
ByteImage
Generic Java
COM.C3D.IMAGE
ShortImage
Generic Java
COM.C3D.IMAGE
FloatImage
Generic Java
IntelBImage
Runs best on MMX
Windows only
IntelImage
Runs best on MMX
Windows Only
IntelFImage
Runs best on PIII
Windows Only
An example program
•
•
•
•
•
•
•
•
•
•
•
class Jimageshow extends Frame { Create Jimage
Jimageshow(String filename) {
with byte pixels
super("Jimage show Example");
Jimage raw=new ByteImage(100,200,3);
if (raw.getImage(filename)){
Jimage cooked = (Jimage)raw.times(0.3);
Multiply by 0.3
add(new ImageCanvas(cooked.getAWTImage()),
BorderLayout.CENTER);
setSize(700, 540);
show();
}
}
Convert to AWT
for display
Pixel Representations
When dealing with displays it is conventional to assume that
pixels are bytes holding numbers in the range 0 to 255.
0 Is assumed to be black
1 Is assumed to be white or maximum brightness of any
given colour.
For multicolour displays with 3 colour components, the
convention is to have 3 fields of range 0..255 to hold the
colour information.
Pixel Representations AWT
For multicolour displays with 3 colour components, the
convention is to have 3 fields of range 0..255 to hold the
colour information. The AWT does this with the class
Color.
• public Color(int rgb)
– Creates a color with the specified RGB value, where the
red component is in bits 16-23 of the argument, the green
component is in bits 8-15 of the argument, and the blue
component is in bits 0-7. The value zero indicates no
contribution from the primary color component.
– A Jimage returns this format with int rgbpixel().
Pixel Representations: Bytes
The byte data type in Java does not take on the values
0..255. Instead it takes on the values -128 to 127.
There are no unsigned bytes in Java.
This creates a problem for the representation of pixels in
Jimages.
The solution adopted is to adopt the following
representation
• -128 = black
• 0
= mid grey
• 127
= white
Pixel Representations: Floats
If byte pixels are signed then so must other representations
be.
The solution adopted is to adopt the following
representation for floats
• -1
= black
• 0
= mid grey
• 1
= white
Conversions between representations
unsigned
min value
-1
maxval
255
medianval m
0
range r
255
bytes
shorts
0
-128
-2048
127
2047
127.5
-0.5
255
4095
float
1
-0.5
2
As shown in table a pixel prin representation r is converted
to a pixel ps in representation s by the operation:
ps = ms+(rs(pr-mrrr
Signed Pixels : advantages
Signed pixels seem at first to be counter-intuitive but they
have numerous advantages.
•
•
•
A value of 0 or mid grey can be viewed as the ‘most likely’ value
that a pixel takes on in the absence of other information.
If you do arithmetic on images, in particular subtract one image
from another, then negative values of pixels naturally arise.
Signed pixels allow straightforward implementation of contrast
adjustments. For instance multiplying an image by 0.5 halves
the contrast in the image.
Signed Pixels : contrast adjustment
Signed pixels allow straightforward implementation of contrast
adjustments. For instance multiplying an image by 0.5 halves
the contrast in the image.
 0.5
 0.5
1
0.5
0.25
-0.25
-0.5
-1
Initial contrast range
Finalcontrast
range
Image Multiplication

Image Addition
+
Image subtraction
-
What Is Convolution
• Convolution takes a kernel of coefficients
and multiplies each pixel in a
neighbourhood by the corresponding
coefficient, and then sums the result
• x y p[I+x, j+y]*k[x,y]
• Will give the convolved pixel at position i, j
1 D convolution
• A 1 D convolution takes a one
dimensional array as a kernel and applies
it first in the X and then in Y dimension.
• This can often be performed faster than a
2d convolution
Image Convolution: smoothing
marble =
double[] k=
Note sum of
coefficients =1
marble.convolve(k)=
{0.1,0.1,0.2,0.2,0.2,0.1,0.1};
Image Convolution: sharpening
marble =
double[] k=
Note sum of
coefficients =1
number terms is odd
marble.convolve(k)=
{-0.3,1.6,-0.3}
Convolution in Java2D
• Java 2D provides a standard library for
convolution of buffered images
• This uses the class Kernel and
ConvolveOp
Kernels in JAVA 2D
float[] blur={ 0.0f, 0.1f, 0.0f,
0.1f, 0.6f, 0.1f,
0.0f, 0.1f, 0.1f};
Kernel k= new Kernel(3,3, blur);
im = new ConvolveOp(K).filter(im,null);
This will blur the image im by applying the 3 by 3 kernel
blur to it.
Importance of speed
• Image may contain a million pixels,
• Arithmetic may be required on each one
• Important to optimise operations or they
are very time consuming
• May need to use assembler kernels
• May need to use special purpose
instructions
Multimedia Extensions MMX
• Intel and other CPU manufacturers have
been adding to the instruction sets of their
computers new extensions that handle
multi-media data.
• The aim is to allow operations to proceed
on multiple pixels each clock cycle
MMX 2
• Standard Intel register set
8 General Registers
8 floating point registers
fp0
fp1
fp2
fp3
fp4
fp5
fp6
fp7
eax
ebx
ecx
edx
esp
ebp
esi
edi
32 bit
64 bit
MMX 3
• Standard Intel register set operating in
MMX mode
8 General Registers
8 multimedia registers
mm0
mm1
mm2
mm3
mm4
mm5
mm6
mm7
eax
ebx
ecx
edx
esp
ebp
esi
edi
32 bit
64 bit
MMX 4 motivation
• Existing operating systems must still work
unchanged
• Applications not using MMX run
unchanged
• No new state added to the CPU
Hence, shared use of the FP registers, since
these are already supported by exising
OS’s
MMX data formats
One 64bit integer QUADWORD
Two 32 bit integer DOUBLEWORDS
Four 16 bit WORDS
Eight 8 bit BYTES
Problem of overflows
•
•
A problem with limited precision arithmetic is that overflows frequently
occur. This can give rise to meaningless results: consider
200+175 = 375 but in 8 bit binary
11001000
+10101111
=101110111
Leading 1 is discarded
This leaves an answer of 119
decimal – clearly wrong
Using saturation
• You can fix this by using conditionals
unsigned char p1,p2,p3;
int I3= (int)p1 + (int)p2;
p3=(I3>255?255:(unsigned char)I3);
Expansion of the code 1
12:
00401043
00401046
00401048
0040104A
0040104D
0040104F
00401051
00401053
00401056
00401059
0040105C
0040105F
00401062
00401065
mov
xor
mov
mov
xor
mov
add
mov
mov
add
mov
mov
add
mov
j=(int)(*p1++)+(int)(*p2++);
ecx,dword ptr [ebp-4]
edx,edx
dl,byte ptr [ecx]
eax,dword ptr [ebp-8]
ecx,ecx
cl,byte ptr [eax]
edx,ecx
dword ptr [ebp-14h],edx
edx,dword ptr [ebp-8]
edx,1
dword ptr [ebp-8],edx
eax,dword ptr [ebp-4]
eax,1
dword ptr [ebp-4],eax
Expansion 2
13:
14:
00401068
0040106F
00401071
00401078
0040107A
0040107D
00401080
00401083
00401086
15:
00401088
0040108B
0040108E
*p3 = (unsignedchar)(j>255?255:j);
cmp
dword ptr [ebp-14h],0FFh
jle
main+6Ah (0040107a)
mov
dword ptr [ebp-18h],0FFh
jmp
main+70h (00401080)
mov
ecx,dword ptr [ebp-14h]
mov
dword ptr [ebp-18h],ecx
mov
edx,dword ptr [ebp-0Ch]
mov
al,byte ptr [ebp-18h]
mov
byte ptr [edx],al
p3++;
mov
ecx,dword ptr [ebp-0Ch]
add
ecx,1
mov
dword ptr [ebp-0Ch],ecx
Total of 26 instructions in the kernel
Alternative using mmx
Type represents 8 by 8bit integers
•
•
•
•
•
•
•
Iu8vec8 *v1,*v2,*v3;
int i,j,k;
for(i=0;i<31;i++){
*v3=(*v1++)+(*v2++);
v3++;
}
_mm_empty();
Arithmetic on 8 bytes at a time
Indicates MMX regs are now free
Optimised Assembler Loop
Go round only 32 times not 256
mov ecx ,32
; load counter with 32
l1: movq mm0,[esi]
; load 8 bytes
add esi,8
; inc src pntr
paddusb mm0,[edx] ; packed unsigned add bytes
add edx,8
; inc src pntr
movq [edi],mm0
; store 8 byte result
add edi,8
; inc dest pntr
loop nz,l1
; dec counter,
; repeat non zero
Total of 6 instructions in kernel
Speed Gain
• On image of 256x256 pixels
• Old C code executes 26*256*256
instructions = 1,703,936 instructions
• Optimised mmx code executes 6*256*32
instructions = 49,152
• Note that no compiler currently will give
the optimised code. It has to be hand
assembled.
Image Processing Library
• Intel Provide an image porcessing library
that can be downloaded from their web
site.
• It provides efficient access to the MMX
hardware.
• It provides frequently used Image
Processing Operations.
• It requires a set of DLLs in your path to
run
Image Processing Library 2
At the core of IPL is the ability to write to a
single API and get the best possible
results for any Intel processor. The
libraries have as many as six processorspecific branches for each function and
six sets of carefully written assembly
code, but only one entry point to each
function.
Image Processing Library 3
Image Processing Library 4
•
•
•
•
•
Use of Intel IPL complex and requires C
I have provided 2 java classes that call the IPL.
IntelBImage and IntelFImage. These are
documented in the Jimage web pages. They
inherit from ByteImage and FloatImage
To use them the Intel IPL must have been
installed on your machine and be on the path.
If you are forced to use Unix machines the
libraries will not be available to you.
Where to get more information
•
http://www.javasoft.com/products/jdk/1.2/d
ocs/api/java/awt/package-summary.html
• http://developer.intel.com/vtune/perflibst/ip
l/index.htm
• http://developer.intel.com/vtune/perflibst/ip
l/ipapi.htm
Feedback
• What did you not understand here?
• What would you like more information on?