CS 352: Computer Graphics
Input
and
Interaction
Chapter 3 - 2
Interactive Computer Graphics
What good is Computer Graphics?


JS1k canvas examples: 1 2 3 4 5
Games, visual demos…of what value? Is
there a Christian perspective?

Communication of information to the user
Data visualization, simulation, GUI
Even a word processor is an "interactive graphics
program"

Communication between users
"Collaborative environments" are hot (multi-player games?)
Social networking is transforming the world…

Interaction is an essential component
Chapter 3 - 3
Interactive Computer Graphics
Interaction


Much of the fun and utility of graphics is in
interaction: accepting user input that affects
the display
Paint programs, modeling, games, word
processors, spreadsheets, powerpoint …


User initiates input events such as clicking on a
menu or drawing a circle
Computer response by changing graphical display
Chapter 3 - 4
Interactive Computer Graphics
Chapter 3 - 5
Interactive Computer Graphics
Projects 3

Projects 3 will be a paint program


we'll learn 2-D graphics, interaction, event-loop
programming, double-buffering simple animation, basic
image processing
Features:





Pen, line, and rectangle tools
Color, pen size selectors
Image processing (sharpen, blur, detect edges)
Open, save images
Toolbar and menu for controlling application
Chapter 3 - 6
Interactive Computer Graphics
How to set up an application UI?




How
How
How
How
to
to
to
to
make
make
make
make
a menu?
a color picker?
a toolbar?
toolbar buttons pop in and out?
Chapter 3 - 7
Interactive Computer Graphics
How to paint?


How do I make a colored line follow the
mouse or fingertip?
Interactive prog:



Input devices
Event handlers
Event loop
Chapter 3 - 8
Interactive Computer Graphics
Input devices

Interaction requires handling input devices

Physical: mouse, fingertip, keyboard, joystick,
digitizer, accelerometer, head tracker…

Logical:






Text
Locator
Valuator (slider)
Stroke
Color picker
How to read?
Chapter 3 - 9
Interactive Computer Graphics
Input devices

Sample mode


Event mode



Program reads the current state of input device
(frequently)
Each click or motion generates an event that goes
on a queue
Program can process events from queue
HTML?
Chapter 3 - 10
Interactive Computer Graphics
Event-loop programming

Events: button press, slider value change,
menu selection, mouse move, keypress

Event loop:




Poll (or wait) for events
Process events, update state
Update display
(ideally: wait for rest of frame time to elapse)
Chapter 3 - 11
Interactive Computer Graphics
State-driven

Typically, the event loop is driven by a big
state table



e.g. depressing the paintbrush tool releases other
tools, puts me in "paint mode"
Good libraries will handle some of the
bookkeeping for you
You still typically have to handle some state
Chapter 3 - 12
State diagram
Interactive Computer Graphics
Chapter 3 - 13
Interactive Computer Graphics
Event loop in HTML/JS

HTML/JS provides event queue, support for
many basic events (mousedown, mouseup
mouseover, mousemove, keypress, key release,
value change, button click, etc.)


You are on your own for higher-level events,
e.g. clicking on a toolbar tool
It's also possible to set a function to
run every 15 ms, sample input devices
Chapter 3 - 14
Interactive Computer Graphics
Events for painting?

mousedown:




mouseup:


go into paint mode
store mouse position
draw initial dot
exit paint mode
mousemove:

if in paint mode


draw a line from old mouse position to current
set old mouse position to current position
Chapter 3 - 15
Interactive Computer Graphics
Event handling with jQuery

Binding events to functions
$(cpaint.canvas).bind('mousedown', cpaint.drawStart)

Processing events
cpaint.drawStart = function(ev) {
var x, y;
x = ev.pageX - $(cpaint.canvas).offset().left;
y = ev.pageY - $(cpaint.canvas).offset().top;
ev.preventDefault();
cpaint.paintmode = true;
cpaint.oldX = x;
cpaint.oldY = y;
Chapter 3 - 16
Interactive Computer Graphics
Looking neat and spiffy

How to avoid crinkles in your paint strokes?
Chapter 3 - 17
Interactive Computer Graphics
Looking neat and spiffy

How to avoid crinkles in your paint strokes?

Draw connected paths

Or just use line caps
Chapter 3 - 18
Interactive Computer Graphics
Menus

There are many jQuery menu plug-ins…
<ul id="mainmenu">
<li>File
<ul>
<li id="menuNew">New</li>
<li id="menuOpen">Open</li>
<li id="menuSave">Save</li>
------------------------------------$('#menuNew').bind('click', cpaint.clear);
$('#menuOpen').bind('click',cpaint.open);
$('#menuSave').bind('click',cpaint.save);
Chapter 3 - 19
Interactive Computer Graphics
Toolbar

How to make a toolbar?

How should buttons behave?

State diagram?
Chapter 3 - 20
Interactive Computer Graphics
Buttons

Widgets may have several states
State should be evident in visual feedback
E.g. how exactly does a button work? States?

State transition diagram



Most buttons: six states, with six different appearances








neutral
neutral-highlighted
neutral-depressed
selected
selected-highlighted
selected-depressed
Events: mousedown, mouseup, enter, exit
Transitions: what happens in each state under each event?
Chapter 3 - 21
Interactive Computer Graphics
Button state diagram
(buttons selectable, not unselectable)
N
S
NH
SH
N: neutral
H: highlighted
NHD
Press
Move
SHD
(usually mouse over)
S: selected
D: mouse down
Chapter 3 - 22
Interactive Computer Graphics
Other button considerations

Could also consider





Tooltips
Whether button merely clicks and releases or can
be selected
Whether button can be unselected
(e.g. B/I/U vs. Left, Center, Right)
Want consistent appearance, behavior over
whole program – or whole computer
Really need a library implementation and a
strict set of UI guidelines…
Chapter 3 - 23
Interactive Computer Graphics
4-State toolbar buttons: CSS
.toolbarCell {
background-color:#ddd; width:20pt; height:20pt;
border: solid #eee 2px;
box-shadow: 2px 2px 2px #666; }
#markerButton { background: url(img/paintbrush.png)
no-repeat center center; }
.selected { box-shadow: inset 2px 2px 2px #666; }
.toolbarCell:hover { border:solid #555 2px; }
Chapter 3 - 24
Interactive Computer Graphics
4-State toolbar buttons: JS
$('#markerButton').bind('click', {tool:"marker"},
cpaint.selectTool);
-------------------
cpaint.selectTool = function(ev) {
cpaint.tool = ev.data.tool;
// get tool name
$('.toolbarCell').each(function(index) { // unselect
$(this).removeClass('selected');
// others
});
var tool = '#' + cpaint.tool + 'Button'; // get ID
$(tool).addClass('selected');
// select
}
Chapter 3 - 25
Interactive Computer Graphics
Paintbrush size selector

How?
Chapter 3 - 26
Interactive Computer Graphics
Color picker

Google "jQuery color picker"…

<input type=“color”> works in some browsers
Chapter 3 - 27
Interactive Computer Graphics
How draw a line?

What kind of feedback is normal?


Rubber Banding
Events and actions?
Chapter 3 - 28
Interactive Computer Graphics
Save & restore



Create your own off-screen canvas
Copy it back on each mouse movement
Events and actions?
Chapter 3 - 29
Interactive Computer Graphics
Save & Restore event handling

Mousedown





Mousemove



enter line mode
remember mouse position as startx, starty
save screen
draw initial dot
paste saved screen
draw line from startx, starty to current mouse pos
Mouseup

exit line mode
Chapter 3 - 30
Interactive Computer Graphics
Analysis

Drawbacks?

slow – eats memory bandwidth for breakfast


copy smallest possible rectangle? Only points from line?
possible flickering

use double buffering?
Chapter 3 - 31
Interactive Computer Graphics
Double buffering







Have two frame buffers, "front" and "back"
Draw into back buffer
At vertical retrace time, swap buffers
This is a fundamental graphics technique…
…not built into canvas…
…though, some drawing aggregation seems
to happen automatically, behind the scenes;
…not usually necessary in canvas, but
browser dependent
Chapter 3 - 32
Interactive Computer Graphics
Fake Double Buffering in Canvas

Create off-screen canvas
canvasBuffer = document.createElement('canvas');
canvasBuffer.width = canvas.width;
canvasBuffer.height= canvas.height;
canvasBufferContext= canvasBuffer.getContext('2d');

Draw into off-screen canvas
canvasBufferContext.[drawSomeStuff…]

Copy onto display canvas
context.drawImage(canvasBuffer, 0, 0);
Chapter 3 - 33
Interactive Computer Graphics
How to erase?

Draw background color…
Chapter 3 - 34
Interactive Computer Graphics
Save, Load?

Save


Copy pixels to an image so users can right-click,
save-as
Load





What are the security risks?
Can only load files from the same server
Can use a file chooser if it's a local app
Security policies are not entirely in place
Consider chrome option
--allow-file-access-from-files
Chapter 3 - 35
Interactive Computer Graphics
Resizing


How would you allow the user to shrink,
expand image?
What would happen to image resolution?
Chapter 3 - 36
Interactive Computer Graphics
Store drawing commands

Realistically, to redraw effectively, you have
to save all drawing commands
pen color red
fill color blue
line width 5
circle 5 10 12
textfont Times 12
text 10 10 "hello world"



Replay commands to redraw scene
Could store commands in a file (e.g. Mac PICT file)
For some kinds of drawings, files are smaller and
more accurate than bitmaps
Chapter 3 - 37
Interactive Computer Graphics
Other toolkits






HTML offers basic UI capabilities in a crossplatform context
Writing your own UI extensions is tedious
jQuery plugins extend capabilities
Ideally, all UI elements ought to be built in
In the real world, they come in platformspecific 'toolkits' or 'windowing libraries'
E.g. MFC, QT, OpenLook, Cocoa …
Chapter 3 - 38
Interactive Computer Graphics
Image processing

Image processing examples:









blur, sharpen
detect edges
enhance contrast
noise reduction
posterize
fisheye
redeye reduction
find faces
…
Chapter 3 - 39
Interactive Computer Graphics
How to fade an image?


Could average pixel colors with white
Or just decrease alpha…
Chapter 3 - 40
Interactive Computer Graphics
How to blur an image?
Chapter 3 - 41
Interactive Computer Graphics
How to blur an image?






Blurring is an averaging process
Convolution: apply a convolution kernel, e.g.
1/9
1/9
1/9
1/9
1/9
1/9
1/9
1/9
1/9
Center kernel on pixel of interest
Multiply each kernel value by color value underneath
Add results
Gaussian smoothing
Chapter 3 - 42
Interactive Computer Graphics
Gaussian smoothing
Chapter 3 - 43
Interactive Computer Graphics
Chapter 3 - 44
Interactive Computer Graphics
How to sharpen an image?


It's a differentiation process
What's Unsharp Masking?
Chapter 3 - 45
Interactive Computer Graphics
How to sharpen an image?




It's a differentiation process
What's Unsharp Masking?
Subtract blurred version from original
Add back to original
Chapter 3 - 46
Interactive Computer Graphics
Sharpening




Subtract neighbors
Like subtracting a blurred version of the
image
Unsharp Masking
E.g. convolve with a kernel like one of these:
Chapter 3 - 47
Interactive Computer Graphics
Edge detection?
Wikipedia
Chapter 3 - 48
Interactive Computer Graphics
Sobel Edge Operators



Look for horizontal and vertical variation
1
2
1
-1
0
1
0
0
0
-2
0
2
-1
-2
-1
-1
0
1
Could do this at different sales or resolutions
Note: results maybe positive or negative
numbers; must normalize
Chapter 3 - 49
Interactive Computer Graphics
Image Compression



What if you created say six different lowerresolution images and only stored the
difference at each resolution?
Note: most of the data is in the highestfrequency component
An early image compression technique
Chapter 3 - 50
Interactive Computer Graphics
Fourier Transform
Chapter 3 - 51
Interactive Computer Graphics
The alg that changed imagery…


Discrete cosine
transform (DCT)
(Wikipedia)
Chapter 3 - 52
Interactive Computer Graphics
JPEG image compression




DCT to convert to frequency domain
Perceptual modeling
Coefficient quantization
Entropy coding
Chapter 3 - 53
Interactive Computer Graphics
Research in image processing
and computer vision


How would you clear up a blurry
photograph?
How would you recognize a face in a
photograph?
Chapter 3 - 54
Interactive Computer Graphics
Face recognition


One approach: statistically derive a set of
“standardized face ingredients” from many
examples, resulting in
“eigenfaces”
Represent each face as
linear combination
of eigenfaces
Chapter 3 - 55
Interactive Computer Graphics
Summary








Event loop programming
Interaction and event handling in HTML
State diagrams
Painting
Rubber banding
Double buffering
Basic image processing and convolutions
"Photoshop Nano"