Multimedia
Beta-version
http://www.youtube.com/watch_popup?v=j
EjUAnPc2VA#t=20
Overview -1
 Required background
 C , C++ or Java
 CS240 (Data Structures) or CS212 (for Engineers)
 This is NOT a graphics course (see CS460/560)
 Labs
 Data compression
 Elements of graphics in OpenGL
 Photo manipulation
 GIMP
 Steganography
 Creating & Rendering models
© D.J. Foreman 2012
2
Topics (general order of coverage)
 Compression
 Encoding/Decoding
 Codecs
 Huffman encoding example lab!
 Working with Images
 Image creation
 Photography
 Drawing
 RIB (Renderman Interface
Bytestream)
 Modeling
 OpenGL – lab!
 Makehuman, 3DCanvas, etc. lab!
 Image manipulation
 Gimp, Photoshop, etc. lab!
 Rendering
 Bitmaps
 Ray tracing
© D.J. Foreman 2012
 HTML, CSS & Javascript - lab!
 Animation
 Hand drawn
 Program generated
 Stop-motion
 Alice lab!
 DirectX & OpenGL
 Hardware acceleration
 RIB revisited lab!
 Shaders
 Steganography lab!
 Additional topics (time
permitting)
 Augmented reality
 Geospatial data systems
3
Software we might use (all free)
 Modelers and environments
 Ayam – ortho & 3D
 3DCanvas – 3D
 Blender – ortho & 3D
 Makehuman - 3D
 Renderers
 Aqsis
 Pixie – no GUI, file input
© D.J. Foreman 2012
4
Image formats
 Bitmap
 Also called “raster images”
 Stored as individual pixels
 Resolution dependent
 Screen resolution = 100 ppi
 Printing requires 150-300 ppi
 Vector
 Scalable primitives & formulas
 Resolution independent
© D.J. Foreman 2012
5
Bitmap formats
 BMP







(program generated)
GIF
(“, but © limits on use of format)
JPEG, JPG (photo)
PNG
(like GIF, w/o © limits)
PICT (“, Mac, BMP+Vector)
PCX
(“, Personal Computer Exchange)
PSD
(Photoshop)
TIFF
(“, can also be a container)
© D.J. Foreman 2012
6
Vector formats
 AI
 CDR
 CMX
 CGM
 DXF
 WMF
© D.J. Foreman 2012
(Adobe Illustrator)
(CorelDRAW)
(Corel Exchange)
(Computer Graphics Metafile )
(AutoCAD )
(Windows Metafile )
7
© D.J. Foreman 2012
8
Codecs
 A device or program for encoding and/or decoding a
digital data stream or signal
 Encoding A->D for storage or transmission
 DecodingD->A for playback
 Lossy vs. lossless
 Raw uncompressed Pulse Code Modulation (PCM)
audio, a digital representation of an analog signal
where the magnitude of the signal is sampled
regularly at uniform intervals
 E.g.; (44.1 kHz, 16 bit stereo, as represented on an audio CD
or in a .wav or .aiff file) is a standard across multiple
platforms.
© D.J. Foreman 2012
9
Codecs - 2
 May emphasize aspects of data
 Video:
 motion vs. color
 Audio:
 latency (cell phones) vs. high-fidelity (music)
 Data size:
 transmission speed or storage size vs. data loss
© D.J. Foreman 2012
10
Codec References
 Forensic Computing: A Practitioner's Guide by T.
Sammes & B. Jenkinson (Springer, 2000),
ISBN: 9781852332990 .
 http://www.garykessler.net/library/file_sigs.html
© D.J. Foreman 2012
11
Considerations
 Tradeoffs
 compression speed
 compressed data size
 quality (data loss)
 Areas of study
 information theory
 rate-distortion theory
 Popular algorithms (all lossless)




Lempel-Ziv (LZ)
LZW (fast decompression)
LZR (LZ-Renau) (ZIP files)
LZW (Lempel-Ziv-Welch) (for GIF files)
© D.J. Foreman 2012
12
Lossless Encoding
 Huffman compression
 Bit string representing any symbol is never a prefix of
the bit string representing any other symbol
i.e.; if 010 is the code for a symbol, then no other
symbol starts with 010.
 Frequency table must be known, computable or
included
 Lossless – every character gets encoded
 Arithmetic encoding
 Probabilistic algorithm
 Slightly superior to Huffman, but often patented!
© D.J. Foreman 2012
13
Methodology
 Compute a model of the data
 e.g.; a Huffman tree based on probability
 Probability determined from input file
 Map the data according to the model
 Read 1 symbol
 Search the model (tree) for that symbol




If it’s a Huffman tree, going left=0, right=1
Append each 0 or 1 to the code string
When symbol is found, you are done
Note: all symbols will have < 8 bits (thus compression)
© D.J. Foreman 2012
14
Huffman lab – more details
 Read /write are always in bytes.
 You may need to pad.
 You may have extraneous letters in the output due
to padding
 You don’t need to add a header to specify size.
 You have 2 weeks to do the lab
 Code may be in C, C++ or Java
 The readme specifies how to compile the program.
 Encoded data is sent to a binary file or a simple text
file (but isn’t “readable” as text).
 Encoded data can only be “checked” with a decoder
© D.J. Foreman 2012
15
Sample of prefix coding & compression
A simple Huffman-like code
(w/o probability basis)
A
110
B
010
C
0001
E
10
R
011
S
111
T
001
0
1
B e a r c a t s (8 bytes of text):
010101100110001110001111 encoded as
123456781234567812345678 3 bytes
© D.J. Foreman 2012
16
Compression-Model types
 Static
 Read all the data
 Compute frequencies
 Re-read the data & encode
 Dynamic
 Build simple basic model
 Modify model as more data is processed
© D.J. Foreman 2012
17
Standard Compressed-File Content
 Three parts:
 24-byte header with magic # (defines codec)
 variable-length annotation block
 contiguous segment of audio data.
 Storage methodology
 network (big-endian) byte order
 multi-byte audio data may require byte reversal in
order to operate on it by the arithmetic unit of certain
processors
© D.J. Foreman 2012
18
Example 1: Header file for .AU files
Following is from: #include <multimedia/libaudio.h>
typedef unsigned long
u_32;
// unsigned 32-bit integer
u_32
magic;
// the “magic number”
u_32
hdr_size;
// byte offset to start of data
u_32
data_size;
// length (optional)
u_32
encoding;
// data encoding enumeration
u_32
sample_rate;
// samples per second
u_32
channels;
// # of interleaved channels
typedef struct {
} Audio_filehdr;
AUDIO_FILE_MAGIC ((u_32)0x2e736e64) /* “.snd” */
© D.J. Foreman 2012
19
Audio Encoding Enumerations
AUDIO_FILE_ENCODING_MULAW_8
(1)
/* 8-bit ISDN u-law */
AUDIO_FILE_ENCODING_LINEAR_8
(2)
/* 8-bit linear PCM */
AUDIO_FILE_ENCODING_LINEAR_16
(3)
/* 16-bit linear PCM */
AUDIO_FILE_ENCODING_LINEAR_32
(5)
/* 32-bit linear PCM */
AUDIO_FILE_ENCODING_FLOAT
(6)
/* 32-bit IEEE floating point */
AUDIO_FILE_ENCODING_DOUBLE
(7)
/* 64-bit IEEE floating point */
AUDIO_FILE_ENCODING_ADPCM_G721
(23)
/* 4-bit CCITT g.721 ADPCM */
AUDIO_FILE_ENCODING_ADPCM_G723_3
(25)
/* CCITT g.723 3-bit ADPCM */
AUDIO_FILE_ENCODING_ALAW_8
(27)
/* 8-bit ISDN A-law */
“Linear” values are SIGNED int’s.
Floats are signed, zero-centered, normalized to ( -1.0 <= x <= 1.0 ).
© D.J. Foreman 2012
20
Example 2: ZIP files (PkWare)
Purpose
Size in bytes
Signature header
4
Required version
2
GP flags
2
Method
2
Last mod time
2
Last mod date
2
CRC-32
4
Compressed size
4
Uncompressed size
4
Filename length
2
Extra field length
2
File name
variable
extra
variable
Ref: http://livedocs.adobe.com/flex/3/html/help.html?content=ByteArrays_3.html
© D.J. Foreman 2012
21
More info on encoding
 http://www.pkware.com/documents/casestudies/
APPNOTE.TXT
 http://www.fileinfo.com/filetypes/compressed
© D.J. Foreman 2012
22
Recording Bit-Rate
 Constant (CBR)
 rate at which a codec's output data should be
consumed is constant
 Max bit-rate matters, not the average
 Uses all available bandwidth
 Not good for storage (lossy)
 Variable (VBR)
 Quantity of data/time unit (for output) varies
 Better quality for audio and video
 Slower to encode
 Supported by most portable devices post 2006
© D.J. Foreman 2012
23
Lab assignment – 1 – part 1
Write a program with 1 parameter
 If you are on an encoding team:




If you are on a decoding team:




Open a test file, (e.g.; “xyz.txt”)
Compress it using simple Huffman compression, using the
frequency table from my FTP site
Output compressed file (e.g.; “xyz.enc”) to SAME folder as
input
Open compressed file “xyz.enc”
Decompress the input file
Output file: “xyz.dec” so it can be compared to “xyz.txt”
The parameter is the full input file path & name
(i.e.; file names MUST NOT be hard-coded)
Remember: a code can be a single bit!
© D.J. Foreman 2012
24
Lab assignment 1 – part 2
 Use ONLY MY TABLE for the frequencies
 Use ONLY uppercase letters of the table
 Do NOT use the special character tables
 Find someone in class with the opposite
program and TEST your program to see if it
properly encodes or decodes a message
 Grading: tree construction, tree following,
proper binary output (encoders), proper
character output (decoders)
© D.J. Foreman 2012
25
Modeling
Containers
 A container is a FILE format, NOT a code
scheme
 E.g.; AVI & WAV are container formats
 Formats for storage
 Independent of encoding or content
 Others:
 Ogg, ASF, QuickTime, RealMedia, Matroska, DivX,
and MP4.
© D.J. Foreman 2012
27
Container file format (MP4/QT)
Movie header
Movie (metadata)
Track header
Track
Media
Media
header
Media
Handler
Media Information
Video
header
Data
information
Sample Table
© D.J. Foreman 2012
Chunk
map
Comp
time
Chunk
offset
Key
frame
Desc.
Decod
e time
Priorit
y map
Size
map
28
Pixels
 A pixel is 3 color-dots (squares) in a collection of dots







(squares) making up the picture.
Each color-dot is for one of the three primary colors,
RGB.
Minimum of 1 byte per color (depending on bit-depth,
which varies with camera), thus at least 3 bytes per
pixel.
A 10MP camera (30 million color-dots) needs 30M bytes
This gets compressed by the camera (JPEG format)
Final file size = 30 million bytes divided by the amount of
compression (compression ratio).
An 8:1 compression ratio would reduce that 30 million
bytes to 3.75 megabytes (as done on a 4 MP camera)
A RAW file has NO compression
© D.J. Foreman 2012
29
Image creation
 Basic mechanisms
 Photography
 Film
 Digital –Photoshop, Gimp
 Drawing
 Line
 BrushModeling
 Do it yourself
 OpenGL (2.x and 3.x)
 Modeling/rendering programs
 Breeze designer
 Ayam
 3DCanvas
 Makehuman
 Alice
© D.J. Foreman 2012
30
Photography
(film)
 Silver halide crystals (AgCl, AgBr, AgI)
 Exposure to light turns them black
 Developer removes the loose (exposed) halide leaving pure
silver (a negative image)
 Fixing bath flushes out unexposed AgHalide
 Greatest desire of photographers - a medium with:
 High speed (gathers light quickly)
 Low noise (no undesired artifacts)
 High resolution (very detailed images)
 Visible lines per mm (lpm) at a specified contrast level
 Notes:
 Larger “clumps” are faster, but limit resolution
 Smaller “clumps” are hard to coat evenly
 Randomness of coating prevents Moiré patterns
http://en.wikipedia.org/wiki/Moir%C3%A9_pattern
© D.J. Foreman 2012
31
FYI – a little chemistry
crystal
© D.J. Foreman 2012
Radiation (light)
32
Resolution
 35 mm film for comparison
 A 36 x 24 mm frame of ISO 100-speed film contains (≈) the equivalent
of 20 million pixels. 1 There are NO pixels in film!
 Full-frame digital (36 x 24mm = 3/2 image ratio)
 Canon EOS 5D, Nikon D3 (21MP) - $5K2
 Medium format digital (6 x 4.5cm)
 Phase One P40+ (39MP) ($22K)
 APS-C sized sensor (≈24x16mm = 3/2 image ratio)

(note: 2009 data below)
 Nikon D90 (12.3MP) - $900 , Canon Rebel (15MP) $700
 Used in most DSLR’s (note: image ratio 3/2)
 Other
 SONY Alpha (note: image ratio 4/3)
1 Langford, Michael. Basic Photography
(7th Ed. 2000). Oxford: Focal
Press. ISBN 0 240 51592 7.
2 Prices as of 2009, given for comparison purposes only
© D.J. Foreman 2012
33
Photography (digital)
 Image capture formats
 RAW – 3 separate images (RGB)
 [12 bits/pixel * (4288 * 2848)]/8=18,318,336 bytes per picture
 JPEG – compressed (16x, 8x, 4x)
 Some cameras do BOTH (RAW + JPEG) for each image
 Pixel notes:
 Sensor size 35mm (or larger) vs. APS-C
 Pixel count – any increase →
 decrease in pixel size or increase in sensor size
 Pixel size – smaller pixels give us:
 Greater resolution (finer details) - Good
 More “noise” (incorrect values) - bad
 Possible “adjacency” issues - bad
 Loss of sensitivity - bad
© D.J. Foreman 2012
34
Some Comparisons
 Digital media vs. film
 Pixels are the same (RGB)
 Pixel size varies with device (mfgr dependency)
 Pixel density varies
 Zooming digitally does NOT give you more detail
 (in spite of what they do on TV shows)
 Optical zoom is useful
 Uses optical magnification of the image on the target
 Expansion of light rays, not multiplication of pixels
© D.J. Foreman 2012
35
Working Definitions - 1
 Graphic Model
 data structure
 nodes represent random variables
 Pairs of nodes connected by arcs correspond to
variables that are not independent
 Graphic modeling
using a Graphic Model to simulate a real-world
object (this is not a formal definition)
 Rendering
generating a 2D image from a 3D graphic model
© D.J. Foreman 2012
36
Working Definitions - 2
 Geometric primitives –
 Points, line segments and polygons
 Described by vertices
 Control points –
 Special points attached to a node on a Bézier curve
 Alter the shape and angle of adjacent curve segment
 Evaluators –
 Functions that interpolate a set of control points
 Produce a new set of control points
© D.J. Foreman 2012
37
Renderman Interface Bytestream (RIB) © Pixar
 A standardized interface
 https://renderman.pixar.com/products/rispec/rispec_
pdf/RISpec3_2.pdf
 A plaintext file created by modeling programs
 Input to rendering programs
© D.J. Foreman 2012
38
Context




Context is the “version” of OpenGL in use
OpenGL 2.x (current to 2009)
OpenGL 3.0 (2.0 + deprecated functions)
OpenGL 3.1 only newer graphics cards
 Many old (<= 3.0) functions are deprecated
 Many actions now done via shaders
 1st create a context (like the chicken & the egg)
 create an old (e.g.; 3.0) context
 activate it
 create the new context
 deactivate old context
 Old context needed to create new one
© D.J. Foreman 2012
39
OpenGL 2.0 Fixed Function
Pipeline
Vertex Data
Pixel Data
© D.J. Foreman 2012
Vertex
Processor
Fragment
Processor
Frame
Buffer
Per-vertex operations:
•Eye-space coordinates
•Colors
•Texture coordinates
•Fog coordinates
•Point size
40
The Vertex Processor
 Lights object vertices {V}
 Determine color to assign
 Pass color to pixel processor
 This is then interpolated across the polygon
 Transforms object vertices {V}
 Construct projection matrix
 transform {V} from world space to camera space
 Transform {V} from camera space to screen space
 Multiply vectors with matrices
 i.e.;
vectora * column-vectorb
 [x0, y0,z0] * [x1,y1,z1]  x0*x1 + y0*y1 + z0*z1
 Has no information regarding connectivity
 Therefore can’t do back-face clipping
© D.J. Foreman 2012
41
OpenGL 2.0 Vertex Processor
Pipeline
Vertex Shaders
Vertex
Coordinates
Model View
Matrix
Normal Vector
Model View
Matrix
replace everything in
the dotted box
Primitive
Setup
Color Values
Texture
Coordinates
Projection
Matrix
Clipping
Texture
Matrix
Fog
Coordinates
© D.J. Foreman 2012
42
DirectX vs. OpenGL
 Direct3D:
 designed as a h/w
interface
 Application manages
h/w
 Mostly for games
© D.J. Foreman 2012
 Opengl:
 “may” be hardware
accelerated
 Implementation
manages h/w
 Mostly for pro use
(more general-purpose)
43
The Model-view Matrix
 Used to transform vertices that enter the 3D API
before they are rendered to the screen
 Called the “world and view” matrix in DirectX
© D.J. Foreman 2012
44
Graphic fragment
 The data needed to shade a pixel
 The data needed to test whether the fragment
survives
 Based on front-to-back relations
© D.J. Foreman 2012
45
Fragment Processing
From primitive setup
Bitmaps/Pixel
Rectangles
Texture
Mapping
Color
Summation
fragment
shader
replaces
these
Per-pixel
Fogging
Fragment Tests
(survival, etc)
Frame Buffer
© D.J. Foreman 2012
46
Lab 2- basic modeling
 Write a program using the OpenGL 2.0 interface:
 Create 3 figures:
 Triangle (2D)
 Rectangle (2D)
 Irregular pentahedron (5-sided, 3D figure) (a pyramid)
 See: http://en.wikipedia.org/wiki/Pentahedron for examples
 ANY one Vertex pinned at 0,0,0
 Allowed to be non-equilateral
 Draw all 3 axes (optional, but HIGHLY recommended)
 The triangle is within the bounds of the rectangle
(or vice-versa), both objects in same plane, different colors
 The pyramid must have different colors on all 5 sides
 Create movement controls for the pyramid:




Cursor keys ←→↑↓ change camera location for world view
L & R keys rotate pyramid (CW/CCW) about Z-axis
Pinned pyramid Vertex STAYS at 0,0,0
The pyramid rotates, the “camera” position remains fixed
© D.J. Foreman 2012
47
Lab 2 grading
 Point values
 Rectangle and triangle both exist
 Pyramid exists
 All have colors applied
 View rotation (up/down & left/right)
 Pyramid can be rotated (+ and -)
© D.J. Foreman 2012
5
10
5
10 pts EACH
20 pts
48
Some Examples for lab 2
0,0,0
0,0,1
0,0,0
6-vertex
pentahedron
Valid for lab
Still has 5 sides!
0,0,1
© D.J. Foreman 2012
49
OpenGL
 PC Requirements
 OPENGL library - for manipulating the model
 opengl32.lib
 opengl32.dll
(comes with Windows/XP & 7)
 glu32.dll
“
 glu.h and gl.h
“
 FreeGlut (newer – Open Source)
© D.J. Foreman 2012
50
Window Control
 ONE of the following:
 Glut - note the “t” after the “glu”
http://www.opengl.org/resources/libraries/glut/glut_down
loads.php#windows
 glut32.dll
 glut32.lib
 glut.h
”
”
 FreeGlut (newer – Open Source)
http://freeglut.sourceforge.net/
 freeglut.dll
 freeglut.lib
 glut.h
 glut.h does a #include of gl.h and glu.h
© D.J. Foreman 2012
51
Programming in OpenGL





Define window
Define objects
Initialize “callback” functions
Initialize window
Run the gl main loop
 Useful tutorial:
http://www.thepcspy.com/read/opengl__introduct
ion_and_getting_started
Note double underscore in name above
© D.J. Foreman 2012
52
OpenGL Program Structure
 Two parts
Modeling, coloring, etc.
1.


Use standard C code, with calls to OpenGL functions
Your program runs as subroutines WITHIN the
OpenGL MainLoop
2. Callback functions
 Called by OpenGL, from OUTSIDE your program
 Display
 Reshape
 Purpose of “registering” callbacks


passes a pointer to your functions
allows OpenGL to call them.
© D.J. Foreman 2012
53
Views
 Orthographic
 All views are 2-dimensional
 Not good for games ignores the z-axis
 Perspective
 See model from “camera position” via “viewport”
Φ
camera
© D.J. Foreman 2012
viewport
54
Image Rotation
Left-handed
tX2 + c tXY - sZ tXZ + sY
tXY+sZ tY2 + c tYZ - sX
tXZ - sY tYZ + sX tZ2 + c
0
0
0
Right-handed
0 tX2 + c tXY + sZ tXZ - sY
0 tXY-sZ
tY2 + c tYZ + sX
0 tXY + sY tYZ - sX tZ2 + c
1
0
0
0
0
0
0
1
c = Cos (theta)
s = Sin (theta)
t = 1-Cos (theta)
<X,Y,Z> is the unit vector for your arbitrary axis.
By using the axis/angle representation for your
rotations, it is possible to avoid gimbal lock.
© D.J. Foreman 2012
55
Program structure
 Movement functions
 Shape defining functions (lines, polygons, etc.)
 Reshape function (for window re-sizing)
 Init function
 Display function
 Main (includes call to glutMainLoop)
NOTE: gl, glu & glut prefixes on functions.
Be careful!
© D.J. Foreman 2012
56
GlutReshapeFunc
 Transforms geometry for new window size
 Used after every frame is drawn or after a re-size
void reshape (int w, int h)
{
glViewport (0, 0, w, h);
glMatrixMode (GL_PROJECTION);
glLoadIdentity ( );
glOrtho(….);
glMatrixMode (GL_MODELVIEW);
}
© D.J. Foreman 2012
57
GL_Load_Identity
 Replace current matrix with Identity matrix
 Prevents applying multiple perspectives
© D.J. Foreman 2012
58
gluLookAt
 Creates a viewing matrix
 set the camera position and orientation
 derived from
 an eye point – where the viewer is located
 a reference point for the center of the scene
 an UP vector – maps the given vector TO the y axis
 Describes any “tilt” of the camera
 Defines the new value for “up” as seen by the viewer
© D.J. Foreman 2012
59
Matrix Modes
 GL_PROJECTION sets up our 'camera'
 GL_MODELVIEW for drawing
 GL_TEXTURE adjusts how textures are drawn
onto shapes
© D.J. Foreman 2012
60
Keyboard-related functions
 glutSpecialFunc
 glutKeyboardFunc
© D.J. Foreman 2012
61
Basic Functions in “main”
 Register your “callback” functions
glutSpecialFunc (your function name);
glutKeyboardFunc (your function name);
display
reshape
 Run the program
dj_init(); // your init routine (if any)
glutMainLoop();
© D.J. Foreman 2012
62
Skeleton of program – slide 1
#include <GL/glut.h>
#include <stdio.h>
#include <math.h>
#include <stdlib.h>
float ex,ey,ez,theta;
void leftMotion (int x, int y);
void rightMotion (int x, int y);
GLfloat pvertices [][3]= {
};
// and then their colors
GLfloat pcolors[][3]={
};
// now define the polygon that USES those vertices
//define any ONE face of the pyramid at a time
void pface(int a, int b, int c) // pface is a name I made up for “pyramid face”
{// 3 vertices define a face
glBegin(GL_POLYGON);
glEnd();
}
© D.J. Foreman 2012
63
Skeleton of program – slide 2
void draw_pyramid() // implement the faces of the figure
{
glPolygonMode (GL_FRONT_AND_BACK, GL_FILL);
}
void draw_square()
{
}
void draw_triangle()
{
}
void draw_axes()
{
glPushMatrix ();
glBegin(GL_LINES); // as pairs of points (default action of GL_LINES)
glEnd();
glPopMatrix ();
}
© D.J. Foreman 2012
64
Skeleton of program – slide 3
void dj_display()
//callback for glutDisplayFunc
{ /* this is the function that draws the graphic object in the precreated window */
glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT);// clear window
glLoadIdentity();
//
eye
at
up
gluLookAt(ex, ey, ez, 0,0,0, 0,1,0);
draw_square();
draw_triangle();
draw_axes();
// last in code, means display on top.
draw_pyramid();// now draw the pyramid
glFlush();
glutSwapBuffers ();
}
© D.J. Foreman 2012
65
Initialize the program – slide 4
void dj_init() // define the palette & background
{
glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE);
glClearColor(0.0, 0.0, 0.0, 0.0); // set clear color to black
// now specify that parameters will be RGB values & opacity
glColor4f(0.0, 0.4, 0.5, 0.5);
}
© D.J. Foreman 2012
66
Create a colored line – slide 5
Void draw_a_line() {
glPushMatrix ();
glBegin(GL_LINES);
glColor3f(0.75, 0.0, 0.0); // RGB
glVertex3f(0.0, 0.0, 0.0); // point 1 (x,y,z coordinates)
glVertex3f(15.0, 0.0, 0.0); // point 2
glEnd();
glPopMatrix ();
}
NOTE: function names end in “3f”
3=number of arguments
f=floating point values
© D.J. Foreman 2012
67
Skeleton of program – slide 6
int main(int argc, char*argv[])
{
// don’t leave this out!!
glutInit(&argc, argv);
glutInitWindowSize(wd,ht);
glutInitWindowPosition(wx,wy);
glutCreateWindow("DJ's 1st"); // define name on window
glutSpecialFunc(myspecialkeys);
/* register the call-back functions with Glut*/
glutKeyboardFunc(mykey);
glutDisplayFunc(dj_display);
// more setup
glutReshapeFunc(dj_reshaper);
dj_init();
// Some texts show init being called AFTER the glutDisplayFunc call,
// but it doesn't actually CAUSE display action, it just sets up the environment.
glutMainLoop();
// the only real executable stmts in program
return 0;
// (except if, for, etc)
}
© D.J. Foreman 2012
68
Basic Display Function
glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT);
glLoadIdentity();
//
eye_point,
at(x,y,z)
up (x,y,z)
gluLookAt (ex, ey, ez,
0,0,0,
0,1,0);
draw_triangle/rectangle/pyramid, etc.
draw_axes;
glutSwapBuffers (); //if double-buffered, back buffer moves to
front at retrace. Also does a glFlush
glFlush();
// only needed if glutSwapBuffers is not used
© D.J. Foreman 2012
69
openGL 3.1
 Using shaders & Vertex Array Objects
glBindVertexArray(my_vao_ID[0]);
// select 1st VAO
glDrawArrays(GL_TRIANGLES, 0, 3); // draw 1st object
 note difference from glBegin (GL_POLYGON)….
 Note: name of out(put) variable in vertex shader
must be the same as in(put) variable in fragment
shader .
 i.e. vertex -> fragment (as in original pipeline)
© D.J. Foreman 2012
70
The Vertex Shader
 Transform vertex's virtual 3D position to 2D
screen coordinate
 Manipulate
 position, color, and texture coordinate
 Output to:
 Geometry shader
 generate new graphics primitives, from those at
beginning of pipeline
 Rasterizer
© D.J. Foreman 2012
71
Drawing
 Last drawn object is in “FRONT”
 MUST set bit depth AND enable bit depth test
 Rotation
 Around a vector starting at 0,0,0
Use glRotate(Ѳ, x,y,z)
 Around any vector NOT anchored at 0,0,0
Requires pre-multiplication of matrices with the identity
matrix
© D.J. Foreman 2012
72
Views
 Orthographic
 view direction is perpendicular to projection plane
 Matrix mode
 Modelview
 Projection View
© D.J. Foreman 2012
73
The RenderMan Interface Bytestream
 "The RenderMan Interface is a standard interface
between modeling programs and rendering
programs capable of producing photorealistic
quality images."
 http://www.faqs.org/faqs/graphics/renderman-faq/
 http://www.faqs.org/faqs/graphics/rendermanfaq/#ixzz0ggQ8591Y
 RIB programs the target device
 (like PostScript does for printers)
© D.J. Foreman 2012
74
RIB files
 The specification from Pixar
 https://renderman.pixar.com/products/rispec/rispec_
pdf/RISpec3_2.pdf
 Course notes from SIGGRAPH conferences:
 Siggraph '90 course #18 - The RenderMan Interface &
Shading Language
 Siggraph '92 course #21 - Writing RenderMan
Shaders
© D.J. Foreman 2012
75
Image Manipulation
 Photographic images
 Gimp, Photoshop, etc
 Modeling
 OpenGL – for creating and lighting a model
 modeling programs
 Rendering
 converting a 3D (possibly wireframe) model to a 2D
image, then lighting and (possibly) shading it
 Ray tracing
 another way of lighting & shading an image
 very CPU intensive
© D.J. Foreman 2012
76
Ray tracing
 Method for calculating the path of waves/ particles
through a system.
 Two distinct forms:
 Ray tracing (physics), which is used for analyzing optical
and other systems
 Ray tracing (graphics)
 Used to generate a 3D image
 Trace the path of light through pixels in an image plane
 Simulate the effects of striking virtual objects
 Simulate reflection , refraction, scattering, and chromatic
(color) aberration.
 Compute intensive
 Ray-casting + follow reflections, refractions & shadows
© D.J. Foreman 2012
77
Light terms
 Absorption – unreflected+refracted light
 Reflection – change of direction at an interface
 e.g; air/solid
 Angle = angle of incidence
 Refraction – change of direction through media
 e.g. air/water
 Angle depends on refractive indices of both media
© D.J. Foreman 2012
78
Ray tracing methodology
 Rays start at viewpoint, go through viewplane
 Eliminates unused rays from source
 Rays that do not strike objects
 Rays that are blocked by objects closer to viewer
 Alternative (from light source) is photon mapping
 Each ray tested for intersections with objects
 Rays can be computed in parallel
 Algorithms
 Estimate the incoming light at the intersection
 Examine the material properties of the object
 Combine above to calculate final color of the pixel
© D.J. Foreman 2012
79
Basic Ray tracing algorithm - 1
 Initiate a ray (from viewpoint)
 shade_pixel (original ray)
 This is a recursive call
 (see next slide)
 Apply “final” shade to pixel
 Flat shading – assumes mono-colored object
 Cook-Torrance – uses physical attributes of object
 Lambertian shading (cosine method)
 Point brightness based on normal vector at point and
vector from point to light source
http://www.cs.unc.edu/~rademach/xroads-RT/RTarticle.html
© D.J. Foreman 2012
80
Ray tracing algorithm - 2
 Shade_pixel (ray)
 Find 1st intersection
 C1= color of intersection point
 If reflective
 Create reflection ray
 C2=Shade_pixel (new ray)
 If transparent
 Create refraction ray
 C3=Shade_pixel (new ray)
 Return combined color value (C1, C2, C3)
© D.J. Foreman 2012
81
Final coloring
shade = light_vector • normal_vector
if ( shade < 0 )
shade = 0
point_color = object_color *
( ambient_coefficient +
diffuse_coefficient * shade )
© D.J. Foreman 2012
82
Breeze Designer/Ayam - lab
 Create a model of NCC1701 in BD
 Notes:
 The “front” view is actually the back view
 Perspective view is not accurate








Save your model (it will be a .CAD file)
Use the “print screen” key (copy to clipboard)
Open RIB with Notepad & print it
Paste into Word & print it
Export your model as a RIB file
Open RIB file (use “open”, not “import”) in Ayam
Click on “Run”
Use print screen again
© D.J. Foreman 2012
83
© D.J. Foreman 2012
84
Operational concepts
 3 windows:
 Image – contains the actual results
 Tools – operators, such as clone, paint, etc
 Layers – portions of the final image
 Avoids need for changes to original
 Stackable
 Allows separation of collections of changes
 Allows complex changes to be applied independently of
other changes
© D.J. Foreman 2012
85
Gimp - lab








Load a digital landscape or cityscape photo
Load a picture of yourself
Select the image portion that is just yourself
Extract the image of yourself – save it as a JPG
Insert it (inverted 90-180 °) into the ‘scape
Save the new image as a new JPG file
Send all 3 files (‘scape, you, merge) to the TA
Grading:
 Image properly selected (20 pts)
 Image rotated (20 pts)
 Image properly overlayed (60 pts)
© D.J. Foreman 2012
86
Video games vs. Movies
 Elements to manipulate
 Content
 Perspective
 Lighting/shading
 Sequence
 Coloring
 Video game images
 Dynamic
 Movie images
 Pre-determined
© D.J. Foreman 2012
87
Alice
 Developed at CMU
 Language-free programming environment
 Point & click usage
 Rapid prototyping
© D.J. Foreman 2012
88
Alice environment
 List of “world” objects
 Insert into methods
 Modify characteristics (length, width, etc.)
 List of world details
 Properties
 Methods (user created)
 Functions (built-in methods)
 World window (results)
 Method builder
© D.J. Foreman 2012
89
Alice - Programming
 Programmer
 Selects “verbs” from list
 For
 While
 Etc.
 Applies values for properties
 Statement structure pre-defined
 Blocks pre-defined
 Function calls
 for
 if… then… else
 No need to learn a “language”
© D.J. Foreman 2012
90
Alice Lab
 Open any of the Alice worlds
 Apply the following rules:
1.
2.
3.
4.
5.
6.
Display 3 articulated characters (A, B and C)
A flips onto its head and rotates slowly (5 times)
B waves both of its arms slowly (5 times)
C walks around A and B while (2 and 3 happen)
C changes direction and walks around A and B once
(while 2 and 3 repeat)
Stop
Timing, size, shape, position are your choice
© D.J. Foreman 2012
91
STEGANOGRAPHY
and LSB analysis
What is it?
 Hiding a message in plain sight
 Inside another message
 Original is called the “cover”
 Presence of message is not obvious
 Can be done with text or graphics
© D.J. Foreman 2012
93
How is it done?
 Many algorithms for hiding a message
 Embedding is easier than detection
 Extraction is straight-forward if:
 you know there is a hidden message
 you know the algorithm used
 Can be very complex to detect & extract
© D.J. Foreman 2012
94
LSB embedding – using
Matlab/Freemat
 Concept:
 Change the least significant bit of a set of bytes
 1→0 and 0→1
 For all bytes
 Repeat for multiple sets of bytes
 Example with grayscale images
 One pixel is 8 bits
 Change the last bit
 Changes that pixel’s shade VERY slightly.
 Repeat for every pixel
© D.J. Foreman 2012
95
LSB embedding - algorithm
Embedding function Emb (using Matlab or Freelab syntax)
c = imread(‘my_decoy_image.bmp’);
% Grayscale cover image
% ‘b’ is a vector of m bits (secret message)
k = 1;
% k is an index into the message bits
for i = 1 : height
for j = 1 : width
LSB = mod(c[i, j], 2);
if k>m | LSB = b[k]
%m is msg length in BITS
s[i, j] = c[i, j];
else
s[i, j] = c[i, j] + b[k] – LSB;
end
k = k + 1;
end
end
imwrite(s, ‘stego_image.bmp’, ‘bmp’); % write new image “s” to disk
© D.J. Foreman 2012
96
imread
 The return value “c” is an array containing the
image data.
 If the file contains a grayscale image, “c” is an M-by-N
array.
 If the file contains a truecolor image, “c” is an M-byN-by-3 array.
 The class of “c” depends on the bits-per-sample of
the image data, rounded to the next byte boundary.
 E.g.; imread returns 24-bit color data as an array of
uint8 data because the sample size for each color
component is 8 bits.
© D.J. Foreman 2012
97
LSB extraction - algorithm
Extraction function Ext (Matlab syntax)
s = imread(‘stego_image.bmp’); % Grayscale stego image
k = 1;
for i = 1 : height
for j = 1 : width
if k  m
% how do I set the value of m??
b[k] = mod(s[i, j], 2);
k = k + 1;
end
end
end
% b is the extracted secret message as a bit string
© D.J. Foreman 2012
98
Properties of LSB flipping
 LSBflip(x) = x + 1 – 2(x mod 2)
 FlipLSB(x) is idempotent, e.g., LSBflip(LSBflip(x)) = x for all x
 LSB flipping induces a permutation on {0, …, 255}
0  1, 2  3, 4  5, …, 254  255
 LSB flipping is “asymmetrical” (e.g., 3 may change to 2 but never to 4)
 | LSB(x) – x | = 1 for all x (embedding distortion is 1 per pixel)
© D.J. Foreman 2012
99
Augmented Reality
 Addition of computer-generated imagery to a
real-world view
 Differs from “mediated reality”
 Real-world modified (+ or -)
 Real-time
 Adds interactivity to real world view
 “Computer vision”
 Systems that extract info from images
 Medical/photo/video scanners
 “Object recognition”
© D.J. Foreman 2012
100
AR usage
 Robotics
 Machining
 Assembly
 Human interaction
 Auditory & visual cueing
 Machine control
© D.J. Foreman 2012
101
AR developments
 Heads-up displays
 Visual data projected onto a transparent screen
 Windshield, cockpit, etc
 Used in auto and military ground- & air-craft
 Head-mounted displays
 Device with glasses, goggles, etc
 Also used in VR environments
 Incorporated technology




GPS
Optical & wireless sensors
Gyroscopes
Accelerometers
© D.J. Foreman 2012
102
Adobe Flash
 (Originally Macromedia Flash)
 Vector graphics
 Uses:
 Video in web pages (.swf)
 Movies
 Video games
(.swf)
(.swf)
(.flv or .exe)
 Standalone video
 Used with a flash player
 Telephones and other web-enabled devices
© D.J. Foreman 2012
103
Flash - 2
 Support
 Adobe Flash CSn
 Adobe Integrated Runtime
 Installs (silently) with Adobe Reader
 Usage in a web page
<object data="movie.swf”
type="application/x-shockwave-flash"
width="500” height="500">
<param name="movie" value="movie.swf" />
</object>
© D.J. Foreman 2012
104
Virtual Reality Modeling Language
 VRML
 Text file format
 URLs may be associated with graphical components
 May have JS or Java attached
 Called “worlds”
 File extension =.wrl
 Many modeling programs can output VRML
 Supported by OpenVRML
 Succeeded by X3D
© D.J. Foreman 2012
105
X3D
 Extensible 3D Graphics
 ISO standard
 XML-based file format
 Successor to VRML
 Applications which can parse X3D files:
 Project Wonderland
 Blender
 Freewrl
© D.J. Foreman 2012
106
X3d-Sample.x3d
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE X3D PUBLIC "ISO//Web3D//DTD X3D 3.2//EN"
"http://www.web3d.org/specifications/x3d-3.2.dtd">
<X3D profile="Interchange“ version="3.2"
xmlns:xsd="http://www.w3.org/2001/XMLSchema-instance"
xsd:noNamespaceSchemaLocation="
http://www.web3d.org/specifications/x3d-3.2.xsd ">
<Scene> <!-- viewable using freewrl -->
<Shape>
<IndexedFaceSet coordIndex="0 1 2">
<Coordinate point="0 0 0 1 0 0 0.5 1 0"/>
</IndexedFaceSet>
</Shape>
</Scene>
</X3D>
© D.J. Foreman 2012
107
DIGITAL AUDIO
Terms
 Level – power or volume
 "dB" – decibel - signal strength ratio
 0 dB is the human hearing threshold
 Perception is proportional to log10 values of level
 Breathing = 10dB
 A whisper = 30dB
 Lawnmower = 85dB
 Snowmobile = 105dB
 Jet engine (running) = 130 dB
 Rock concert = 110 – 140 dB
 Hearing damage starts at 85 dB
 Pain starts at 130dB
© D.J. Foreman 2012
109
Basic editing
 Effects
 Filter – specific freq
 Equalize - volume
 Modulate (speed, reverb)
 Time & pitch – stretch, change pitch
 Noise reduction – breathing, background, crackles
 Delay & echo – "room" effects
 Clip - cut segments or limit volume/frequency
 Copy/move/paste - segments
© D.J. Foreman 2012
110
Multi-track editing-1
© D.J. Foreman 2012
111
Multi-track editing-2
 Editing left/right channels of stereo signal
 Comparison (e.g.; forensics)
 Merging
 Appending
 Re-arranging
MP3
© D.J. Foreman 2012
MID
112
Music creation
 Midi – Musical Instrument Digital Interface
 Specifies musical instructions
 Instrument to play
 Notes, durations, volumes
 Binary file much smaller than MP3
 Two ways:
1. From instrument
 Instrument connects to PC
 PC software creates file/sheet music
2. From written instruction

© D.J. Foreman 2012
Use a GUI to define notes, positions
113
DIGITAL VIDEO
© D.J. Foreman 2012
114
TV Formats
 Digital (ATSC)




Over the Air (OTA) – full HD (1080P)
Scrambled cable – full HD (1080P)
"standard definition" (SD=480i)
QAM – NOT scrambled – 720P to 1080P
 Cable-box NOT required
 Includes ATSC and converted NTSC
 Limited availabilty
 Analog (NTSC)
 Unscrambled cable (limited availability)
 VCR output
 Only in SD
© D.J. Foreman 2012
115
Recording
 Digital Cameras
 SD card files - AVI format
 DVD –MPG-2 or -4
 Tape – DV format requires real-time transfer
 Digital TV tuners
 Receives pre-encoded (OTA/cable) MPG-2 to PC
 E.g.; SiliconDust HD Homerun
 Analog TV tuners
 Receives NTSC QAM & cable/VCR analog
 Does hardware encoding to MPG-2
 E.g.; Hauppauge WinTV HVR 2250 also does OTA digital
© D.J. Foreman 2012
116
Remote Playback
 Hardware
 Boxee - pre-recorded formats + internet
 WDTV Live – pre-recorded formats + internet
 Sage TV – pre-recorded formats + Tuner scheduler
 Supports only SageTV streaming receiver
 Bought by Google, removed from market




Roku - pre-recorded formats + Apple HLS (HTTP Live TV streaming)
Xbox – WMV only
PS3 - MPG-2 only
Wii – flash only using Opera browser
 Software
 Tversity
 http://www.maximumpc.com/article/streaming?page=0,1
 WMC –Windows 7 (>=Home Premium)+ Xbox
 SageTV
 Hauppauge WinTV
© D.J. Foreman 2012
117
DV EDITING
© D.J. Foreman 2012
118
Terms
 Asset list - Collection of clips, frames & stills
 Preview
 Editable version of show
 Allows playback control + editing
 Timeline - Time-marks above/below path of frames
 Footage – DV segments measured in time units
 Effects - Re-size, crop, tone, color, super-imposing
 Transitions
 Straight cut (no transition)
 Fade in/out, dissolve, cross-dissolve, etc.
© D.J. Foreman 2012
119
DV Editing -1
 Import clips, frames, stills
 Trim footage (recurring action)
 Insert/remove segments
 Re-arrange segments
 Does not alter original file
 Linear edit
 Simple re-arranging, insertion, removal
 Non-linear edit (next slide)
 Note: next slide, compare ALL edits to original
© D.J. Foreman 2012
120
DV Editing - 2
 Non-linear editing
 original
 Ripple
 (middle expands 1)
a b c d 4 5 6 7
C D E
a b c d 4 5 6 7
8 C D E
F G
F G
 Rolling
 (middle expands 1)
 (following In changed)
a b c d 4
5
6 7 8 D E F G
 Slip
 (middle In/Out change)
a b c d 1
2 3 4 C D E F
G
 Slide
 (Middle moves left 2)
 (Prev Out, next In
change)
© D.J. Foreman 2012
a b 4 5 6 7
A B C D E
F G
121
DV Editing -3
 Ripple – changes overall length
 Following edits maintain overall length
 Rolling – shortens/lengthens following clip by amount
of insertion/deletion
 Slip – only the edited clip's In and Out points change
 Slide –changes prev Out, next In
© D.J. Foreman 2012
122
DV formats
 VCD
 VHS quality
 Plays anywhere
 74/80 minutes recording for 650/700MB CD's
 SVCD
 35/60 minutes
 Many standalone DVD players, all CD/DVD-ROM
 CVD = lower resolution, less noise than SVCD
 DVD
 2 hrs of "good quality" video (4.37 GB)+ Dolby/DTS audio
 Blu-ray
 High quality video
© D.J. Foreman 2012
123
DVD types
 Single/double-sided 4.37GB, 8.75GB
 DVDForum
 DVD – R
non-rewritable
 DVD – RW re-writable
 DVD+RW Alliance
 DVD + R
non-rewritable
 DVD + RW re-writeable
 DVD ± R DL
 Dual layer 15.9GB
 DVD-RAM
 NOT player compatible ≈ HDD
 + and – differences are old h/w designs
© D.J. Foreman 2012
124
DVD types -2
 Blu-ray
 25 GB per layer, up to 2 layers
 AVCHD (Panasonic, SONY)
 For Hi-DEF camcorders
 HD-DVD (MS, Toshiba, NEC, Sanyo)
 discontinued in 2008
 VHS (everyone) vs. Betamax (SONY)
© D.J. Foreman 2012
125