Introducing Computer
Programming with
Multimedia
Barb Ericson
Institute for Computing Education
Georgia Institute of Technology
ericson@cc.gatech.edu
Computer Science Crisis
• 40% drop in majors since 2001 Nationwide
– Up to 60% drop in some places
• Percentage of women is down to about
12%
– From a high of 30% in the 1980s
• Never attracted many African Americans,
Hispanics, Native Americans
Why the Decline?
• Dot com crash in 2001
– Loss of jobs
• AP Exam switched from C++ to Java
– Many schools quit teaching AP Computer Science
• Visual Basic changed
– In .net to be object-oriented
• Perceptions of Computer Science
– boring, antisocial, not creative, and just programming
Looming Shortage
• Only a few jobs have gone overseas
– About 2-3%
• We have recovered from the dot com
crash
– There are now more jobs in this area than in
2001
• Computing jobs are some of the fastest
growing
– US Dept of Labor predicts 45.93% growth
Reversing the Trend
• We need to get more students interested
in computer science
– Not drive them away
• We need to appeal to a diverse group of
students
– Not just geeks
• All students need to have some
understanding of computer science
– Our society relies on technology
Our Solution
• Use multimedia projects to introduce
computing concepts
– Pictures, sounds, movies, text
• Use open-ended projects to encourage
creativity
– Create an image collage, create a sound
collage, create a movie
• Make computing social
– Encourage students to show off their work
Choice of Languages
• Python
– Used by many companies: IL&M, Google, Nextel, etc.
– Easier syntax than many languages
– Can use to teach functional, procedural, or objectoriented programming
– Used as a scripting or extension language for many
applications
– Growing in popularity
• Java
– Popular language used for AP CS
Digitizing Pictures
• We digitize pictures into
lots of little dots
• Enough dots and it looks
like a continuous whole to
our eye
• Each picture element is
referred to as a pixel
– Each pixel knows its color
– It also knows where it is in
its picture
Encoding Color: RGB
• In RGB, each color has
three component colors:
– Red, Green, and Blue
• In most computer-based
models of RGB, a single
byte (8 bits) is used for
each
– So a complete RGB color
is 24 bits, 8 bits of each
– 0,0,0 is black
– 255,255,255 is white
– 255,0,0 is red
Make and Show a Picture
Use pickAFile function
To select a picture file
(ends in .jpg)
Use makePicture(file)
to create the picture
Use show(picture) to
display the picture
Use MediaTools
• Click on MediaTools
• Select the picture to
explore
• Move the cursor while
holding down the left
mouse button
– To see x,y,r,g,b
• Click on zoom
– To zoom in or out
Creating a Python Function
Define
a
function
Load
the
function
Try the
function
Rules for Python Functions
• Use the keyword def
to define a function
def pickAndShow():
myFile = pickAFile()
– Followed by the
myPicture =
function name
makePicture(myFile)
• Use parentheses after
show(myPicture)
the name
– Name parameters
• End with a colon
• Indent the body of the
function
Using a Loop
def decreaseRed(picture):
for p in getPixels(picture):
value = getRed(p)
setRed(p,value * 0.5)
More Picture Functions
def greyscale(picture):
for p in getPixels(picture):
redness=getRed(p)
greenness=getGreen(p)
blueness=getBlue(p)
luminance=(redness+blueness+greenness)/3
setColor(p,
makeColor(luminance,luminance,luminance))
def negate(picture):
for px in getPixels(picture):
red=getRed(px)
green=getGreen(px)
blue=getBlue(px)
negColor=makeColor(255-red,255-green,255blue)
setColor(px,negColor)
How Sound Works:
• Sounds are waves of
air pressure
– Sound comes in
cycles
– The frequency of a
wave is the number of
cycles per second
(cps), or Hertz
– The amplitude is the
maximum height of the
wave
Volume and Pitch
• Our perception of volume is related
(logarithmically) to changes in amplitude
– If the amplitude doubles, it’s about a 3 decibel (dB)
change.
• Our perception of pitch is related
(logarithmically) to changes in frequency
– Higher frequencies are perceived as higher pitches
– We can hear between 5 Hz and 20,000 Hz (20 kHz)
– A above middle C is 440 Hz
Digitizing Sound
• We can estimate the
sound curve
– Analog-to-digital
conversion (ADC) will
give us the amplitude
at an instant as a
number: a sample
– How many samples do
we need?
• 2 x max frequency
• CD quality is 44,100
samples per second
Digitizing sound in the computer
• Each sample is stored as a number
– 16 bits, 216 = 65,536 combinations
– But we want both positive and negative
values
• To indicate compressions and rarefactions.
• Each sample can be between -32,768 and
32,767
Exploring a Sound
Use
MediaTools
to explore
a Sound
Working with Sounds
• We’ll use getSamples to get all the sample
objects out of a sound
– Use getSample to get the value from a Sample obj
• We can also get the value at any index with
getSampleValueAt
• Sounds also know their length (getLength) and
their sampling rate (getSamplingRate)
• Save a sound to a file using
writeSoundTo(sound,”file.wav”)
Decreasing the volume
def decreaseVolume(sound):
for sample in getSamples(sound):
value = getSample(sample)
setSample(sample,value * 0.5)
This works just
like decreaseRed,
but decreasing the
amplitude
(volume) instead
of the red color.
Multimedia Java CS1
• We use DrJava
– A free and easy development environment
– With an interactions pane
• We have created Java classes
– For the students to use and modify
•
•
•
•
•
World
Turtle
Picture
Sound
FileChooser
Java Syllabus
• Introduction to Java
– Math operators, printing results, data types, casting,
relational operators, Strings, variables
• Introduction to Programming
– Creating and naming objects
• Using a turtle and a world
– Creating new Turtle methods
• Draw simple shapes
• Using parameters
Syllabus - Continued
• Modifying Pictures
using Loops
– One-dimensional
arrays
– Use for-each, while,
and for loops to
– Increase/decrease
colors, fake a sunset,
lighten and darken,
create a negative, and
grayscale
Create and Show a Picture
Java Negate Method
/**
* Method to negate the picture
*/
public void negate()
{
Pixel[] pixelArray = this.getPixels();
Pixel pixelObj = null;
int redValue, blueValue, greenValue = 0;
// loop through all the pixels
for (int i = 0; i < pixelArray.length; i++)
Java Negate Method - Cont
{
// get the current pixel
pixelObj = pixelArray[i];
// get the values
redValue = pixelObj.getRed();
greenValue =
pixelObj.getGreen();
blueValue = pixelObj.getBlue();
// set the pixel's color pixelObj.setColor(
new Color(255 - redValue,
255 - greenValue,
255 - blueValue));
}
}
Testing Negate
Syllabus - Continued
• Modifying Pixels in
a Matrix
– Two-dimensional
arrays
– Nested loops
– Copying, mirroring,
blending, rotation,
scaling
Syllabus - Continued
• Conditionally
Modifying Pixels
– Replacing a color,
reducing red-eye,
edge detection,
sepia-toned,
posterize, highlight
extremes, blurring,
background
subtractions,
chromakey
Syllabus - Continued
• Drawing on Pictures
– Using existing Java
classes
• Inheritance
• Interfaces
– Drawing simple
shapes, drawing text,
general copy, general
scale, shearing,
gradient paint, general
blending, clipping
Syllabus - Continued
• Modifying all Samples
in a Sound
–
–
–
–
1D arrays
Loops
Conditional execution
Change volume,
normalizing a sound
(make it as loud as
possible), force to
extremes
Syllabus - Continued
• Modifying Samples
using Ranges
– Loops
– Clipping, splicing,
reversing, mirroring
Syllabus - Continued
• Combining and
Creating Sounds
– Class and private
methods
– Composing
sounds, blending
sounds, changing
frequencies, and
creating echoes
– MP3 and MIDI
Syllabus - Continued
• Creating Classes
– Defining a class
– Overloading constructors
– Creating and initializing an
array
– Creating getters and setters
– Creating a main method
– Javadoc comments
– Using Inheritance
Turtle ConfusedTurtle
Syllabus - Continued
• Creating and Modifying
Text
– Reading from and writing
to files
• Handling Exceptions
– Modifying programs
– Getting text from networks
– Creating random
sentences
– Using text to shift between
media
Syllabus - Continued
• Making Text for the
Web
– Throwing exceptions,
HashMap, Generics,
and Iterators
– Create a web page from
a directory
– Create a web page from
other web pages
– Creating a web page
from a database
Syllabus - Continued
• Encoding,
Manipulating, and
Creating Movies
– Frame-based
animations with simple
shapes and text
– Special effects – fade
out, fake sunset, and
chromakey
Syllabus - Continued
• Speed
– What makes programs
fast?
• Compilers and Interpreters
• Searching
• Algorithms that can’t be
written
– What makes computers
fast?
• Clock rates, Storage, Display
Syllabus - Continued
• Javascript
– Syntax
– User Interfaces
– Multimedia
Multimedia CS2 in Java
• Driving question:
“How did the
wildebeests
stampede in The
Lion King?”
CS2 Ideas
• Teach data structures using multimedia
– Stacks and queues for simulations
– Linked lists of pictures
• Order in linked list is order in scene
• Order in linked list is the layer the picture is in
• Circular linked list for animation
– Linked lists of sounds
• Weave and repeat musical phrases
– Trees of pictures
• Scene graph
– Trees of sounds
• Different traversal gives different result
Examples of Student Work
SoupAudio
Collage
CanonLinkedList of
(MIDI) Music
Results
• Python CS1 is used for non-majors at Georgia
Tech
– Business students went from 49% to 88% pass rate
– Similar results at Gainesville and U.Ill-Chicago
• Java CS1 is used in Georgia High Schools
– Both in an introductory course and in Advanced
Placement courses
– Also used at Dennison and Duke
• Java CS2 is used at Georgia Tech
– Students are transferring into computer science
Why Media Computation?
• Works on nearly all computers
– Unlike Alice which requires newer computers
• Low cost
– Unlike LEGO Mindstorms or other robots
• Students can see (and hear) if their programs work
• Computer Science concepts are introduced to solve
problems
– Looping instead of changing thousands of pixels by hand
• Open-ended assignments allow students to be creative
– Students can use their own media
• Students can publicly share their media artifacts with
others
What’s on the CD
• CS1-Python materials
–
–
–
–
Chapters from the Media Computation book in Python
Course slides
Free Jython Environment for Students (JES)
Instructor Resources
• CS1-Java materials
– Chapters from the Media Computation book in Java
– Course Slides
– Java Classes
What’s on the CD - Continued
• CS2-Java materials
– Course notes
– Course slides
– Java Classes
• DrJava
– Free Development environment for Java
• Mediasources
– Royalty free pictures and sounds
Additional Resources
• Web Site for Media Computation
– http://coweb.cc.gatech.edu/mediaComp-plan
• CS1 Python Book
– Introduction to Computing and Programming in
Python, A Multimedia Approach by Mark Guzdial,
ISBN: 0131176552
• CS1 Java Book
– Introduction to Computing and Programming with
Java, A Multimedia Approach by Mark Guzdial and
Barbara Ericson, ISBN: 0131496980
• CS2 Java Book
– Coming in 2007
Funding Sources
• National Science Foundation
• Georgia Tech's College of Computing
• Georgia’s Department of Education
• Al West Fund
• Toyota Foundation