Media Computation: Introducing Computing
Contextualized in Video and Audio Processing
Mark Guzdial and Barbara Ericson
Story
• Computing is important for everyone
– Our track record isn’t so good.
• Media Computation: Making computing relevant and
creative.
– In Python and Java, CS1 and Data Structures
– Implementing video and audio effects:
• Echoes and reversing
• Background subtraction and chromakey
• Connecting to other approaches
– Alice + Media Computation
– Robotics + Media Computation
• Assessment Results
2
The challenge of 1999
• Fall 1999: All students at Georgia Tech must
take a course in computer science.
• Why?
– Computing was a College.
– The tools of learning for computational scientists
and engineers brought into the classroom.
– Making competitive distinctions for Liberal Arts
• We have spent nearly 10 years trying to make
this work!
3
Richard Dawkins on Fresh Aire, April 2007
GROSS: You close your book saying, "I am thrilled to be
alive at a time when humanity is pushing against
the limits of understanding." How do you think that's
happening in your field of evolutionary biology?
Mr. DAWKINS: Well, it's the most exciting time to be a
biologist…Since Watson and Crick in 1953, biology
has become a sort of branch of computer science. I
mean, genes are just long computer tapes, and they
use a code which is just another kind of computer
code. It's quaternary rather than binary, but it's read
in a sequential way just like a computer tape. It's
transcribed. It's copied and pasted. All the familiar
metaphors from computer science fit.
4
Only one class: Pass (A, B, or C)
vs. WDF (Withdrawal, D or F)
Pass
02 Fall
02 Spring
01 Fall
WDF
Total
74.01%
26.74%
Female
62.99%
36.65%
Male
77.00%
22.90%
Total
65.03%
34.87%
Female
65.56%
34.44%
Male
64.81%
35.04%
Total
70.98%
29.02%
Female
59.55%
40.45%
Male
73.63%
26.37%
At first, only one class met
the requirement
5
Contextualized Computing
Education
• Since Spring 2003, we now
teach 3 intro CS courses.
– Responding to research results
about CS being “irrelevant”
– Based on Margolis and Fisher
“alternative paths”
• Each course introduces
computing using a context
(examples, homework
assignments, lecture
discussion) relevant to
majors.
6
Our Three CS1’s
• CS1301 Introduction to Computing
Traditional CS1 for our CS majors, Science
majors (math, physics, psychology, etc.). Now:
All robotics.
• CS1371 Computing for Engineers
CS1 for Engineers. Same topics, but using
MATLAB with Engineering problems in
homework and examples.
• CS1315 Introduction to Media Computation
7
CS1315 Introduction to
Media Computation
• Average 400/term
– Overall, CS1315 has been 51% female
– Required in Architecture, Management, Ivan Allen College
of Liberal Arts, and Biology
• Focus: Learning programming and CS concepts within
the context of media manipulation and creation
– Converting images to grayscale and negatives, splicing and
reversing sounds, writing programs to generate HTML,
creating movies out of Web-accessed content.
– Computing for communications, not calculation
8
Media Computation:
Teaching in a Relevant Context
• Presenting CS topics with
media projects and
examples
– Iteration as creating negative
and grayscale images
– Indexing in a range as
removing redeye
– Algorithms for blending both
images and sounds
– Linked lists as song fragments
woven to make music
– Information encodings as
sound visualizations
9
def clearRed(picture):
for pixel in getPixels(picture):
setRed(pixel,0)
def greyscale(picture):
for p in getPixels(picture):
redness=getRed(p)
greenness=getGreen(p)
blueness=getBlue(p)
luminance=(redness+blueness+greenness)/3
setColor(p,
makeColor(luminance,luminance,luminance))
def negative(picture):
for px in getPixels(picture):
red=getRed(px)
green=getGreen(px)
blue=getBlue(px)
negColor=makeColor(255-red,255-green,255-blue)
setColor(px,negColor)
10
Examples of Student Work
SoupAudio
Collage
CanonLinkedList of
(MIDI) Music
11
Effects Examples from
Media Computation
• Simple audio effects
• From simple image manipulations,
to video special effects.
12
Reversing a sound
def backwards(filename):
source = makeSound(filename)
target = makeSound(filename)
Simply reversing
values in an array – but
when it’s a sound, it’s
fun!
sourceIndex = getLength(source)
for targetIndex in range(1,getLength(target)+1):
sourceValue = getSampleValueAt(source,sourceIndex)
setSampleValueAt(target,targetIndex,sourceValue)
sourceIndex = sourceIndex - 1
return target
13
Adding sounds
14
A function for adding two sounds
def addSounds(sound1,sound2):
for index in range(1,getLength(sound1)):
s1Sample = getSampleValueAt(sound1,index)
s2Sample = getSampleValueAt(sound2,index)
setSampleValueAt(sound2,index,s1Sample+s2Sample)
15
Uses for adding sounds
• We can mix sounds
– We even know how to change the volumes of the
two sounds, even over time (e.g., fading in or
fading out)
• We can create echoes
• We can add sine (or other) waves together to
create kinds of instruments/sounds that
never existed in nature, but sound complex
16
A Sunset Effect
• How do we turn this
beach scene into a
sunset?
• What happens at
sunset?
– Theory: As the sun sets,
less blue and green is
visible, which makes
things look more red.
17
A Sunset-generation Function
def makeSunset(picture):
for p in getPixels(picture):
value=getBlue(p)
setBlue(p,value*0.7)
value=getGreen(p)
setGreen(p,value*0.7)
18
SlowSunset as a movie
Just one canvas
repeatedly being
manipulated
def slowsunset(directory):
canvas = makePicture(getMediaPath("beach-smaller.jpg")) #outside the loop!
for frame in range(0,100): #99 frames
printNow("Frame number: "+str(frame))
makeSunset(canvas)
# Now, write out the frame
writeFrame(frame,directory,canvas)
def makeSunset(picture):
for p in getPixels(picture):
value=getBlue(p)
setBlue(p,value*0.99) #Just
value=getGreen(p)
setGreen(p,value*0.99)
1% decrease!
19
SlowSunset frames
20
Example Movies
21
Fading by background subtraction
• Background subtraction:
def swapbg(person, bg, newbg,threshold):
for x in range(1,getWidth(person)):
for y in range(1,getHeight(person)):
Threshold as an
personPixel = getPixel(person,x,y)
input. Using the
bgpx = getPixel(bg,x,y)
frame number
personColor= getColor(personPixel)
bgColor = getColor(bgpx)
if distance(personColor,bgColor) < threshold:
bgcolor = getColor(getPixel(newbg,x,y))
setColor(personPixel, bgcolor)
22
SlowFadeout
23
Example Movies
24
Chromakey Method
public void chromakey(Picture newBg, Color color
double dist)
{
Pixel currPixel = null;
Pixel newPixel = null;
// loop through the columns
for (int x=0; x<getWidth(); x++)
{
// loop through the rows
for (int y=0; y<getHeight(); y++)
{
25
Chromakey Method - Cont
// get the current pixel
currPixel = this.getPixel(x,y);
/* if the color at the current pixel is mostly blue
* (blue value is greater than red and green combined),
* then use the new background color
*/
double currDist = currPixel.colorDistance(color);
if (currDist <= dist)
{
newPixel = newBg.getPixel(x,y);
currPixel.setColor(newPixel.getColor());
}
}
}
}
26
Code for Kids on Moon Movie
public void makeKidsOnMoonMovie(String dir)
{
String kidsDir = FileChooser.getMediaPath("kids-blue/");
String moonF = FileChooser.getMediaPath("moon-surface.jpg");
Picture moonP = new Picture(moonF);
FrameSequencer frameSequencer = new FrameSequencer(dir);
Picture currP = null;
// get the array of files in the directory
File dirObj = new File(kidsDir);
String[] fileArray = dirObj.list();
27
Code for Kids on Moon Movie - Cont
// loop through the array of files
for (int i = 0; i < fileArray.length; i++)
{
if (fileArray[i].indexOf(".jpg") >= 0)
{
currP = new Picture(kidsDir + fileArray[i]);
currP.chromakey(moonP,Color.black,100.0);
frameSequencer.addFrame(currP);
}
}
// play the movie
frameSequencer.play(30);
}
28
Example
Barb, I don’t have this one
29
Using Media Computation with
Other Approaches
• Alice + Media Computation
• Robotics + Media Computation
30
Alice + Media Computation
• Barb, please put a couple slides here, plus
movie?
31
A Context for CS1 for
CS majors: Robotics
• Microsoft Research has funded
the Institute for Personal
Robotics in Education
– Tucker Balch, Directing
Joint between Bryn Mawr and
Georgia Tech
– http://www.roboteducation.org
• Goal is to develop a CS1 and
CS2 with robotics as the context.
– Added a camera and media
computation abilities
32
Robot Movies
• Robots have cameras, and Myro has media
computation primitives.
• Wonderful project: Creative and Collaborative
– Robots are characters.
– One robot is camera
•How do you zoom?
Aim and go forward!
– Post-processing media computation for eerie
disappearing effects.
33
Example movie
34
A Media Computation Data Structures Course
• Driving question:
“How did the
wildebeests
stampede in The
Lion King?”
35
Connecting to the Wildebeests
It’s all about data structures
36
Similar Assignments,
but with Objects and Agents
37
Results of four years of evaluation
• MediaComp students are more motivated and
engaged (retention data, interviews), and find the
course social, creative, and “relevant.”
– Replicated at several institutions now.
• Students in the contextualized courses program
outside of class.
– Immediately (engineers) and even a year later
(MediaComp)
• Students in MediaComp classes (both, and new
Architecture course) spend extra time on homework
“because it’s cool.”
38
Results: CS1315 “Media Computation”
Pass
04 Fall
04 Spring
03 Fall
WDF
Total
72.55%
27.45%
Female
82.90%
17.10%
Male
77.46%
22.54%
Total
89.87%
9.37%
Female
91.94%
7.58%
Male
87.50%
11.41%
Total
86.47%
12.54%
Female
88.36%
10.27%
Male
84.71%
14.65%
39
Success Rates for Specific Majors
Success rates in traditional CS1 for students in various majors average
Fall ’99 to Fall ’02, compared to Spring ’03 to Fall ’05 in Media
Computation.
40
“Did the class change how you
interact with computers?”
• Results from a survey a year later:
– “Definitely makes me think of what is going on behind the
scenes of such programs like Photoshop and Illustrator.”
– 'I understand technological concepts more easily now; I am
more willing and able to experience new things with computers
now’
– 'I have learned more about the big picture behind computer
science and programming. This has helped me to figure out
how to use programs that I've never used before, troubleshoot
problems on my own computer, use programs that I was
already familiar with in a more sophisticated way, and given me
more confidence to try to problem solve, explore, and fix my
computer.’
41
The Other Results
• We don’t know if they learn the same.
– The challenge of comparative studies when there
is no common reality.
• While engaging, majority of students do not
find the MediaComp courses relevant to their
degrees or professions.
– Many do find it relevant to their lives.
• Students distinguish between “more
MediaComp classes” and “more CS classes”
42
Summary
• Media Computation is an approach to
teaching computing about image, audio, and
video effects.
– Teaching computing through developing the
techniques behind gaming and other digital
media.
• Ties easily to other approaches.
– Alice, Robotics…gaming?
• Assessment results are promising.
– Retention result is very strong
43