“Introduction to Media
Computation”
A New Core Area B Course
Mark Guzdial, 18 July 2002
Today’s story
• What’s computation good for
– Why you should care
– Examples
• A course in media computation
– Yes, still teaching programming
• How the course will be organized
• Your feedback!!!
What’s computation good
for
• Computer science is the study of recipes
• Computer scientists study…
– How the recipes are written (algorithms,
software engineering)
– The units used in the recipes (data structures,
databases)
– What can recipes be written for (systems,
intelligent systems, theory)
– How well the recipes work (human-computer
interfaces)
Specialized Recipes
• Some people specialize in crepes or
barbeque
• Computer scientists can also specialize on
special kinds of recipes
– Recipes that create pictures, sounds, movies,
animations (graphics, computer music)
• Still others look at emergent properties of
computer “recipes”
– What happens when lots of recipes talk to one
another (networking, non-linear systems)
KEY: The COMPUTER
does the recipe!
• Make it as hard, tedious, complex as you
want!
• Crank through a million genomes? No
problem!
• Find one person in a 30,000 campus? Yawn!
• Process a million dots on the screen or a
bazillion sound samples…MEDIA
COMPUTATION!
Why should you need to
study “recipes”?
• To understand better the recipe-way of
thinking
– It’s influencing everything, from computational
science to bioinformatics
– Eventually, it’s going to become part of
everyone’s notion of a liberal education
• To work with and manage computer
scientists
• AND…to communicate!
– Writers, marketers, producers communicate
through computation
Computation for
Communication
• All media are going digital
• Digital media are manipulated with
software
• You are limited in your communication by
what your software allows
– What if you want to say something that
Microsoft or Adobe or Apple doesn’t let you
say?
Programming is a
communications skill
• If you want to say something that your tools don’t
allow, program it yourself
• If you want to understand what your tools can or
cannot do, you should understand what the
programs are doing
• If you care about preparing media for the Web,
for marketing, for print, for broadcast… then it’s
worth your while to understand how the media are
manipulated.
• PROGRAMMING KNOWLEDGE = FREEDOM
“But PhotoShop is great!”
• Okay, but so are ProAudio Tools,
ImageMagick and the GIMP, and Java and
Visual Basic
• You can learn lots of tools
OR
You can learn a key set of tools, and a
general set of principles and programming
skills AND learn new tools easier
It’s still computer
science
• For example, all the “under the hood”
standard data structures are there
still:
– Sounds are arrays of samples
– Pictures are matrices (two-dimensional
arrays) of pixels (x, y, color in
red+green+blue)
– Movies are arrays of pictures!
Example: Opening a
Picture
fp=pickafile()
p=picture(fp)
show(p)
Example: A “Photoshoplike” filter
def decreaseRed(pic):
for x in pixels(pic):
setRed(x, 0.95 *
getRed(x))
Before
After
(twice)
Example: Background
Subtraction
• Let’s say that you
want to build a
picture of your
daughter on the
moon.
• Take a picture of
her against the
wall, then just the
wall.
Example Solution
• For each pixel (dot) in the picture
– Get the pixel in the picture
– Get the pixel in the frame (without the
kid)
– Are the colors pretty darn close?
• Must be where the kid ISN’T, so grab the
picture of the moon
Example code
#Picture with person, background, and
newbackground
def swapbg(pic1, bg, newbg):
for x in range(1,pic1.getWidth()):
for y in range(1,pic1.getHeight()):
p1px = getPixel(pic1,x,y)
bgpx = getPixel(bg,x,y)
if (distance(getColor(p1px),getColor(bgpx)) < 15.0):
setColor(p1px,getColor(getPixel(newbg,x,y)))
return pic1
What if we used a
different threshold?
Why doesn’t it look
better?
• Can you figure out where the
light source was?
• The colors on the shirt and the
colors of the wall were awfully
similar
New Approach:
Chromakey!
• It’s what weather
people do
• Pose in front of a
blue screen
• Swap all “blue” for
the background
Example Solution
def chromakey2(source,bg):
for p in pixels(source):
if (getRed(p)+getGreen(p) < getBlue(p)):
setColor(p,getColor(getPixel(bg,x(p),y(p))))
return source
Another way of saying
the same thing
def chromakey(source,bg):
# source should have something in front of blue, bg is the new
background
for x in range(1,source.getWidth()):
for y in range(1,source.getHeight()):
p = getPixel(source,x,y)
# My definition of blue: If the redness + greenness <
blueness
if (getRed(p) + getGreen(p) < getBlue(p)):
#Then, grab the color at the same spot from the new
background
setColor(p,getColor(getPixel(bg,x,y)))
return source
Can I do this by masking
in Photoshop?
• Of course!
– How do you think Photoshop does it?
• But you can do it better, differently,
faster, and for more kinds of
pictures if you know how it works
Example: Segmenting and
Splicing Sounds
• Recorded myself reading the first
part of the preamble
• Then used a sound editor to find the
sample number where each word
ended
Let’s keep only the first
part
•
•
•
•
•
•
•
•
•
•
#Segmenting
# Here's what I got from exploring with the editor
# Word Endpoint
# We 15730
# the 17407
# People 26726
# of 32131
# the 33413
# United 40052
# States 55510
def segmentMain():
# Grab the sound JUST up to "We the people of the United
States"
fs="/Users/guzdial/mediasources/preamble.wav"
s=sound(fs)
nfs = "/Users/guzdial/mediasources/sec1silence.wav"
#An empty sound
ns=sound(nfs) # This is where we'll build the new sound
nsi=1 # New sound index, starting from 1
for si in range(1,55510):
# Where the samples are in the sound
setSampleAt(ns,nsi, getSampleAt(s,si))
nsi = nsi + 1
play(ns)
writeSoundTo(ns,"preamble-start.wav")
Now, let’s make me say
something I didn’t say
• “We the UNITED People of the
UNITED States”
# Splicing
# Using the preamble piece, making "We the united people"
def spliceMain():
fs="/Users/guzdial/mediasources/preamble-start.wav"
s=sound(fs)
ns=sound(fs) # This is where we'll build the new sound
nsi=17408 # New Sound Index starts at just after "We
the" in the new sound
for si in range(33414,40052):
# Where the word "United" is in the sound
setSampleAt(ns,nsi, getSampleAt(s,si))
nsi = nsi + 1
for si in range(17408, 26726): # Where the word "People"
is in the sound
etSampleAt(ns,nsi, getSampleAt(s,si))
nsi = nsi + 1
for index in range(1,1000): #Stick some empty space after
that
setSampleAt(ns,nsi,0)
nsi = nsi + 1
play(ns) #Let's here and return the result
return ns
Last Example: Titling
# # Try titling
#
def testTitle():
f=
"/Users/guzdial/medias
ources/barbara.jpg"
newpic =
title(f,"Copyright 2001
Mark Guzdial")
writePictureTo(newpic,"/
Users/guzdial/mediaso
urces/t-barbara.jpg")
def title(fname,string):
p = picture(fname)
drawText(p,30,30,string)
show(p)
return p
What if you have 100’s of
files?
# # Try titling a whole directory
#
def testTitling():
import os.path
os.path.walk("/Users/guzdial/mediasources/kid-in-bgseq/",titleEach,"")
def titleEach(arg,dir,files):
for f in files:
newpic = title(dir+f,"Copyright 2001 Mark Guzdial")
writePictureTo(newpic,dir+"t-"+f)
Other Fun Examples
• “Morphing” one image into another
• Animations
• Digital video special effects, e.g.,
adding an animated figure to a video
• Creating a synthesizer
• Creating hip-hop forward-reverse
record sounds
What is this language?
Jython
• Python is an easy-to-use language that’s
aimed at Web and database programming
(http://www.python.org)
– It is something you’ll see in want ads
– Neat features: Indentation defines blocks
• Jython (http://www.jython.or) is Python
written in Java
– Anything one can do in Java, one can do in
Jython
How the class will be set
up
• Lecture and optional recitation
– 3 credit hours
• Several kinds of assignments:
– Pre-quizzes (worksheet like a quiz) given just
before a quiz (3)
– Homework, collaborative (4-6 per semester)
– Labettes (weekly, on-own activities)
– Projects: NON-collaborative homework (4 per
semester)
Not all programming
assignments
• For example, a lot of reading and
modifying code at first
• There will be a good bit of math
involved, but high-school level—and
very different from CS1321
– For example, computing a sine wave at
the right frequency to generate a given
sound
Rough Syllabus
• Weeks 1-2: Getting started with programming
– How to debug
• Weeks 3-10: Lots of experience with little/no new
CS
– Sounds, pictures, movies
– Manipulating files
– Media transformations, e.g., text (numbers) to graphics
• Weeks 11-15: Can more CS make this easier?
– Objects, functional decomposition, recursion, a little
Java
– Why do some programs take so long?
JES: Jython Environment
for Students
Planning
• Right now: Developing the materials,
technology, course notes
• Fall 2002: Planning assignments,
• First offering: Spring 2003 for 100
students
• Summer 2003: Train TAs, set up
procedures for scaling
• Full-scale: Fall 2003, two sections of
200/250/300 each
For more information…
• http://coweb.cc.gatech.edu/mediaCo
mp-plan
• guzdial@cc.gatech.edu
Feedback?
• Will this interest students in your
major?
– If not, how could it be made a better
fit?
• What do I absolutely have to do to
make this work?