Making and
Manipulating Media
for Learning
Mark Guzdial
College of Computing/GVU
Georgia Institute of Technology
Story

The Dynabook model for what a computer is for.



Learning through manipulating media.
30 years of literature suggests that it’s the right
idea.
Two examples of trying it:


Learning gravity by simulating it in elementary
school
Learning programming by manipulating media as a
context.
Dynabook to Personal
Computer
The Personal Computer* as we know it today was
invented in pursuit of the Dynabook
(*And object-oriented programming, too!)
Alan Kay’s Dynabook (1972)

Alan Kay sees the Computer as Man’s first metamedium
A medium that can represent any other media: Animation,
graphics, sound, photography, etc.
 Programming is yet another medium


The Dynabook is a (yet mythical) computer for creative metamedia
exploration and reading
Handheld, wireless network connection
 Writing (typing), drawing and painting, sound recording,
music composition and synthesis
 One goal: End-user programming.
 But WHY?

Prototype Dynabook
(Xerox PARC Learning Research Group)
A Dynabook is for Learning

The Dynabook offers a new way to learn new kinds of
things…and perhaps old things in better ways
Knowledge representation: Think about your own thinking
(Papert, 1980)
 Programming (Kay & Goldberg, 1977)


But need a system for creative expression

In a time when “windows” were made of glass, and
“mice” were undesirable rodents
Smalltalk-72

For the Dynabook,
WIMP was
invented:
overlapping
Windows
 Icons
 Menus
 mouse Pointer

What ~30 Years of Learning Sciences
tell us about the Dynabook model

Since the 1970’s, Learning Scientists (Resarchers
in Cognitive Science, Education, AI, etc.) have
studied learning



Do people learn from watching multimedia?
Do people learn from creating multimedia?
How well do people learn programming?
Do media make a difference in
learning?

Not obvious
“Equivalenced” media show little difference in learning
benefit (Clark, 1983)
 But it may be that equivalencing is the problem (Kozma,
1991)

Mixing media may make a big difference
The case of animation


Some studies show animations improve learning (Mayer,
1988)
 Most show animation has no benefit in learning (Stasko,
Badre, Lewis, 1993), and can even hinder learning
(Palmiter, Elkerton, Baggett, 1991)

Do people learn from creating
media?

Yes, and it’s a clearer case



Children building educational software for other
kids learn fractions and science (Harel, 1988; Kafai
& Harel, 1990)
Students building multimedia end up with higher
grades (Hay et al., 1994)
“Multimedia literacy” may be a component of
expertise

For example, expert chemists constantly shift
between media and representations (Kozma et al.,
1996)
How well do people learn
programming?

Really, really badly

Freshmen and Sophomores at Yale couldn’t handle error
conditions in loops in 1982 (Soloway et al.)
Freshmen and Sophomores in 3 countries and 4 institutions
can’t build calculators in 2001 (McCracken et al., 2001)
Failure/withdrawl rate in CS1 is double-digits in most
institutions



Percentage of women and minorities in CS is falling
Bottom line: Not a bad idea!

30 years of research says that the Dynabook
is a pretty good model.
 Creation
of media is a real “killer app” for
learning with computers.
 Programming is much harder to teach than
we thought,
but maybe more of a Dynabook focus would
help there, too!
Two Stories


Using Squeak to teach elementary school science
students about gravity…by simulating it!
Using a media computation context to learn
programming by non-majors.
Squeak

Smalltalk-80 running on modern machines (over 30
platforms)

http://www.squeak.org
Supporting wide range of media: Flash, MIDI, AIFF/WAV, MPEG
 Open-source


Alan Kay, Dan Ingalls, Ted Kaehler from original Xerox
PARC LRG

Apple to Disney to Viewpoints Research
http://www.squeakland.org
Breaking the Lines
Squeak Books
Example of Squeak scripting:
Building the car
Testing Gravity, by timing it
Segmenting the video, measuring
the acceleration
Duplicate the measurement
Simulate the Reality
Computer science is more
important than Calculus

In 1961, Alan Perlis argued
that computer science should
be part of a liberal education.



Explicitly, he argued that
all students should learn to
program.
Calculus is about rates, and
that’s important to many.
Computer science is about
process, which is important to
everyone
How close are we to being
able to teach everyone CS?

Not very


At many departments, CS retention rates are lower than the
rest of campus



CS1 is one of the most despised courses for non-majors
At Georgia Tech: 65% for 1995 cohort, vs. 73% for
Engineeering
Drop-out rates near 50% at many institutions
Female enrollment in CS has been dropping nationally
Why?

Several recent studies and books
claim that CS instruction tends to
dissuade anyone but white males

“Tedious,”
“taught without application
relevance,”
“boring,”
“lacking creativity,”
“asocial”
The best uses for computing
technologies will come from others

Thomas Edison vs. D.W. Griffith




Suggestion: D.W. Griffith knew things that Edison
didn’t.
If we want computing technologies to become
useful, they have to get out of our hands.
It can’t be just through applications.
Computer science will never have the
potential that it might, if future practitioners
hate our introductory course!
The Challenges


We need to motivate CS, potential CS, and non-CS
students to care about computing
We need to make it
social,
creative,
relevant,
exciting, and
not tedious

Which is how many of us already see Computing, but
that’s not getting communicated
Our Attempt: Introduction to
Media Computation

A course for non-CS and non-Engineering majors


120 students this semester,
planning 400-600 in the Fall



International Affairs, Literature, Public Policy,
Architecture, Management, Biology, etc.
2/3 female in this semester’s CS1315
Focus: Learning programming and CS concepts within the
context of media manipulation and creation
Language: Python (Jython)
Motivating the Computing
As professionals, these students will often
the use the computer as a communications
medium.
 All media are going digital,
and digital media are manipulated with
software.
 Knowing how to program, then, is a
communications skill.

def clearRed(picture):
for pixel in getPixels(picture):
setRed(pixel,0)
def greyscale(picture):
for p in getPixels(picture):
redness=getRed(p)
greenness=getGreen(p)
blueness=getBlue(p)
luminance=(redness+blueness+greenness)/3
setColor(p,
makeColor(luminance,luminance,luminance))
def negative(picture):
for px in getPixels(picture):
red=getRed(px)
green=getGreen(px)
blue=getBlue(px)
negColor=makeColor(255-red,255-green,255-blue)
setColor(px,negColor)
def chromakey(source,bg):
for x in range(1,getWidth(source)):
for y in range(1,getHeight(source)):
p = getPixel(source,x,y)
# My definition of blue: If the redness + greenness < blueness
if (getRed(p) + getGreen(p) < getBlue(p)):
#Then, grab the color at the same spot from the new background
setColor(p,getColor(getPixel(bg,x,y)))
return source
Use a loop!
Our first picture recipe
def decreaseRed(picture):
for p in getPixels(picture):
value=getRed(p)
setRed(p,value*0.5)
original
Used like this:
>>> file="/Users/guzdial/mediasources/barbara.jpg"
>>> picture=makePicture(file)
>>> show(picture)
>>> decreaseRed(picture)
>>> repaint(picture)
Recipe to Increase the Volume
def increaseVolume(sound):
for sample in getSamples(sound):
value = getSample(sample)
setSample(sample,value * 2)
Using it:
>>> f="/Users/guzdial/mediasources/gettysburg10.wav"
>>> s=makeSound(f)
>>> increaseVolume(s)
>>> play(s)
>>> writeSoundTo(s,"/Users/guzdial/mediasources/louder-g10.wav")
A Sunset-generating function


How do we turn this
beach scene into a sunset?
What happens at sunset?


Tried increasing the red,
but that failed.
New Theory: As the sun
sets, less blue and green is
visible, which makes
things look more red.
A Sunset-generation Function
def makeSunset(picture):
for p in getPixels(picture):
value=getBlue(p)
setBlue(p,value*0.7)
value=getGreen(p)
setGreen(p,value*0.7)
SlowSunset
Just one canvas
repeatedly being
manipulated
def slowsunset(directory):
canvas = makePicture(getMediaPath("beach-smaller.jpg")) #outside the loop!
for frame in range(0,100): #99 frames
printNow("Frame number: "+str(frame))
makeSunset(canvas)
# Now, write out the frame
writeFrame(frame,directory,canvas)
Not showing you
def makeSunset(picture):
for p in getPixels(picture):
value=getBlue(p)
setBlue(p,value*0.99) #Just 1% decrease!
value=getGreen(p)
setGreen(p,value*0.99)
writeFrame()
because you
know how that
works.
SlowSunset frames
Introducing IF:
Making Barb a redhead
def turnRed():
brown = makeColor(57,16,8)
file = r"C:\Documents and Settings\Mark Guzdial\My
Documents\mediasources\barbara.jpg"
picture=makePicture(file)
for px in getPixels(picture):
color = getColor(px)
if distance(color,brown)<50.0:
redness=getRed(px)*1.5
setRed(px,redness)
show(picture)
return(picture)
Original:
Generalizing Algorithms


We talk about algorithm complexity later in the
course, after the media is done.
We talk about different approaches to the same
problem, where the criteria might be aesthetics or
correctness, instead of speed or size


For example, generating greyscale
During the media, we point out similar themes in
different functions.
Scaling the picture down
def copyBarbsFaceSmaller():
# Set up the source and target pictures
barbf=getMediaPath("barbara.jpg")
barb = makePicture(barbf)
canvasf = getMediaPath("7inX95in.jpg")
canvas = makePicture(canvasf)
# Now, do the actual copying
sourceX = 45
for targetX in range(100,100+((200-45)/2)):
sourceY = 25
for targetY in range(100,100+((200-25)/2)):
color = getColor(getPixel(barb,sourceX,sourceY))
setColor(getPixel(canvas,targetX,targetY), color)
sourceY = sourceY + 2
sourceX = sourceX + 2
show(barb)
show(canvas)
return canvas
Scaling the picture up
def copyBarbsFaceLarger():
# Set up the source and target pictures
barbf=getMediaPath("barbara.jpg")
barb = makePicture(barbf)
canvasf = getMediaPath("7inX95in.jpg")
canvas = makePicture(canvasf)
# Now, do the actual copying
sourceX = 45
for targetX in range(100,100+((200-45)*2)):
sourceY = 25
for targetY in range(100,100+((200-25)*2)):
color = getColor(getPixel(barb,int(sourceX),int(sourceY)))
setColor(getPixel(canvas,targetX,targetY), color)
sourceY = sourceY + 0.5
sourceX = sourceX + 0.5
show(barb)
show(canvas)
return canvas
Recipe for halving the
frequency of a sound
def half(filename):
source = makeSound(filename)
target = makeSound(filename)
This is how a
sampling synthesizer
works!
sourceIndex = 1
for targetIndex in range(1, getLength( target)+1):
setSampleValueAt( target, targetIndex,
getSampleValueAt( source, int(sourceIndex)))
sourceIndex = sourceIndex + 0.5
Here are the
play(target)
return target
pieces that
do it
Compare these two
def copyBarbsFaceLarger():
# Set up the source and target pictures
barbf=getMediaPath("barbara.jpg")
barb = makePicture(barbf)
def half(filename):
canvasf = getMediaPath("7inX95in.jpg")
source = makeSound(filename)
canvas = makePicture(canvasf)
target = makeSound(filename)
# Now, do the actual copying
sourceX = 45
for targetX in range(100,100+((200-45)*2)):
sourceIndex = 1
for targetIndex in range(1, getLength( target)+1): sourceY = 25
for targetY in range(100,100+((200-25)*2)):
setSampleValueAt( target, targetIndex,
color = getColor(
getSampleValueAt( source,
getPixel(barb,int(sourceX),int(sourceY)))
int(sourceIndex)))
setColor(getPixel(canvas,targetX,targetY), color)
sourceIndex = sourceIndex + 0.5
sourceY = sourceY + 0.5
sourceX = sourceX + 0.5
play(target)
show(barb)
show(canvas)
return target
return canvas
Both of them are sampling

Both of them have three parts:


Set up the objects
Loop over samples or pixels and copy from one
place to another
To decrease the frequency or the size, we take each
sample/pixel twice
 In both cases, we do that by incrementing the index by
0.5 and taking the integer of the index


Finishing up and returning the result
Using your personal pictures
And messin’ with them
Data-first Computing


Real users come to a user with data that they care about,
then they (unwillingly) learn the computer to manipulate
their data as they need.
Introduction to Media Computation works the same.



We use pictures of students in class demonstrations.
Students do use their own pictures as starting points for
manipulations.
They started doing this in the second week


How often do students use concepts from the second week of CS1
on their own data?
How does that change the students’ relationship to the material?
Rough overview of Syllabus


Defining and executing functions
Pictures


Sounds



Psychophysics, data structures, defining functions,
for loops, if conditionals
Text


Psychophysics, data structures, defining functions,
for loops, if conditionals
Converting between media, generating HTML, saving
text from database and using in HTML, text searching
Web pages (e.g., for data like the temperature)
Movies
Then, Computer Science
Computer science as a solution
to their problems

“Why is PhotoShop so much faster?”



“Movie-manipulating programs take a long time
to execute. Why?”


Compiling vs. interpreting
Machine language and how the computer works
Algorithmic complexity
“Writing programs is hard! Are there ways to
make it easier? Or at least shorter?”


Functional programming and recursion
Object-oriented programming
Assignments encourage
creativity

For several homeworks, the task is to manipulate
media in some way, but we don’t care what media


For example, creating a collage or building an
animation
Encouraging homework results to be posted to
CoWeb (collaborative website) in galleries
First Homework assignment
Homework 1:
Write a program named hw1 to accept a picture as input,
and change its pixels as follows:
•
•
•
Set the green component to 125% of its current value
Decrease the blue by 25%
Decrease the red by 75%
Solutions shared in the CoWeb
Grade distribution
Much better than
anticipated.
Homework #3: Make a collage with images that
you modify by code only—any images you want
Grades on Homework #3
Assessment results so-far


Of the 120 students who started, only two
dropped the course.
97% of the students on a midterm survey
answered Yes to “Are you learning to program?”

Compared with 88% in our traditional CS1
What do you like about the class?





“I like the feeling when I finally get something to work.”
“Very applicable to everyday life.”
‘I dreaded CS, but ALL of the topics thus far have been
applicable to my future career (& personal) plans- there isn't
anything I don't like about this class!!!”
“When I finally get a program to work like I want it to.”
“The professor answers questions in class and online and is
concerned about our success in the class. He also seems to
understand that most of us are not engineers and most likely
won't do straight programming in the future- just the way of
thinking is important.”
What have you learned that you
found interesting or surprising?



“The most useful things I have learned are the
basics about computers and pictures/sound. I
think when we learn HTML- that will be
interesting and useful to real life applications.”
“Just general concepts about programming. It's
pretty logical, sort of like in math, so it's
understandable.”
“Programming is fun and ANYONE can do it!”
The Power of the Dynabook


The Dynabook has given us the desktop user
interface and object-oriented programming.
But it also offers its original purpose: A model for
using the computer for learning.


Allowing students to learn from the real world by
manipulating media from it and simulating it.
Motivating students to learn hard things in
interesting ways.
Acknowledgements




Kim Rose for the Squeakers DVD
Course materials development: Jason Ergle, Claire Bailey,
David Raines, Joshua Sklare, Adam Wilson, Andrea
Forte, Mark Richman, Matt Wallace, Alisa Bandlow.
Assessment: Andrea Forte, Rachel Fithian, Lauren Rich
Thanks to Bob McMath and the Al West Fund, to GVU
and CoC, and the National Science Foundation
For further information





Squeak: http://www.squeak.org
Viewpoints Research:
http://www.viewpointsresearch.org
Course CoWeb:
http://coweb.cc.gatech.edu/cs1315
Where we planned the course:
http://coweb.cc.gatech.edu/mediaComp-plan
guzdial@cc.gatech.edu