Media Computation: Introducing Computing Contextualized in Video and Audio Processing Mark Guzdial and Barbara Ericson Story • Computing is important for everyone – Our track record isn’t so good. • Media Computation: Making computing relevant and creative. – In Python and Java, CS1 and Data Structures – Implementing video and audio effects: • Echoes and reversing • Background subtraction and chromakey • Connecting to other approaches – Alice + Media Computation – Robotics + Media Computation • Assessment Results 2 The challenge of 1999 • Fall 1999: All students at Georgia Tech must take a course in computer science. • Why? – Computing was a College. – The tools of learning for computational scientists and engineers brought into the classroom. – Making competitive distinctions for Liberal Arts • We have spent nearly 10 years trying to make this work! 3 Richard Dawkins on Fresh Aire, April 2007 GROSS: You close your book saying, "I am thrilled to be alive at a time when humanity is pushing against the limits of understanding." How do you think that's happening in your field of evolutionary biology? Mr. DAWKINS: Well, it's the most exciting time to be a biologist…Since Watson and Crick in 1953, biology has become a sort of branch of computer science. I mean, genes are just long computer tapes, and they use a code which is just another kind of computer code. It's quaternary rather than binary, but it's read in a sequential way just like a computer tape. It's transcribed. It's copied and pasted. All the familiar metaphors from computer science fit. 4 Only one class: Pass (A, B, or C) vs. WDF (Withdrawal, D or F) Pass 02 Fall 02 Spring 01 Fall WDF Total 74.01% 26.74% Female 62.99% 36.65% Male 77.00% 22.90% Total 65.03% 34.87% Female 65.56% 34.44% Male 64.81% 35.04% Total 70.98% 29.02% Female 59.55% 40.45% Male 73.63% 26.37% At first, only one class met the requirement 5 Contextualized Computing Education • Since Spring 2003, we now teach 3 intro CS courses. – Responding to research results about CS being “irrelevant” – Based on Margolis and Fisher “alternative paths” • Each course introduces computing using a context (examples, homework assignments, lecture discussion) relevant to majors. 6 Our Three CS1’s • CS1301 Introduction to Computing Traditional CS1 for our CS majors, Science majors (math, physics, psychology, etc.). Now: All robotics. • CS1371 Computing for Engineers CS1 for Engineers. Same topics, but using MATLAB with Engineering problems in homework and examples. • CS1315 Introduction to Media Computation 7 CS1315 Introduction to Media Computation • Average 400/term – Overall, CS1315 has been 51% female – Required in Architecture, Management, Ivan Allen College of Liberal Arts, and Biology • Focus: Learning programming and CS concepts within the context of media manipulation and creation – Converting images to grayscale and negatives, splicing and reversing sounds, writing programs to generate HTML, creating movies out of Web-accessed content. – Computing for communications, not calculation 8 Media Computation: Teaching in a Relevant Context • Presenting CS topics with media projects and examples – Iteration as creating negative and grayscale images – Indexing in a range as removing redeye – Algorithms for blending both images and sounds – Linked lists as song fragments woven to make music – Information encodings as sound visualizations 9 def clearRed(picture): for pixel in getPixels(picture): setRed(pixel,0) def greyscale(picture): for p in getPixels(picture): redness=getRed(p) greenness=getGreen(p) blueness=getBlue(p) luminance=(redness+blueness+greenness)/3 setColor(p, makeColor(luminance,luminance,luminance)) def negative(picture): for px in getPixels(picture): red=getRed(px) green=getGreen(px) blue=getBlue(px) negColor=makeColor(255-red,255-green,255-blue) setColor(px,negColor) 10 Examples of Student Work SoupAudio Collage CanonLinkedList of (MIDI) Music 11 Effects Examples from Media Computation • Simple audio effects • From simple image manipulations, to video special effects. 12 Reversing a sound def backwards(filename): source = makeSound(filename) target = makeSound(filename) Simply reversing values in an array – but when it’s a sound, it’s fun! sourceIndex = getLength(source) for targetIndex in range(1,getLength(target)+1): sourceValue = getSampleValueAt(source,sourceIndex) setSampleValueAt(target,targetIndex,sourceValue) sourceIndex = sourceIndex - 1 return target 13 Adding sounds 14 A function for adding two sounds def addSounds(sound1,sound2): for index in range(1,getLength(sound1)): s1Sample = getSampleValueAt(sound1,index) s2Sample = getSampleValueAt(sound2,index) setSampleValueAt(sound2,index,s1Sample+s2Sample) 15 Uses for adding sounds • We can mix sounds – We even know how to change the volumes of the two sounds, even over time (e.g., fading in or fading out) • We can create echoes • We can add sine (or other) waves together to create kinds of instruments/sounds that never existed in nature, but sound complex 16 A Sunset Effect • How do we turn this beach scene into a sunset? • What happens at sunset? – Theory: As the sun sets, less blue and green is visible, which makes things look more red. 17 A Sunset-generation Function def makeSunset(picture): for p in getPixels(picture): value=getBlue(p) setBlue(p,value*0.7) value=getGreen(p) setGreen(p,value*0.7) 18 SlowSunset as a movie Just one canvas repeatedly being manipulated def slowsunset(directory): canvas = makePicture(getMediaPath("beach-smaller.jpg")) #outside the loop! for frame in range(0,100): #99 frames printNow("Frame number: "+str(frame)) makeSunset(canvas) # Now, write out the frame writeFrame(frame,directory,canvas) def makeSunset(picture): for p in getPixels(picture): value=getBlue(p) setBlue(p,value*0.99) #Just value=getGreen(p) setGreen(p,value*0.99) 1% decrease! 19 SlowSunset frames 20 Example Movies 21 Fading by background subtraction • Background subtraction: def swapbg(person, bg, newbg,threshold): for x in range(1,getWidth(person)): for y in range(1,getHeight(person)): Threshold as an personPixel = getPixel(person,x,y) input. Using the bgpx = getPixel(bg,x,y) frame number personColor= getColor(personPixel) bgColor = getColor(bgpx) if distance(personColor,bgColor) < threshold: bgcolor = getColor(getPixel(newbg,x,y)) setColor(personPixel, bgcolor) 22 SlowFadeout 23 Example Movies 24 Chromakey Method public void chromakey(Picture newBg, Color color double dist) { Pixel currPixel = null; Pixel newPixel = null; // loop through the columns for (int x=0; x<getWidth(); x++) { // loop through the rows for (int y=0; y<getHeight(); y++) { 25 Chromakey Method - Cont // get the current pixel currPixel = this.getPixel(x,y); /* if the color at the current pixel is mostly blue * (blue value is greater than red and green combined), * then use the new background color */ double currDist = currPixel.colorDistance(color); if (currDist <= dist) { newPixel = newBg.getPixel(x,y); currPixel.setColor(newPixel.getColor()); } } } } 26 Code for Kids on Moon Movie public void makeKidsOnMoonMovie(String dir) { String kidsDir = FileChooser.getMediaPath("kids-blue/"); String moonF = FileChooser.getMediaPath("moon-surface.jpg"); Picture moonP = new Picture(moonF); FrameSequencer frameSequencer = new FrameSequencer(dir); Picture currP = null; // get the array of files in the directory File dirObj = new File(kidsDir); String[] fileArray = dirObj.list(); 27 Code for Kids on Moon Movie - Cont // loop through the array of files for (int i = 0; i < fileArray.length; i++) { if (fileArray[i].indexOf(".jpg") >= 0) { currP = new Picture(kidsDir + fileArray[i]); currP.chromakey(moonP,Color.black,100.0); frameSequencer.addFrame(currP); } } // play the movie frameSequencer.play(30); } 28 Example Barb, I don’t have this one 29 Using Media Computation with Other Approaches • Alice + Media Computation • Robotics + Media Computation 30 Alice + Media Computation • Barb, please put a couple slides here, plus movie? 31 A Context for CS1 for CS majors: Robotics • Microsoft Research has funded the Institute for Personal Robotics in Education – Tucker Balch, Directing Joint between Bryn Mawr and Georgia Tech – http://www.roboteducation.org • Goal is to develop a CS1 and CS2 with robotics as the context. – Added a camera and media computation abilities 32 Robot Movies • Robots have cameras, and Myro has media computation primitives. • Wonderful project: Creative and Collaborative – Robots are characters. – One robot is camera •How do you zoom? Aim and go forward! – Post-processing media computation for eerie disappearing effects. 33 Example movie 34 A Media Computation Data Structures Course • Driving question: “How did the wildebeests stampede in The Lion King?” 35 Connecting to the Wildebeests It’s all about data structures 36 Similar Assignments, but with Objects and Agents 37 Results of four years of evaluation • MediaComp students are more motivated and engaged (retention data, interviews), and find the course social, creative, and “relevant.” – Replicated at several institutions now. • Students in the contextualized courses program outside of class. – Immediately (engineers) and even a year later (MediaComp) • Students in MediaComp classes (both, and new Architecture course) spend extra time on homework “because it’s cool.” 38 Results: CS1315 “Media Computation” Pass 04 Fall 04 Spring 03 Fall WDF Total 72.55% 27.45% Female 82.90% 17.10% Male 77.46% 22.54% Total 89.87% 9.37% Female 91.94% 7.58% Male 87.50% 11.41% Total 86.47% 12.54% Female 88.36% 10.27% Male 84.71% 14.65% 39 Success Rates for Specific Majors Success rates in traditional CS1 for students in various majors average Fall ’99 to Fall ’02, compared to Spring ’03 to Fall ’05 in Media Computation. 40 “Did the class change how you interact with computers?” • Results from a survey a year later: – “Definitely makes me think of what is going on behind the scenes of such programs like Photoshop and Illustrator.” – 'I understand technological concepts more easily now; I am more willing and able to experience new things with computers now’ – 'I have learned more about the big picture behind computer science and programming. This has helped me to figure out how to use programs that I've never used before, troubleshoot problems on my own computer, use programs that I was already familiar with in a more sophisticated way, and given me more confidence to try to problem solve, explore, and fix my computer.’ 41 The Other Results • We don’t know if they learn the same. – The challenge of comparative studies when there is no common reality. • While engaging, majority of students do not find the MediaComp courses relevant to their degrees or professions. – Many do find it relevant to their lives. • Students distinguish between “more MediaComp classes” and “more CS classes” 42 Summary • Media Computation is an approach to teaching computing about image, audio, and video effects. – Teaching computing through developing the techniques behind gaming and other digital media. • Ties easily to other approaches. – Alice, Robotics…gaming? • Assessment results are promising. – Retention result is very strong 43