Motion Detector
Project 2 Documentation, Creation and Computation
Heather Simmons
Overview
This motion detector has multiple applications. I first used it (Figure 1) to project lifesized dancing figures on a screen, based on the movement of people in the physical world.
In this version, people interact with the device by dancing in front of the built-in
computer camera, and pressing the mouse as instructed on the screen. Once the mouse is
pressed, the dancer is given “feedback” on their dancing. The feedback consists of one of
three sound clips from Austin Powers’ movies, played randomly.
I then discovered code (Learning Processing, Example 16-17) that allows a hand or other
object in the physical world to control the location and colour of a ball on the screen. In
effect, the hand becomes the controller for the ball. The child interacting with this “ball”
game can hear one of 3 “bouncy ball” songs when they press the mouse. See Figure 2.
Figure 1 – Projection of girl dancing in front of motion detector, version 1
1
Figure 2 – Moving the onscreen ball by waving hand in front of motion detector,
version 2
2
Figure 3 - Moving hand to lower left moves the ball and changes ball colour to blue
Context and Related Research
Interactive motion detection games are some of the hottest on the market. In 2010,
Microsoft released the Kinect as a peripheral for the Xbox 360. It uses a sensor and
camera device to track motion of the players themselves, without the need for a
controller. Seeking to broaden the core Xbox 360 audience, Microsoft successfully
targeted families and casual gamers with the Kinect.
3
Figure 4 - Kinect targets more females, parents, casual gamers than Xbox.
Source: Interpret LLC
The Kinect soon became the world’s fastest selling consumer electronics device.
4
Figure 5 - Kinect Initial Sales Figures Outpace Similar Devices. Source: Neowin.net
According to market researcher Newzoo, the Xbox 360 will be the top selling gaming
console in the US this fall. By February 2013, Microsoft had sold 24 million Kinect units
and 76 million Xbox 360 consoles. In terms of applications, Just Dance 3, an Xbox
360/Kinect game which has players mimic on-screen dance moves and then rates their
performance, was named one of the top 10 best games for the Kinect in 2011. The
popular motion detection game market, with its multiple applications, was the inspiration
for this project.
5
Figure 6 - Number of US gamers by console type, Fall 2013. Source: Newzoo
In terms of potential, the success of the Kinect indicates that a motion detection
application would have broad audience appeal and engagement if it were set up on a big
screen in an urban or large-scale community venue (Nuit Blanche, AGO, etc.). (In fact, a
motion detection application that played sounds louder and louder as people approached
it was set up at Nuit Blanche in Paris, see below:
6
Figure 7 - Triptych motion detector installation, Nuit Blanche, Paris, 2007. Source:
http://onedotzero.com/projects/uvatriptych
It could be used to play a variety of games or trigger onscreen events. The approach in
this sketch could also be used to set off an alarm if motion is detected in a restricted area,
and capture “outline” video images of the intruder. The Capture class in Processing can
also be used to remove the background from an image, play a pre-recorded video from a
camera, or modify images (scale, pixelation, etc.).
The following books and website references were useful in understanding the market for
motion detection games, as well as how to create the motion detector, move the ball, and
play sound clips:
Capture class, Processing reference (n.d.). Retrieved from
http://processing.org/reference/libraries/video/Capture.html
Donohue, Brian (2010). Kinect hits 4 million units, outpacing Nintendo…Retrieved from
http://www.neowin.net/news/kinect-hits-4-million-units-outpacing-nintendo-in-samelaunch-period
Hong, Y. (2011). Xbox 360 and Kinect user demographics . Retrieved from
http://www.interpretllc.com/blog/2011/05/17/xbox-360s-kinect-appeals-to-a-newkind-of-gamer/
Makuch, E. (2013). Kinect sales reach 24 million. Retrieved from
http://www.gamespot.com/articles/kinect-sales-reach-24-million/1100-6403766/
7
Movie class, Processing reference (n.d.). Retrieved from
http://processing.org/reference/libraries/video/Movie.html
Qualls, E. (n.d.). Top 10 Best Kinect Games. Retrieved from
http://xbox.about.com/od/toppicks/tp/Top-5-Best-Kinect-Games.htm
Reas, C., Fry, D. (2010). Getting Started with Processing. Sebastopol, CA: O’Reilly Media,
Inc.
Shiffman, D. (2008). Learning Processing. A Beginner’s Guide to Programming Images,
Animation, and Interaction.” Burlington, MA: Morgan Kaufmann Publishers.
Smith, Catharine (2011). Microsoft Kinect: World’s Fastest Selling Consumer Electronics
Device. Retrieved from http://www.huffingtonpost.com/2011/03/09/microsoft-kinectfastest-selling-consumer-electronics_n_833706.html
Solution to inability to put multiple strings in Movie class object (2010). Retrieved from
http://www.processing.org/discourse/beta/num_1273290282.html , user PhiLho.
Takahashi, D. (2013). Xbox 360 to be the top-selling console this fall (exclusive).
Retrieved from http://venturebeat.com/2013/10/10/xbox-360-to-be-the-top-sellingconsole-this-fall-exclusive/
Triptych installation at Paris Nuit Blanche 2007, photo of (n.d.). Retrieved from
http://onedotzero.com/projects/uvatriptych
Coding Notes
These coding notes refer to version 2, the more complex motion detector which controls
the location of an onscreen ball with physical movements in the real world. (Version 1 is
just a simpler iteration of this code). The version 2 code consists of, first, a “motion
detector” which works by comparing, frame by frame, the color of each individual pixel on
the computer screen, to the color of that same pixel one frame earlier. If the color has
changed by more than a threshold amount (indicating motion), the pixel color is set to a
blue/gray colour. If not, it is set to a light gray colour. This creates the image on the
screen. Second, the average location of motion (basically, the location with the most
movement, in this case the hand waving) is calculated, and the x and y coordinates of the
ball are set to that location, thus moving the ball to that spot. Third, a “map” function sets
the red colour value of the ball to a number between 0 and 255 depending on the average
horizontal location of motion (which is also the x coordinate of the ball), and similarly
sets the blue colour value based on the y coordinate of the ball. The green colour value is
set to a static 100. In this way, the colour of the ball changes as it moves across the screen
to follow the moving hand. Fourth, a reusable Ball class was created to make it easier to
draw a ball as needed, and a reusable function, playSoundClip() was created to make it
8
easy to change the sound clips to be played based on the application. Finally, text
instructions to the user are displayed in an artsy “Nanum_Brush” font at the top and
bottom of the screen.
The motion detector code was modified from Exercise 16-13 in the book, Learning
Processing, by Daniel Shiffman. The sketch utilizes the Capture class of the Video library
in Processing. The Capture class makes it possible to grab frames of video from an
attached device (in this case, the webcam integrated in the MacBook Pro), such as a video
camera. By comparing colour in subsequent images pixel by pixel, movement can be
detected and displayed onscreen by setting the pixel colour. I then discovered the
“average location of motion” code in Exercise 16-17 of Shiffman’s book, and combined the
two to create the motion detector which allows objects in the real world to determine
location of objects on the screen. I added code to affect the colour of the ball based on
location, and added the artsy font. Finally, I created the sound clip player and the Ball
class to make changes simpler and to be able to re-use the code on subsequent projects.
Creating the sound clip player required some research. I reviewed the Movie class in
Processing, and determined that it should be able to play sound. Processing allowed me
to create a Movie array object , soundClips, but would not allow me to populate the array
with multiple movie titles (or in this case, sound clips). After further internet research, I
discovered that the constructor for the Movie class is (parent, filename). (The parent is
the applet, or “this”, and the filename is a single string). In other words, the Movie class
will not allow multiple strings, representing multiple file names, when initializing an
object of the class. So, I discovered, at
http://www.processing.org/disc/beta/num_1273290282.html, that I needed to create
another array, clipNames, to hold the 3 string filenames. Using the random function, I
was then able to pull one file at a time from clipNames, and drop it into the Movie object,
soundClips, to be played. The following two lines of code do this:
soundClips[r] = new Movie(this, clipNames[r]);
soundClips[r].play();
This worked, and I created my own function, playSoundClip(), which plays one of three
random sound clips whenever the mouse is pressed. It has a constructor and can easily
be modified to accommodate larger numbers of clips.
Starting the web cam automatically, showing the sketch full screen, and adding the text
were fairly straightforward, and were accomplished via the following lines of code:
Start webcam automatically:
video.start();//start the webcam automatically, in void setup()
Show sketch full screen:
boolean sketchFullScreen(){
return true;
9
}//show sketch full screen for projector purposes, in void draw()
Display text/font:
font = loadFont("NanumBrush-48.vlw");//in void setup()
textFont(font);//in void setup()
textSize(36);//set font size to 36//in void draw()
text ("Move the ball with your hand to change colour", 60,40);//in void draw()
Full code for version 2 is at
https://github.com/heathersimmons/Creation2/blob/master/Project 2 Motion Detector
Full code for version 1 is at
https://github.com/heathersimmons/Creation2/blob/6c2e5a0b75e4870d61b1e637403
6a48809e5d0fb/Heather Simmons Project 2 - Groovy Dance Machine
Vimeo video is at https://vimeo.com/77241139
Setup Notes
The Processing sketch automatically turns on the built-in camera in the MacBook Pro and
run at full screen when the “Run” button is pressed. The MacBook Pro was connected via
USB cable to an Epson EX5210 projector, and simply projected onto a wall in the house
for viewing. It could also be projected onto any large, blank space in an urban
environment.
Figure 8 – Initial project sketch
10
Figure 9 - Dancer projected onto wall in house
11
Figure 10 - Epson EX5210 projector connected via USB to MacBook Pro
12