Labeling images with a
computer game
By von Ahn & Dabbish
Center for Computational Analysis of
Social and Organizational Systems
http://www.casos.cs.cmu.edu/
Motivation
• ‘Our goal is ambitious: to label the majority of images on
the World Wide Web.’
• Machine learning to identify and label image components
is generally ineffective.
Peter Landwehr
2
Past Work
• Lempel & Soffer, 2002
• Duygulu et al., 2002
Peter Landwehr
3
Peter Landwehr
4
Implementation notes
•
•
•
•
•
•
•
•
•
Image reintroduction possibility
Age restriction possibility
Play alone possibility
Taboo threshold (X=1)
Image completion
Randomized partners
350K images from random.bounceme.net
73K word dictionary
‘[T]he game doesn’t ask the players to describe the
image: all they are told is that they have to “think like
each other” and type the same string.’
Peter Landwehr
5
August 9 – December 10, 2003
•
•
•
•
13,360 Players
1,271,451 labels for 293,760 images
80% of players returned at least once, 33 spent 50
hours playing
Mean = 3.89 labels/image/minute of play, std. dev =
0.69.
Peter Landwehr
6
Validation (1)
car
dog
man
woman
stamp
Witherspoon
smiling
Alias
cartoon
green
Peter Landwehr
7
Validation (2)
15 participants aged 20-25, 20 images with >5 labels
1. ‘Please type the six individual words that you feel best
describe the contents of this image. Type one word per
line below; words should be less than 13 character’
– For all of the images…
• at least 5 (83%) of the 6 labels produced by the game were
entered by at least one participant.
• the three most common words entered by participants were
contained among the labels produced by the game.
Peter Landwehr
8
Validation (3)
15 participants aged 20-25, 20 images with >5 labels
1. ‘How many of the words above would you use in
describing this image to someone who couldn’t see it?’
– Mean = 5.105, std. dev = 1.3087 (85% useful)
2. ‘How many of the words have nothing to do with the
image (i.e., you don't understand why they are listed
with this image)?’
– Mean = 0.105, std. dev = 0.2529 (1.7% useless)
Peter Landwehr
9
Broad implications
• ‘At this rate, 5,000 people playing the ESP game 24
hours a day would label all images on Google
(425,000,000 images) [with one tag] in 31 days.’
• ‘…[O]ur main contribution stems from the way in which
we attack the labeling problem. …[W]e have shown that
it’s conceivable that a large-scale problem can be solved
with a method that uses people playing on the Web.
We’ve turned tedious work into something people want
to do.’
Peter Landwehr
10
‘The ESP game can be used, with minor modifications, to label sound or
video clips (i.e., there is nothing inherent about images).’
Peter Landwehr
11
‘Other problems that could be solved by having people play games include
categorizing web pages into topics and monitoring security cameras.’
Peter Landwehr
12
More recent work
Walsh & Golbeck, 2010
Peter Landwehr
13
Recent Work
•
•
•
•
•
McGonigal, 2003
Law, von Ahn, Dannenberg, and Crawford, 2007
Hacker and von Ahn, 2009
Dong and Fu, 2010
Walsh and Golbeck, 2010
Peter Landwehr
14
Questions?
Center for Computational Analysis of
Social and Organizational Systems
http://www.casos.cs.cmu.edu/