Odd Leaf Out
Combining Human and Computer Vision
Arijit Biswas, Computer Science and Darcy Lewis, iSchool
Derek Hansen, Jenny Preece, Dana Rotman-University of Maryland’s iSchool
David Jacobs, Eric Stevens-University of Maryland Computer Science
Jen Hammock, Cynthia Parr-The Smithsonian Institution
Refining Metadata Associated with Images
Existing Image Crowdsourcing Games
How our game is different
• Anyone can play and can provide us with
useful information.
• No expertise necessary
• Capitalizes on strengths of humans and
algorithms
– Humans are better than algorithms at identifying
similarity of images
Game Mechanics
Game Mechanics
How Leaf Sets Are Constructed
• Designed to bring in useful data
• Not too easy or too hard
• Curvature based histograms used to get
features from leaf shapes.
– These features are used to find distance between
all possible pairs of leaves.
What’s in it for us if people play this game?
• Identify errors in the dataset
• Discover if color helps humans identify leaves
• Feedback on how enjoyable or difficult the
game is
Game Variations
Feedback Mechanism
Mul tipl e Gues s es
Ski p
Contes t a fter Ga me i s Fi ni s hed
Contes t Previ ous Round
Before Leaf is Chosen
When Feedback Occurs
After Leaf is Chosen
Mechanical Turk Trial
30
Number Correct
25
20
15
10
5
0
1
2
3
Enjoyment
4
5
Mechanical Turk Trial
5
Enjoyment
4
3
2
1
1
2
3
Difficulty
4
5
Summary
• Anyone can help in Computer Vision research
work.
• Games can be fun for players and useful for
researchers.
• Humans are better than machines in judging
the similarity of two images.
Funding
This work is made possible by National Science Foundation
grant number 0968546