Elena Simperl
Web and Internet Science, University of Southampton
11 April 2013

• "online, distributed problem- solving and production model“
[Brabham, 2008]
• Varieties: wisdom of the crowds/collective intelligence, open innovation, human computation...
• Why is it a good idea?
– Cost and efficiency savings
– Wider acceptance, closer to user needs, diversity
• Approaches
– Collaborative ontology engineering
– Challenges/competitions
– Games with a purpose
– Microtask/paid crowdsourcing
• In combination with automatic techniques
2

• Experiments using MTurk, CrowdFlower and established benchmarks
• Enhancing the results of automatic techniques
• Fast, accurate, cost-effective
[Sarasua, Simperl, Noy, ISWC2012]
PRECISION
RECALL
CartP
301-304
0.53
1.0
100R50P
Edas-Iasted
0.8
0.42
100R50P
Ekaw-Iasted
1.0
0.7
100R50P
Cmt-Ekaw
1.0
0.75
100R50P
ConfOf-Ekaw
0.93
0.65
Imp
301-304
0.73
1.0
3

• Quality assurance and evaluation
• Incentives and motivators
• Choice of crowdsourcing approach and combinations of different approaches
• Reusable collection of algorithms for quality assurance, task assignment, workflow management, results consolidation etc
• Schemas recording provenance of crowdsourced data
• Descriptive framework for classification of human computation systems
– Types of tasks and their mode of execution
– Participants and their roles
– Interaction with system and among participants
– Validation of results
– Consolidation and aggregation of inputs into complete solution