Input Contributions TF Crowdsourcing
(led by Tobias Hoßfeld)
Publications and Major Results:
•
Judith Redi and Isabel Povoa. Crowdsourcing for Rating Image Aesthetic
Appeal: Better a Paid or a Volunteer Crowd? 3rd International ACM workshop
on Crowdsourcing for Multimedia (CrowdMM 2014), Orlando, USA,
November 2014
•
Babak Naderi, Ina Wechsung, Tim Polzehl and Sebastian
Möller. Development and Validation of Extrinsic Motivation Scale for
Crowdsourcing Micro-task Platforms. 3rd International ACM workshop on
Crowdsourcing for Multimedia (CrowdMM 2014), Orlando, USA, November
2014
•
Tobias Hoßfeld, Matthias Hirth, Pavel Korshunov, Philippe Hanhart, Bruno
Gardlo, Christian Keimel, Christian Timmerer.Survey of Web-based
Crowdsourcing Frameworks for Subjective Quality Assessment. 16th
International Workshop on Multimedia Signal Processing, Jakarta, Indonesia,
September 2014.
•
Tobias Hoßfeld, Michael Seufert, Christian Sieber, Thomas Zinner. Assessing
Effect Sizes of Influence Factors Towards a QoE Model for HTTP Adaptive
Streaming. 6th International Workshop on Quality of Multimedia Experience
(QoMEX), Singapore, September 2014.
•
Babak Naderi, Tim Polzehl, André Beyer, Tibor Pilz, Sebastian
Möller. Crowdee: Mobile Crowdsourcing Micro-task Platform - for Celebrating
the Diversity of Languages. Proc. 15th Ann. Conf. of the Int. Speech
Communication Assoc. (Interspeech 2014). IEEE, Singapore, September
2014.
•
Simon Oechsner, Boris Bellalta, Desislava Dimitrova, Tobias Hoßfeld. Visions
and Challenges for Sensor Network Collaboration in the Cloud. The Eighth
International Conference on Innovative Mobile and Internet Services in
Ubiquitous Computing (IMIS-2014), Birmingham, UK, July 2014.
•
Isabelle Hupont, Pierre Lebreton, Toni Mäki, Evangelos Skodras, Matthias
Hirth. Is affective crowdsourcing reliable?. 5th International Conference on
Communications and Electronics (ICCE 2014), Da Nang, Vietnam, July
2014.
•
Matthias Hirth, Sven Scheuring, Tobias Hoßfeld, Christian Schwartz, Phuoc
Tran-Gia. Predicting Result Quality in Crowdsourcing Using Application Layer
Monitoring. 5th International Conference on Communications and Electronics
(ICCE 2014), Da Nang, Vietnam, July 2014.
•
Tobias Hoßfeld, Christian Timmerer. Quality of Experience Assessment using
Crowdsourcing. IEEE COMSOC MMTC R-Letter, 5, June 2014.
•
Tobias Hoßfeld. On Training the Crowd for Subjective Quality Studies. VQEG
eLetter, 1, March 2014.
•
Tobias Hossfeld, Christian Keimel, Matthias Hirth, Bruno Gardlo, Julian
Habigt, Klaus Diepold, Phuoc Tran-Gia. Best Practices for QoE Crowdtesting:
QoE Assessment with Crowdsourcing. Transactions on Multimedia, 16, 2014.
•
Tobias Hoßfeld, Christian Keimel. Crowdsourcing in QoE Evaluation in
Quality of Experience. Editor(s):Sebastian Möller, Alexander Raake, Springer,
ISBN 978-3-319-02680-0, Advanced Concepts, Applications and Methods
Series: T-Labs Series in Telecommunication Services, March 2014.
•
Valentin Burger, Matthias Hirth, Christian Schwartz, Tobias
Hoßfeld. Increasing the Coverage of Vantage Points in Distributed Active
Network Measurements by Crowdsourcing. Measurement, Modelling and
Evaluation of Computing Systems (MMB 2014), Bamberg, Germany, March
2014.
•
Bruno Gardlo, Sebastian Egger, Michael Seufert, Raimund
Schatz. Crowdsourcing 2.0: Enhancing Execution Speed and Reliability of
Web-based QoE Testing. IEEE International Conference on Communications
(ICC 2014), Sydney, Australia, June 2014.
•
Filippo Mazza, Matthieu Perreira Da Silva, Patrick Le Callet. Would you hire
me? Selfie portrait images perception in a recruitment context SPIE HVEI XIX
2014, San Francisco, CA USA, February 2014.
•
Pavel Korshunov, Hiromi Nemoto, Athanassios Skodras, and Touradj
Ebrahimi. The effect of HDR images on privacy: crowdsourcing
evaluation. SPIE Photonics Europe 2014, Optics, Photonics and Digital
Technologies for Multimedia Applications, April 2014, Brussels, Belgium.
Summary of Discussion Points
Achievements since last general meeting



Joint Qualinet Crowdsourcing experiment (JOC):
o Further results were achieved and crowdsourcing experiments
conducted.
o Crowdsourcing-based multimedia subjective evaluations: a case study
on image recognizability and aesthetic appeal
o Judith Redi and Isabel Povoa. Crowdsourcing for Rating Image
Aesthetic Appeal: Better a Paid or a Volunteer Crowd? 3rd
International ACM workshop on Crowdsourcing for Multimedia
(CrowdMM 2014), Orlando, USA, November 2014
Crowdlib activity finished by joint survey paper
o Tobias Hoßfeld, Matthias Hirth, Pavel Korshunov, Philippe Hanhart,
Bruno Gardlo, Christian Keimel, Christian Timmerer. Survey of Webbased Crowdsourcing Frameworks for Subjective Quality
Assessment. 16th International Workshop on Multimedia Signal
Processing, Jakarta, Indonesia, September 2014.
o Paper received top 10% paper awards from MMSP 2014.
Best practices for crowdsourcing are reported in a joint Qualinet
recommendation paper
o Best Practices and Recommendations for Crowdsourced QoE Lessons learned from the Qualinet WG2 Task Force Crowdsourcing"
by Tobias Hossfeld, Matthias Hirth, Judith Redi, Filippo Mazza, Pavel
Korshunov, Babak Naderi, Michael Seufert, Bruno Gardlo, Sebastian
Egger, Christian Keimel
o First draft released: October 7, 2014




o Paper will be published October 31, 2014
Output from Qualinet summer school 2013 on Crowdsourcing in Patras
o Pierre Lebreton, Evangelos Skodras, Toni Mäki, Isabelle
Hupont Torres, and Matthias Hirth. Bridging the gap between eye
tracking and crowdsourcing. Accepted for publication in: Proceedings
of the SPIE 9394, Human Vision and Electronic Imaging XX . San
Francisco, USA, February 2015
o Isabelle Hupont, Pierre Lebreton, Toni Mäki, Evangelos Skodras,
Matthias Hirth. Is affective crowdsourcing reliable?. 5th International
Conference on Communications and Electronics (ICCE 2014), Da
Nang, Vietnam, July 2014
o Pavel Korshunov, Hiromi Nemoto, Athanassios Skodras, and Touradj
Ebrahimi. The effect of HDR images on privacy: crowdsourcing
evaluation. SPIE Photonics Europe 2014, Optics, Photonics and
Digital Technologies for Multimedia Applications, April 2014, Brussels,
Belgium.
Other crowdsourcing experiments conducted
o Crowdsourcing study on real streaming experience
o HTTP adaptive streaming
o Aesthetic appeal of images
All objectives for 2014 have been fulfilled.
More details: https://www3.informatik.uniwuerzburg.de/qoewiki/qualinet:crowd:berlinmeeting2014
Planning& Objectives for 2014




The Qualinet Recommendation paper on crowdsourcing for quality
assessment will be finalized.
The JOC experiment is continued to investigate the difference between
crowdsourcing and lab studies.
Discussion on future activities and research topics, e.g.
o Predicting Result Quality in Crowdsourcing Using Application Layer
Monitoring
o eye-tracking and affective crowdsourcing
Continue the successful activities and joint work!
Action points


Finalization of Qualinet Recommendation paper
A Forum paper (in form of interview of experts)
o In a different form, own experiences and best practices will be shared.
o Paper will be initiated by Patrick le Callet and Tobias Hossfeld
Disposition of Input Documents
Document
Title
Authors
Disposition
Qi0558
Summary and major achievements of WG2
Task Force "Crowdsourcing“
Tobias Hoßfeld
Qi05xy
Comparison between lab and crowdsourcing
Ernestasia Siahaan,
uploaded to
document repository
and wiki
uploaded to
results on aesthetic appeal
Judith Redi
Qi0553
CS Taskforce Lessons learned from
Crowdsourcing,
Filippo Mazza
Qi0557
Crowdsourcing 2.x: From Microworkers to
Customers: Lessons learned from
crowdsourcing testing
Predicting Result Quality in Crowdsourcing
Using Application Layer Monitoring
Bruno Gardlo
Bridging the Gap Between Eye Tracking and
Crowdsourcing
Matthias Hirth
Qi0549
Qi0550
Matthias Hirth
document repository
and wiki
uploaded to
document repository
and wiki
uploaded to
document repository
and wiki
uploaded to
document repository
and wiki
uploaded to
document repository
and wiki
Inputs to other WG
WG
1
3
4
5
Topic
Crowdsourcing experiments providing subjective data for different
applications: HTTP adaptive streaming, web browsing, Dropbox
QoE models and influence factors on different aspects of web
browsing (aesthetics, usability, content, performance, task, element
load time, web session)
Joint Qualinet Experiment
Special session on crowdsourcing at ICCE 2014
ACM CrowdMM 2014 Workshop at ACM Multimedia
Dagstuhl Report on Crowdsourcing Seminar