AAAI Technical Report SS-12-06
Wisdom of the Crowd
CrowdREquire: A Requirements Engineering Crowdsourcing Platform
Adedamola Adepetu, Khaja Altaf Ahmed, Yousif Al Abd,
Aaesha Al Zaabi and Davor Svetinovic
Masdar Institute of Science and Technology
Abu Dhabi, United Arab Emirates
Email: {aadepetu,kahmed,yalabd,asalzaabi,dsvetinovic}@masdar.ac.ae
Abstract
corporations, to provide innovative solutions to problems. It
is important to create an atmosphere of healthy competition
among the users as this has the potential of producing excellent results from the crowd. Cash rewards act as a very
important incentive for the crowd as does the sense of prestige among peers, i.e., the CrowdREquire crowd in this case.
It is also important to note that the success of crowdsourcing has been aided by the Internet age as these companies have been structured to work with a closely networked crowd (Howe 2006). Communication is way easier
and faster than pre-Internet times, bringing the crowd together in a global village. Hence, Web 2.0 has been a fundamental factor responsible the success of the crowdsourcing
revolution.
This paper describes CrowdREquire, a platform that
supports requirements engineering using the crowdsourcing concept. The power of the crowd is in the
diversity of talents and expertise available within the
crowd and CrowdREquire specifies how requirements
engineering can harness skills available in the crowd. In
developing CrowdREquire, this paper designs a crowdsourcing business model and market strategy for crowdsourcing requirements engineering irrespective of the
professions and areas of expertise of the crowd involved. This is also a specific application of crowdsourcing which establishes the general applicability and
efficacy of crowdsourcing. The results obtained could
be used as a reference for other crowdsourcing systems
as well.
1
2
Background and Approach
There are four major areas that need to be considered while
developing CrowdREquire and these are the concept of
crowdsourcing, requirements specification, business model
and market strategy. Distributing simple tasks to individuals is not necessarily a daunting process. However, assigning complex tasks require a high level of organization and
enhancement especially with the challenge of dividing problems, assigning the sub-problems to individuals and combining the solutions (Zhang, Horvitz, and Miller 2011). The importance of the efficient division of tasks into subtasks, i.e.,
smaller tasks, getting sub-solutions to these subtasks and
systematically combining the sub-solutions to give a comprehensive solution to the original problem can be seen in
(Zhang, Horvitz, and Miller 2011). One of the motivations
of crowdsourcing is that crowdsourcing cuts out a significant percentage of capital required to solve a problem using outsourcing. Crowdsourcing is not entirely free but it is
cheaper than outsourcing or having conventional employees
(Howe 2006). This is highlighted as one of the major reasons for the success of crowdsourcing. Questions arise over
the accuracy, quality and authenticity of results of crowdsourcing. Wikipedia is an example of a crowdsourcing platform and it was shown (Goodin 2005) that it is accurate despite the fact that it is an encyclopedia developed by millions of users, i.e., a crowd. On the issue of disagreements
on accuracy of Wikipedia articles, Wales (the founder of
Wikipedia) had to settle such cases himself when Wikipedia
was still a fledgling platform. The system is however now
Introduction
This paper is aimed at developing the requirements for a
conceptualized requirements engineering platform based on
a crowdsourced workforce. Requirements engineering involves different stages (elicitation, analysis, specification
and quality assurance (van Lamsweerde 2010)) which if
properly organized, might be crowdsourced. The main idea
of CrowdREquire is to develop requirements for projects
submitted by individuals, corporations or any other external
bodies. Also CrowdREquire would be designed to be webbased in order to maximize the advantages of crowdsourcing
on the Internet.
Labor markets such as InnoCentive (Innocentive.com ),
iStockPhoto (iStockPhoto.com ) and prospectively, CrowdREquire, require specializations and specific talents. CrowdREquire is similar to Amazon Mechanical Turk but a major
difference is that the tasks on Turk are relatively simple and
can be performed by anyone while the process of developing
requirements, as is the aim of CrowdREquire, is more complex. This explains the reason why Amazon Turk has been
described as “crowdsourcing for the masses” (Howe 2006).
CrowdREquire uses a contest model (Vukovic and Bartolini
2010) in which the final solution to a task is selected based
on competition among members of the crowd. This is closely
related to InnoCentive which receives requests from external
c 2012, Association for the Advancement of Artificial
Copyright Intelligence (www.aaai.org). All rights reserved.
2
Business Model
The CrowdREquire business approach is detailed showing
the purpose, goals, stakeholders, work context model and
typical business events of CrowdREquire.
The Purpose of CrowdREquire CrowdREquire incorporates the idea of crowdsourcing with requirements engineering. In other words, the crowd submits requirement specifications as solutions to tasks submitted by clients. The
process of developing requirements is a daunting task and
requires expertise which is not always available, therefore
CrowdREquire is aimed at providing this expertise through
the crowd. Also, CrowdREquire has the potential to attract
people from different fields of interest, knowledge and experience and as a result, clients would be potentially able to
choose one out of many solutions to their defined problem.
Figure 1: Business Model Key Elements and Building Process
self-regulating (McNichol 2011).
Crowdsourcing is more than just creating a website where
people just work. There must be a structure which guides the
clients and the crowd in the desired directions, assists the
clients in describing the challenges they want solved, and
aids the crowd in executing tasks. It is also important that
a crowdsourcing platform has a sufficient crowd, must be
appealing to encourage participation, and must be relatively
easy and feasible for the crowd to use.
According to (Chanal and Caron-Fasan 2008; Osterwalder 2007), the important elements required to build and
clearly define a business model are customer segments,
value proposition, distribution channels, customers relationship, revenue streams, key resources and key activities. The
relationship between these elements is shown in Figure 1.
These elements are incorporated in the CrowdREquire business model development.
Due to the dynamic nature of the project and the relatively novel application of crowdsourcing to support requirements engineering, an agile approach for the development of requirements for CrowdREquire should be applied.
For requirements elicitation, the brainstorming method was
used (van Lamsweerde 2010). This involved multiple sessions where the aims, objectives and potential limitations
of CrowdREquire were discussed. In addition, the Volere
Specification Template (AtlanticSystems 2010) which is an
industry-standard requirements specification template was
partly used. This was done in order to improve the viability of the final requirements specifications.
The market strategy involves using appropriate strategies and corresponding methods for connecting with the
customers. Some of the factors considered in mapping out
the CrowdREquire market strategy are market segmentation strategy, targeting strategy, positioning strategy, corporate message and image, product and service strategy, pricing strategy, distribution channels, promotion and advertising strategy, sales strategy and sales forecasts.
3
CrowdREquire goals CrowdREquire is needed because
it will aid individuals and companies in finding the best
requirements specification for their proposed tasks and
projects. CrowdREquire also provides a communication tool
to connect requirement engineering professionals with the
companies that require their services. The aim of CrowdREquire is to give timely and complete response to clients
who submit a task definition.
The Stakeholders The CrowdREquire stakeholders are as
follows:
1. Client: A client is any individual, company or organization that uses the crowd to gather their requirements. Also,
the requirements process whether implicit or explicit is an
essential process in the success of projects in several fields
of practice. The range of potential clients is very broad
with respect to size, area of specialization and the nature
of projects. The client has the final say on acceptance of
the one of the submitted requirement specifications. The
client is also encouraged and assisted to divide projects
into tasks and subtasks whenever feasible.
2. Crowd: The crowd is the group of those who register on
CrowdREquire and are willing to participate by submitting a solution to any of the projects offered. The crowd
member is ultimately responsible for deciding whether to
participate in defining the requirement specification of a
project or not. If one crowd member’s requirement specifications are chosen by the client, the crowd member shall
receive a corresponding financial reward.
3. CrowdREquire Administration and Staff: The system
administration shall have the full access to CrowdREquire
and have the full authorization to monitor and maintain
CrowdREquire. In addition, the CrowdREquire administration shall be for overseeing payment processes in which
they receive the payment from the client and distribute it
to the rewarded crowd members after deducting charges.
CrowdREquire
Work Context Model The work context model, shown
in Figure 2 summarizes the activities that exist in CrowdREquire and the role of each actor in CrowdREquire.
The CrowdREquire requirements specification, business
model and market strategy are presented and explained in
this section.
3
Client
and economic factors that influence the industry and market shall be identified and monitored. Furthermore, technological breakthroughs such as advancements in Web 2.0
shall be anticipated and swiftly incorporated where possible
into CrowdREquire. Existing customer relationships shall be
maintained and improved where possible.
Crowd
CrowdREquire Requirements Specification
The CrowdREquire functional and nonfunctional requirements are shown in Table 2. These are general CrowdREquire requirements rather than specific and detailed requirements. It should be noted that ‘the system’ in Table 2
refers to CrowdREquire.
CrowdREquire
Admin/Staff
4
Evaluation
As seen in Table 2, necessary modifications were made in
order to achieve a general requirements specification set
for CrowdREquire. The organization and structure of the
CrowdREquire specification defines and describes business
use cases, presents a CrowdREquire use case diagram, explained a work context model, and specified the general
functional and non-functional requirements. The requirements were also checked for conflicts. In addition, the requirements specification was summarized for this paper as
the fit criteria and rationale for each requirement could not
be included.
With respect to the completeness and correctness of the
requirements, the CrowdREquire requirements are written
at a consistent level of detail. Clear and consistent language
is used in describing the requirements. Classes of users and
their characteristics are also defined, with specifications on
how the system is supposed to serve the users. Different
forms of product requirements are detailed, with security
and legal requirements as well. The business side and marketing requirements are also highlighted. Industry standard
methods were employed in developing the CrowdREquire
business model and marketing strategy, and the system requirements were analyzed together with the business and
market requirements.
There are certain challenges that could be faced by CrowdREquire such as Quality Assurance (QA), government and
legal issues, and business viability. These challenges are not
uncommon to crowdsourcing systems (Vukovic and Bartolini 2010). However, there are some methods that have
been explored to tackle these challenges.
The quality challenge traverses all crowdsourcing models, crowd types and sizes (Vukovic and Bartolini 2010). QA
in crowdsourced search results was researched in (Le et al.
2010), evaluating the impact of training question distribution for the crowd. The results obtained show an improvement in the relevance of search results. This use of tests is
similar to the optional client-defined tests incorporated in
the CrowdREquire task definition requirements. These tests
can ensure that each task is undertaken by qualified members of the crowd working. Also, the contest crowdsourcing
model employed by CrowdREquire encourages a competitive environment and this naturally ensures that members of
the crowd produce solutions to the best of their abilities. QA
Figure 2: CrowdREquire Context Model
Figure 3: CrowdREquire Use Case Diagram
Business Events List Table 1 shows a list of the business
events on CrowdREquire. The numbers for each Business
Use Case (BUC) match the numbers of the corresponding
CrowdREquire use cases shown in Figure 3. It also details
the input and output information for each event.
Market Strategy
CrowdREquire shall be a web-based platform and shall
therefore be conceptually open to use around the globe. The
general methods, distribution channels and industry structure for crowdsourcing shall be identified with the main
functionality of CrowdREquire being distributed problem
solving. Also the primary market research channel shall be
the Internet while secondary market research shall also be
conducted to collect important market details from resources
published by government and financial agencies. The cost of
the research shall also be taken into account.
In addition, industry analysis shall cover a comprehensive study on industry segments, current trends, market
size, growth rate, rise of other crowdsourcing-oriented businesses, etc. Potential competitors shall be identified and the
level of competition shall be established. Legal, political
4
No. Event Name
1
Client or crowd member registration
2
Client submits a task for requirements development
3
Qualified crowd member applies
for task
4
Client makes payment to CrowdREquire
5
Crowd member’s requirement specification is selected by client as winning proposal
6
Crowd member submits proposed
requirements
7
Crowd member requests further information on task details
8
Client or crowd member submits
complaints or suggestions about
the platform or about crowd member/client
Table 1: Business Events List
Input/Output
Customer details (in)
Project specification and details
(in)
Project specification and details
(out)
Secure Payment (in/out)
Winner selection (in) and secure
payment (out/in)
Requirements specification and
documentation (in)
task detail inquiry (in) and further
clarification (out)
Complaints/Suggestions (in) and
feedback from administration (out)
in crowdsourcing has also been researched by (Kern et al.
2010) and the use of decision matrices in selecting a QA approach was suggested. Based on the CrowdREquire model
and due to the non-determinacy of results, it is the responsibility of the client to select final requirements specifications,
hence, the validation review method is used by the client.
In addition, the modularization of tasks as recommended by
CrowdREquire improves the quality of results as a crowd
member could choose to focus on a particular module rather
than the system that makes up the modules (Vukovic and
Bartolini 2010).
Due to the subjective nature of gathering requirements
through crowdsourcing, the ability of the crowd to understand the basics of requirements engineering has a significant bearing on the quality of results. As a result, it is important for CrowdREquire to provide resources and information
necessary for the crowd to understand the requirements engineering process.
Potential legal issues for crowdsourcing systems include
Intellectual Property (IP) rights, copyright, employment
laws, data security, payment structures, etc. and some of
these issues have been researched by (Wolfson and Lease
2011). However, if these issues are properly handled would
they would not be obstacles to the success of CrowdREquire.
In general, (Wolfson and Lease 2011) recommends that
crowdsourcing platform providers pay close attention to the
law ensure that contracts are signed clients and crowd members, and clients should clearly define their tasks and expectations. For example, InnoCentive requires that crowd members transfer IP rights in order for them to work on crowdsourcing tasks (Vukovic and Bartolini 2010), and this approach can also be applied in CrowdREquire.
The ability of CrowdREquire to attract clients and the
crowd is of high importance as the purpose of CrowdREquire is defeated without the stakeholders. As a re-
Summary of BUCs
Users are updated on CrowdREquire
database
Requirements task is uploaded on CrowdREquire website
Requirements task and corresponding tools
are made available to the crowd member
Client has funds required for submitting
tasks on CrowdREquire
Crowd member is informed and funds
transferred to the crowd member’s account
Client receives requirements specification
and documentation
Client receives questions and provides
feedback to the crowd member
Complaints or suggestions are put into
consideration and necessary adjustments
made.
sult, a business model and marketing strategy have been
defined. Another factor that can contribute to a successful crowdsourcing business are ensuring minimal business
costs while ensuring brand quality (Vukovic and Bartolini
2010). (Lakhani and Boudreau 2009) defined different types
of business models for crowdsourcing platforms and CrowdREquire employs the two-sided platform model. This involves a crowdsourcing platform owner acting as an affiliate to both clients and the crowd while clients interact
freely with the crowd. However, the crowdsouricing platform owner still has a degree of control declaring system parameters and regulations. A high level of freedom in stakeholder interaction coupled with minimal but fundamental
platform control ensure the viability of the CrowdREquire
model.
5
Summary
While the process of developing requirements is not explicitly stated in all processes, it is essential to the success of
any endeavor. This paper studies a conceptualized crowdsourcing platform which provides requirements engineering
services through on the crowd, and systematically provides
simplified system, business model and market strategy requirements. As a result, the CrowdREquire model and requirements can be applied as a reference for new crowdsourcing systems or used to improve existing crowdsourcing
platforms.
6
Future Work
One area of further research is to execute a cost assessment
model for CrowdREquire in order to better understand its
direct financial implications. Also, a feasibility study for
CrowdREquire can be carried out. This would require a survey of all the target users of the system, both clients and the
5
Table 2: CrowdREquire Requirements Specification
No.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Description
Functional Requirements
The system shall be web-based.
The system shall accept and register users.
The system shall provide substantial information for users.
The system shall require employee-related data and business-partner-related data from crowd members and clients
respectively.
The system shall contain crowd-member-specific tabs on the crowd member profile, i.e., where the crowd member
can view details of tasks, search for tasks, upload requirements specifications, and update account information and
inquiries.
The system shall contain client-specific tabs on the client profile, i.e., task description, task management, task
submissions, task selections and inquiries.
The system shall have a messaging function.
The system shall provide users with timely updates on task suggestions and changes.
The system shall enable clients to define and customize task templates.
The system shall enable clients to define criteria and tests for selection of crowd members.
The system shall inform the client the number of crowd members that are eligible for tasks based on specified task
criteria.
The system shall shall contain task description tools such as a text editor, tables, images (upload), video(upload),
numbering and bullets, and document upload.
The system shall enable the client to modularize a task into subtasks with corresponding payments, with the system
showing the percentage of the payment that goes to CrowdREquire.
The system client shall store a history of all users’ task details.
The system shall require payments from a client before a task can be published to the crowd.
The system shall enable clients to buy CrowdREquire credits.
The system shall have an Application Programming Interface (API) for developing custom tasks.
The system shall have a ranking system for users.
The system shall have desktop and mobile applications where users can work offline.
Non-Functional Requirements
The system shall be usable on all browser and operating system platforms.
The system shall have a user-friendly and simple interface.
The system shall be readily available for usage.
The system shall be robust in order to accommodate as many users as possible at a time without the system
performance being affected.
The system shall be efficient in error handling and prevent loss of data.
The system shall be secure.
The system shall have flexible design tools.
The system shall ensure and maintain a high-level of integrity among users.
The system shall address any form of abuse or breach of legal conditions.
6
crowd, in order to observe the potential of CrowdREquire as
a requirements engineering crowdsourcing system. Furthermore, the potential impact of the success of CrowdREquire
on other task-oriented commercial crowdsourcing systems
such as Amazon Mechanical Turk and InnoCentive could be
researched.
However, the next step in line with this paper is to run
experiments on crowdsourcing requirements engineering
experiments using existing platforms such as Mechanical
Turk, Crowdflower or InnoCentive. An example experimental setup is the generation of requirements for an advanced
and modularized smart grid. In 2009, the Electric Power Research Institute (EPRI) of the United States (US) developed
a model for smart grid interoperability (Dollen 2009). In this
model, the smart grid was divided into seven domains and
further divided to sub-domains. A summary is seen in Table
3. Based on the EPRI smart grid model, a requirements specification for developing a smart grid can be developed using
the CrowdREquire concept. Each sub-domain is presented
as a module in the task and rewards are attached to each
module. The task is set such that the requirements for each
module should be defined based on smart grid applications
particularly with respect to (Dollen 2009). This experiment
can be executed on InnoCentive.com. Also, the challenges
faced and areas of improvement would be documented in
order to understand how CrowdREquire might work as a
crowdsourcing platform.
Table 3: Smart Grid Domains and Sub-Domains (Dollen
2009)
Domain
Sub-domain
Customer
Building/Home Automation, Industrial Automation, Solar Generation,
and Wind Generation.
Market
Market Management, Retailing,
Distributed Energy Resources
(DER)
Aggregation,
Trading,
Market Operations, and Ancillary
Operations.
Service Provider Customer Management, Installation and Maintenance, Building
Management, Home Management,
Billing, and Account Management.
Operations
Monitoring, Control, Fault Management, Analysis, Reporting and
Statistics, Calculations, Training,
Records and Assets, Operational
Planning, Maintenance and Construction, Extension Planning,
Customer Support, Meter Reading
and Control, Supply Chain and Logistics, Financial, Communications
Network, Security Management,
Premises,
Human
Resources,
Business Planning and Reporting,
and Stakeholder Planning and
Management.
Generation
Control, Measure, Protect, Record,
Asset Management, and Stabilization and Optimization.
Transmission
Control, Measure, Protect, Record,
Asset Management, and Stabilization and Optimization.
Distribution
Control, Measure, Protect, Record,
Asset Management, and Stabilization and Optimization.
References
AtlanticSystems. 2010. Volere requirements specification
template.
Chanal, V., and Caron-Fasan, M. L. 2008. How to invent a new business model based on crowdsourcing: The
R
crowdspirit
case. In EURAM Conference (Slovenia).
Dollen, D. V. 2009. Report to nist on the smart grid interoperability standards roadmap. Technical Report 50413,
Electric Power Research Institute (EPRI).
Goodin, D. 2005. ’nature’: Wikipedia is accurate.
Howe, J. 2006. The rise of crowdsourcing. Wired magazine.
Innocentive.com. Home-innocentive. Accessed on 19 January 2012, at https://www.innocentive.com/.
iStockPhoto.com. istock photo: Royalty free stock photography, vector art images, music & video stock footage. Accessed on 19 January 2012, at http://www.istockphoto.com/.
Kern, R.; Thies, H.; Bauer, C.; and Satzger, G. 2010. Quality
assurance for human-based electronic services: A decision
matrix for choosing the right approach. In Current Trends
in Web Engineering, volume 6385 of Lecture Notes in Computer Science. Springer Berlin / Heidelberg. 421–424.
Lakhani, K., and Boudreau, K. 2009. How to manage outside information. Technical Report 50413, Sloan School of
Management, MIT.
Le, J.; Edmonds, A.; Hester, V.; and Biewald, L. 2010. Ensuring quality in crowdsourced search relevance evaluation:
The effects of training question distribution. SIGIR 2010
workshop on crowdsourcing for search evaluation.
McNichol, T. 2011. The wales rules for web 2.0.
Osterwalder, A. 2007. How to Describe and Improve your
Business Model to Compete Better.
van Lamsweerde, A. 2010. Requirements Engineering. Wiley Press.
Vukovic, M., and Bartolini, C. 2010. Towards a research
agenda for enterprise crowdsourcing. In Margaria, T., and
Steffen, B., eds., Leveraging Applications of Formal Methods, Verification, and Validation, volume 6415 of Lecture
Notes in Computer Science. Springer Berlin / Heidelberg.
425–434.
Wolfson, S., and Lease, M. 2011. Look before you leap: Legal pitfalls of crowdsourcing. In ASIST 2011 Annual Meeting (New Orleans, LA).
Zhang, H.; Horvitz, E.; and Miller, R. 2011. Crowdsourcing
general computation.
7