CroRDF: Optimization for RDF Query on
Monetary Cost via Crowdsourcing
Depeng Dang1*, Jingfan Yang1† , Hu Gao1† , Tingting Fang1† ,
Nan Wang1†
1* School of Artificial Intelligence, Beijing Normal University,
Xinjiekouwai Street, Beijing, 100875, China.
*Corresponding author(s). E-mail(s): ddepeng@bnu.edu.cn;
Contributing authors: yangjingfan@mail.bnu.edu.cn;
gao h@mail.bnu.edu.cn; fangtingting@mail.bnu.edu.cn;
493543856@qq.com;
† These authors contributed equally to this work.
Abstract
The proliferation of structured data and the advances in knowledge graph have
enabled the construction of knowledge bases using the RDF data model to represent various resources and their relationships. But some RDF queries cannot
provide knowledges completely. In this paper, we present CroRDF, an inquiry
system that provides users with low cost query services based on existing and
crowdsourced RDF data. The CroRDF system is divided into two phases, the
search phase and the collection phase. If the search phase does not meet the
requirements, then it enters the collection phase. In the collection phase, we collect questions from the crowd and return the answers to the knowledge base.
We propose the best-first search algorithms that finds the CQPs with the lowest monetary cost in the search space based on the basic graph pattern (BGP).
To find the optimal CQPs, we describe the monetary cost estimation algorithm.
When looking for the optimal CQP, the money cost is calculated in turn while
searching according to the best-first path. To evaluate CroRDF, we create different queries on DBpedia dataset. The crowed use Amazon Mechanical Turk to
contribute their knowledge. Experimental results clearly show that our solution
accurately low monetary cost through crowdsourcing platforms and integrating
existing data.
Keywords: Crowdsourcing, RDF, The best-first search, Monetary cost estimation
1
1 Introduction
Since Google optimized its search services with knowledge graphs, knowledge graphs
have grown rapidly. The Resource Description Framework (RDF) is a W3C standard
that describes network resources. It is widely used to represent various resources and
their relationships in the knowledge graph. RDF is a semi-structured data model where
entities are represented as resources; connection between resources are described as
triples composed of subjects, predicates and objects Ali et al (2022).
RDF data model technologies could be especially serviceable for expressing the
knowledge of Internet. RDF clearly represents the subject, predicate and object of
a sentence in the form of triples. Moreover, SPARQL allows applications to perform
complex queries on distributed RDF databases and it is supported by various competition frameworks. Yet, SPARQL can only query the data already on the knowledge
bases. The quality of the data on the knowledge bases determines whether the results
of the query are good or bad. If the data on the knowledge bases is incomplete, it
cannot be queried.
Existing methods and technologies of acquiring RDF cannot guarantee data
integrity. Transiting from text data or XML documents and mining from the semantic web Preda et al (2010) both are inadequate since data resources are limited. Both
methods are offline approaches. Neither method can provide a complete answer to
a query immediately. Therefore, acquiring RDF data that meet the requirement for
completeness in real time remains challenging.
Recently, with the rapid development of the network, people are scrambling to make
full use of network resources. Crowdsourcing integrates the advantages of machines
and manpower to effectively solve complex problems Doan et al (2011); Huang and
Fu (2013); Wahl et al (2022); Ceschia et al (2022), such as the evaluation of search
results Noel-Storr et al (2021), tagging of pictures Saralioglu and Gungor (2022), and
filtering Yaegashi et al (2022). Data can be acquired in real time and on demand
through crowdsourcing systems. Collecting data through crowdsourcing alleviates the
problems while guaranteeing RDF semantic integrity and realizing flexible queries in
real time. So, many researchers combine man and machine to achieve the desired result
Liu et al (2022); Deng and Xiang (2021); Huang et al (2021).
Reviewing previous research in this area are about importance of collecting knowledge from the crowd to complete missing values. Examples include CrowdQ Demartini
et al (2013) and HARE Acosta et al (2018). However, these existing hybrid humanmachine approaches have so far been focused on the data. They did not minimize the
monetary cost.
The monetary cost has not been solved by previous research. To achieve this, in this
paper, we acquire RDF data by using the crowdsourcing approach. However, collecting
data through crowdsourcing is not free and can lead to monetary costs. Therefore, the
goal of the present work is to obtain RDF data through crowdsourcing with the least
monetary cost. The goal of traditional query optimization methods for a database is to
reduce CPU, I/O, and communication costs rather than monetary cost. A few studies
have examined addressing queries on structured data through crowdsourcing with a
goal of reducing monetary cost, but these works are not applicable to the RDF data
2
with semi-structured nature. In addition, no study has sought to obtain RDF data by
crowdsourcing while minimizing the monetary cost.
In this paper, we will describe CroRDF, a crowdsourcing knowledge system, to
provide a query service by combining existing data and crowdsourcing data. We will be
focusing on the crowdsourcing query optimization of the CroRDF system to minimize
the monetary cost.
Collecting the query answers from the existing data is “free”. Thus, given a query
from the end users, CroRDF first searches the answers from the existing data, which
we call the search phase. When the answers do not meet the requirements, the system
collects the remaining answers through crowdsourcing. We denote this step the collect
phase. In this phase, the system first generates a search space that contains all possible
CQPs. Then, we will detail a cost estimation algorithm to evaluate each CQP to obtain
their monetary costs. Finally, we will choose the CQP with the least monetary cost
as the final query.
In CroRDF, an ordered BGP graph can be assigned to different sets of acquisition
rules, resulting in different crowdsourcing plans and different monetary costs. Therefore, in the optimization process, we will define two reasonable evaluating scores that
are used to determine the candidate set of optimal acquisition rules. Then, we will
detail a native algorithm and an improved efficient enumeration algorithm to enumerate all CQPs in the search space. During the process, we save the number of possible
result tuples needed (PossiNum) while crowdsourcing calculation relationship between
the triples to improve the efficiency.
We attribute the cost estimation problem to the PossiNum estimation problem.
The PossiNum estimation is holistic. The PossiNum of a sub-plan depends on the
entire plan. For a triple-pattern sequence, we consider the association types between
the triple patterns and the search direction of the basic graph pattern (BGP) to
estimate the PossiNum of each triple pattern and the cost of the CQP. Finally, we
select the optimized CQP with the lowest monetary cost.
Our main contributions are as follows:
1. We propose the design of CroRDF including a two-phase executive strategy for
RDF crowdsourcing, i.e., a search phase and a collected phase. 2. We describe the
evaluation scores of acquisition rules and the enumerating algorithms. 3. We use the
best first algorithm to find the lowest cost CQP. 4. We demonstrate that our approach
has lower cost through experiments on the real dataset.
The remainder of this paper is structured as follows: Section 2 investigates the
related work. Section 3 introduces the basic architecture of the CroRDF system.
Section 4 elaborates the search phase and explains how to query in the RDF database
first. Section 5 defines the search space of the CQP in CroRDF and proposes a plan
enumeration optimization process that enumerates all the CQPs in the search space
based on the selection of the acquisition rules. Section 6 expands on the cost estimation
algorithm and discusses how CroRDF estimates the execution cost of a CQP. Section
7 presents the experimental evaluation of our system. Finally, section 8 draws some
summarizes and discusses areas we have identified for improvement in the future.
3
2 Related work
In recent years, crowdsourcing has been widely used in various fields as an efficient
and cheap problem-solving model, with demonstrated advantages in human resources
Park and Widom (2014); Malhotra et al (2020). As shown in Chittilappilly et al
(2016), we have been offered an overall picture of the current state of the art techniques in general-purpose crowdsourcing, hence we are essentially dealing with RDF
queries with crowdsourcing. It is the most promising technology to solve the integrity
issues researchers are facing in RDF query. Some data-oriented process systems have
used crowdsourcing data in a crowdsourcing explanation approach. These systems
integrate a crowdsourcing process control in the data collection process. Approaches
such as CrowDB Franklin et al (2011), Deco Parameswaran et al (2012); Chaudhuri
and Shim (1993); Park et al (2012); Balakrishnan and Bhaskar (2020), HARE Acosta
et al (2018),CoEx Deco Shaukat and Shaukat (2016) and CrowOp Fan et al (2015)
target scenarios in which existing microtask platforms are directly embedded in query
processing systems.
There are three important problems in this field: quality control, cost control and
latency control Hettiachchi et al (2022). Here, we briefly review the work on approaches
mentioned above. CrowDB Franklin et al (2011) uses human input via crowdsourcing
to process queries that neither database systems nor search engines can adequately
answer. It exploits the extensibility of the iterator-based query processing paradigm
to add crowd functionality into a DBMS. CrowDB Franklin et al (2011) supports
two types of user interfaces to allow user to input the primary key of the search. It
highlighted two cases where human input is needed: (a) unknown or incomplete data,
and (b) subjective comparisons. CrowDB Franklin et al (2011) extends SQL to address
these cases.
Similarly, Deco Parameswaran et al (2012); Chaudhuri and Shim (1993); Park
et al (2012); Balakrishnan and Bhaskar (2020) is a database system for declarative
crowdsourcing. Syntactically, Deco’s query language is a simple extension to SQL.
Based on the CrowDB Franklin et al (2011), Deco Parameswaran et al (2012); Chaudhuri and Shim (1993); Park et al (2012); Balakrishnan and Bhaskar (2020) proposes
the notions of fetch and resolution rules provide powerful mechanisms for describing
crowd access methods. Fetch rules, specifying how data in the conceptual relations
can be obtained from external sources (including humans). Resolution rules are used
to reconcile inconsistent or uncertain values obtained from external sources.
CrowOp Fan et al (2015) Supports cost-based query optimization. It is capable of
finding the query plan with low latency given a user-defined budget constraint, which
nicely balances the cost and time requirement of users. We develop efficient algorithms
in the CrowOp Fan et al (2015) for optimizing three types of queries: selection queries,
join queries, and complex selection-join queries. CoEx Deco Shaukat and Shaukat
(2016) is also a system that provides the user the facility to submit queries in the
form of comments. CoEx Deco Shaukat and Shaukat (2016) mades the user free to
comment anything about a specific noun in form of triplet and it makes a seamless
integration of user entered data along with data collected from the crowd. When a
query is evaluated the input graph of RDF is matched against the inside variables of
triple patterns.
4
HARE Acosta et al (2018) identifies parts of SPARQL queries that are affected by
incomplete portions of RDF data sets, crowdsources potential missing values and then
efficiently combines the crowd answers with the results from the data set during the
query execution. It uses a crowd knowledge base that captures crowd answers about
missing values in the RDF dataset. And it uses a microtask manager that exploits
the semantics encoded in the dataset RDF properties, to crowdsource SPARQL subqueries as microtasks and update the crowd knowledge base with the results from the
crowd.
CrowDB Franklin et al (2011) just proposes that uses microtask-based crowdsourcing to answer queries that cannot otherwise be answered. No cost control involved.
Deco Parameswaran et al (2012); Chaudhuri and Shim (1993); Park et al (2012); Balakrishnan and Bhaskar (2020) prototype does not yet perform sophisticated query
optimization. Although HARE can enhance the answer of a SPARQL query evaluation, it concentrates more on automatically identifying the completeness of a query
against RDF data and does not consider the optimization of crowdsourcing cost for
a query, which is our specific target. CoEx Deco system Shaukat and Shaukat (2016)
answer the user queries over a Simple Protocol and RDF Query Language (SPARQL)
Query on RDF together obtaining data from crowd in form of triplet, but it mainly
aims to make the SPARQL queries more expressive and does not optimize the crowdsourcing process. Our work focuses on semi-structured data–RDF, which includes the
associated relationship between the triple patterns in a SPARQL query.
3 System architecture
The architecture of query processing in CroRDF is illustrated in Fig. 1. An application
issues requests using CroSparql, a moderate extension of standard SPARQL. Users
can use the CroSparql to call CroRDF query API, so that they can get the answer
from CroRDF. CroRDF consists of two components, the Search phase and the Collect
phase.
In the Search phase, we present the flexible and extensible data model and predicate
index to store the RDF graph data. Then, we can search the results for a query with
existing RDF graph data by graph exploration. If the results of the Search phase do
not satisfy the query target, the results are sent to the Collect phase.
After the Search phase, we enter the Collect phase. CroRDF can generate crowdsourcing question according to the specific acquisition rules in each crowdsourcing
plan. Then, it loads answers from the crowdsourcing platform and uses resolution rules
to filter the answers. Finally, the crowdsourcing results are converted to RDF format
and returned to the knowledge base. CroRDF combines the crowdsourcing results and
the results of the query phase and returns them to the user. The overall framework of
the crowdsourcing query optimization and module functions is presented below, and
the two-phase query execution process the CroRDF system is briefly described.
3.1 Data model
RDF is a graph-based data model, which uses directed edges to connect different
nodes. An RDF tuple is composed of three parts: the subject, predicate, and object.
5
Fig. 1
Architecture of CroRDF
Each tuple represents a fact. The subject generally represents an information entity
(or concept) on the Web by a Universal Resource Identifier (URI). The predicate
describes the relevant properties of the entity, and the object represents the attribute
value corresponding to the subject. The formal representation is as follows Pérez et al
(2009): Given a set of URI I, a blank node set B, a literal description set L, and an
RDF tuple (s, p, o), the information represented by the tuple is as follows:
(s, p, o) ∈ (I ∪ B) × I × (I ∪ B ∪ L)
(1)
A group of RDF triple data can be regarded as a directed graph G = (V, E, L)
Wang et al (2022), where V is the node set representing the subject or object. E is the
directed edge set representing the predicate. L is the label set. L = Lv ∪ Lp , where Lv
is the label set of the nodes and Lp is the label set of the edges. We construct the RDF
graph based on the key-value storage of the data structure of the node (id, value),
where each node represents an RDF entity and is stored as a key-value pair. The
specific form is as follows:
(id, (in − adjacency − list, out − adjacency − list))
(2)
The node id is regarded as the key and the value is entity pointed by an adjacent
arrow. The lists can be divided into two categories according to the direction of the
connected edges, with the structure of (predicate, id) as the basic element. For each
node, we can search its adjacent nodes.
An example is shown in Fig. 2(a) shows the RDF graph data, where ni is the node
id and li is the predicate. Fig. 2(b) shows the key-value storage of node n0 .
There are serval others components to CroRDF’s data model:
• BGP (Basic Graph Pattern).
subject, predicate, object >.
A
6
sequence
of
triple
patterns
<
Fig. 2
Example of the key-value storage structure
• Solutions, the results of extended SPARQL query.
• Acquisition rules, specifying how data in the knowledge base can be obtained from
external sources (including humans). And it defines how to generate crowdsourcing
questions and acquire data from crowdsourcing platforms.
• Resolution rules, used to reconcile inconsistent or uncertain values obtained from
external sources.
• Crowdsourcing Query Plan (CQP), decided by the ordered BGP graphs, the acquisition rules and enumerate plans. It includes the process order and the crowdsourcing
direction. It consists of an ordered BGP graph and the corresponding physical plan.
We will illustrate each of the data model components informally in other chapters.
3.2 Query extension
We used SPARQL to complete a select query on the RDF graph data in the CroRDF.
In contrast to the commonly used join graph representation of BGPs in which each
triple pattern is an ordinary directed edge from a subject node to an object node. The
formal syntax of the BGP is expressed as Q : SELECT ?V1 ...?Vm W HERE{Q1 ...Qn },
where {Q1 ...Qn } represents a set of triple patterns and ?V1 ...?Vm represents a set of
variables that appear in {Q1 ...Qn } and defines the format of the query output.
In order to meet the needs of a query by data collection, we use extended the
SPARQL query language ‘CroSparql’ and we can use it to complete two types of query
targets by crowdsourcing platform.
• Given a threshold n of number of queries, CroRDF can return n with the least cost.
• Return the maximum number of queries within the fixed cost.
For example, the threshold n is set to 5, then CroRDF first return β exact solutions
from knowledge base in the Query parser. If β is less than 5, CroRDF will collect
solutions with crowd in Collect phase. Considering the following example.
Example 1. A user wants to find a doctor and his field of focus. In addition,
doctor is a professor and he works in a hospital. At the same time, the level of the
hospital is three. The answer can be obtained by the following SPARQL query, namely
QF and the query graph is shown in Fig. 3.
First enter the Search phase of CroRDF, according to the query target, query
requests are initialized. The graph exploration module explores the existing knowledge
in accordance with one query plan and returns the partial results. Existing data in the
7
Fig. 3 Query QF and its BGP graph
knowledge base as show in Fig. 4. If the target of β is less than 5, the query process
switches to the Collect phase. In this phase, based on the partial results obtained in
the Search phase, the TPGenerate processor generates ordered BGP graphs according
to certain rules, i.e., different execution sequences of triples. The Acquire processor
determines the crowdsourcing direction and acquisition rule set of each triple pattern
according to the scores of acquisition rule to generate candidate optimal CQPs in the
effective search space. Then, the CostEst module is utilized to estimate the crowdsourcing cost and help to find the optimal plan. Finally, the CreateQuestions and
LoadAnswer processors in the crowdsourcing module perform crowdsourcing questions
and collect the results.
Fig. 4 Existing data in the knowledge base
4 Search phase
In this phase, the SPARQL query process is transformed into a sub-graph matching
problem using graph exploration Zeng et al (2013). The process order of the triple
patterns in the SPARQL query is sorted with {q1 , ..., qn }, and the matching set of the
i − th triple qi is calculated through the whole graph. According to the matching set
of qi, qi + 1 is mapped with the graph exploration query. In an ordered set of triples,
there is an effect of the impact of the interactions between the triples, and each step
8
of the matching operations is based on the previous results to reduce the intermediate
result sets and improve the query performance.
→
Algorithm 1 illustrates the main process of the Search phase. Where q represents a
triple pattern with a direction, i.e., the crowdsourcing direction from the subject to the
object, that indicates the common nodes with another triple pattern as the subject.
And q represents the crowdsourcing direction from the object to the subject. We call
→
←
the source of q and q and call the target of them “tgt”. “p” represents predicate
and “dir” represents to correspond d relationship between “src” and “tgt”. When src
is a variable, the LoadNodes initialize the candidate set by predicate indexes; when
src is a constant, B(src) is initialized as the constant. Then, for each candidate item
in B(src), the SelectByPredicate searches for the suitable candidate set of tgt. The
result is added into R only when the “tgt” matches B(tgt).
Algorithm 1 MatchPatter
→
←
Require: Tripple patter e(e = q or q )
Ensure: The matching set R
1: Initial src, tgt, p and dir from e
2: if src is a variable then
3:
B(src) = LoadN odes(p, dir)
4: else if src is a constant then
5:
for each s in B(src) do
6:
Id ListSet = LoadN eighbors(src, dir)
7:
N = SelectByP redicate(Id ListSet, p)
8:
end for
9: end if
10: for each n in N ∩ B(src) do
11:
R = R ∪ (s, p, o)
12: end for
13: return R
In example 1, assume that the existing data in the knowledge base are as shown in
→
Fig.4. For q ?doctor WorkIn ?hospital, there will be 4 matching results in R according
to Algorithm 1: (wang3, WorkIn, Chinese Medicine Hospital), (wang1, WorkIn, Jishuitan Hospital), (wang1, WorkIn, Beiyi Hospital), (wang2, WorkIn, Beijing Hospital)
.
5 Collect phase
In this phase, based on the result set and the query target in the Search phase, the
query engine triggers the optimal acquisition rules, generates candidate crowdsourcing
plans and questions dynamically. Then, the crowdsourcing platform can handle the
crowdsourcing questions and collect new data later.
9
5.1 Generate ofrdered BGP graphs
For a SPARQL query Q, we first construct a BGP graph to describe the structural
relationship between the triple patterns. Then, all possible ordered BGP graphs of the
triple patterns that describe the process orders are determined. Based on the BGP
graphs, we construct all possible logical plans.
Definition V.1 (Logical Plan). A Logical Plan is a sequence of triple patterns
corresponding to an ordered BGP graph.
Assume the triple pattern set T P1 = {q1 , q2 , ..., qn } as the initial ordered BGP
graph that appears in the query. Where TP represents the set of triple patterns.
Based on TP, the positions of the two pairs of triples are exchanged according to the
acquirement rule position(qi ) ↔ position(qj )(i ̸= j) to form different triple pattern
sequences corresponding to different ordered BGP graphs. When there are n triples,
n triple pattern sequences are generated. The generation process of triple pattern
sequences is shown in Algorithm 2.
Algorithm 2 Enumerate BGP
Require: Initial ordered BGP graph T P1
Ensure: T P sequence set T P set
1: T P set ⇐ {T P1 }
2: for i ∈ [1, n] do
3:
for j ∈ [i, n] do
4:
for T Pi ∈ T P Set do
5:
position(qi ) ↔ position(qj )(i ̸= j)
6:
if f ilter(T Pnew ) then
7:
T P Set ⇐ T P set ∪ T Pnew
8:
end if
9:
end for
10:
end for
11: end for
12: return T P Set
5.2 Evaluate acquisition rules
5.2.1 Acquisition rules
The general form of the acquisition rule is P redicate(subject, object). There are two
specific forms when generating acquisition rules: one is P redicate(?, object), with a
known object and an unknown subject; The other is P redicate(subject, ?), with a
known subject and an unknown object. The acquisition process obtains an unknown
value according to a known value. We can set a certain reward for each acquisition
rule based on the predicate and pay workers when they complete the crowdsourcing
question generated by the acquisition rule later. We take the hospital system as an
example. Some acquisition rules are as follows:
Is(?, doctor): Ask a doctor’s name.
10
W orkT ime(N ame, ?): Ask the working time according to the name of the doctor.
A triple pattern in the WHERE clause of a SPARQL query can generate a specific set of acquisition rules. The triple pattern is formally expressed as ? var1 <
P >? var2orCON ST , where ? var1and? var2 represent variables of subject and
object. P represents predicate. The object may also be a constant. According to
the definition of the acquisition rules, we can generate the following three types of
acquisition rules: I: P (? var1, CON ST ); II: Is(? var1, V AR2), Is(V AR1, ? var2); III:
P (V AR1, ? var2), P (? var1, V AR2). V AR1 and V AR2, respectively, represent the
category where the subject and the object node belong. Different acquisition rules
can be selected under different conditions, and the data for the corresponding triple
pattern can be acquired.
5.2.2 Acquisition rules selection
Definition V.2 (Physical Plan). A Physical Plan is a sequence of acquisition
rules. It is converted from a logical plan by choosing the crowdsourcing direction for
each triple pattern in the logical plan and determining the acquisition rule for the
corresponding triple pattern.
We first compute a set of candidate acquisition rules for each triple pattern
and select the possible-complete acquisition rules (possible-complete means that data
crowdsourced according to the acquisition rules match the triple pattern completely).
The complete acquisition rules of all triple patterns are combined to produce a physical
plan.
We consider a triple pattern q: ? var1 < P >? var2orCON ST . A set of minimum
complete acquisition rules includes three types of rule sets:
A: P (? var1, CON ST )
B: {Is(V AR1, ? var2); P (? var1, V AR2)}
C: {Is(? var1, V AR2); P (V AR1, ? var2)}
Assume that the knowledge base has data, as shown in Fig. 4, we take the query
in Section 3.2 as an example. We take the following two CQPs as an example: A:
← → → → →
→ → → → →
{q2,q1,q3,q4,q5} and B: {q2,q1,q3,q4,q5} . The order of the BGP graph and the
acquisition rules are shown in Fig. 5.
Fig. 5 CQPs and acquisition rules for plans A and B
All acquisition rules conform to one of the three sets described above. Different
physical plans in the search space are formed by the combination of different complete
acquisition rules of triple patterns. Enumerating and combining all possible-complete
11
rules in the candidate acquisition rule sets of each triple pattern will result in a huge
number (i.e., o(2n ∗ n!)) of physical plans, which can affect the query efficiency. We
define two types of evaluation scores for the acquisition rules to calculate their respective contribution to the whole result and select rules with high contribution scores to
reduce the physical plans. The first score of the acquisition rule fhk is calculated as
follows:
score1 (fhk ) =
n
X
([∃j : c(i, j) = h] ×
i=1
1
)
pi
(3)
where n denotes the number of triple patterns, pi denotes the number of variables
in the i-th triple pattern, fhk denotes the k-th acquisition rule in the complete set of
the h-th triple pattern, and c(i, j) indicates whether or not fhk contributes to the i-th
triple pattern.
Ppi
The second score considers the number of acquisition rules
j=1 q(i, j). q(i, j)
refers to the number of acquisition rules required for the j-th variable. The score of
the acquisition rule fhk is calculated as follows:
!
n
X
1
score2 (fhk ) =
[∃ j : c (i, j) = h] × Ppi
(4)
j=1 q(i, j)
i=1
With the calculation and comparison of the scores of different acquisition rules,
rules with high scores are selected for each triple pattern, which will be combined to
generate the possible optimal physical plans. Cost estimation can be conducted on
these plans to determine the optimal CQP.
5.3 Candidate crowdsourcing plan
5.3.1 Enumeration algorithms
Different physical plans have different acquisition rules, and different acquisition rules
have different turns ratios, which indicates that the generated one-to-one crowdsourcing questions need different numbers of result tuples (PossiNum) to find the right
answer. The number of result tuples needed is directly related to the monetary cost
of crowdsourcing.
We now consider the problem of efficiently enumerating all CQPs in the search
space. In CroRDF, the PossiNum estimation is applied at the physical plan level to
help select the optimal CQP. Moreover, the CroRDF PossiNum estimation is holistic
and is based on an ordered triple pattern sequence in which the PossiNum of each
triple pattern partly depends on the other parts of the CQP and affects the other
triple patterns. Therefore, the goal of the enumeration algorithm is to generate a
complete CQP in the search space while maximally reusing the common triple pattern
subsequence. First, we propose a native enumeration algorithm. Then, we propose an
improved efficient enumeration algorithm based on reuse. The performance of the two
enumeration algorithms is compared in the experiment.
12
5.3.2 Native algorithm
The native enumeration algorithm iteratively generates all valid CQPs in the search
space. Algorithm 3 illustrates the whole process. First, all ordered BGP graphs (line
3) are enumerated using the EnumerateBGP algorithm in Section 5.1. For one ordered
BGP, a set of complete acquisition rules is generated and combined according to the
evaluation scores proposed in Section 5.2, which constructs a candidate CQP (lines 4
and 5). The optimal CQP is then selected by using the PossiNum estimation and cost
model (lines 6-9), where P ossEst represents possible cost and plan.poss represents
possible plan.
Algorithm 3 SearchBestPlanOriginal Procedure
1: bestP lan ⇐ N U LL
2: minCost ⇐ ∞
3: for each seqBGP do
4:
for each f etchRuleSet do
5:
plan ⇐ GenerateP lan(seqBGP, f etchRuleSet)
6:
plan.T ripleP ossEst()
7:
cost ⇐ plan.CostEst(plan.poss)
8:
if cost<minCost then
9:
bestP lan ⇐ plan
10:
end if
11:
end for
12: end for
13: return bestP lan
5.3.3 Improved algorithm
The native enumeration algorithm processes each CQP independently. In order to
reduce the number of iterations and find the best CQPs as soon as possible, we improve
the native algorithm. We use the best-first search strategy to find the least PossiNum.
The best-first search is an improvement to the breadth-first search. Its important principle is to search down the path with the least cost according to the calculation result
of the evaluation function. In the process of searching, the costly path is continuously
abandoned, so that the least costly problem is solved. The advantage of the best-first
search algorithm is that we can tune the utility function to produce a complete plan
relatively fast. We move forward in the best priority and use monetary cost estimates
to calculate the cost of each CQP. Specifically, we associate a utility measure with
each partial execution plan. We search according to the utility function of the bestfirst search. At the same time, we calculate the cost when searching and choose the
CQP with the lowest monetary cost.
The core problem of the best-first search is how to construct an evaluation function that estimates the path cost, assuming that n is a node on the best path from
the starting point trumpet to the target point. We use f (n) to represent the cost calculation for this best path. f (n) is defined as: f (n) = g(n) + h(n), where g(n) is the
13
sum of the costs from the starting point to n, and h(n) is the sum of the costs from n
to the end point. The algorithm 4 shows the best-fist search algorithm. The monetary
cost of CQP will be introduced in Section 6.
Algorithm 4 Best-first search
1: Let S be the set of BGP
2: if no sparql query (validity query) then
3:
stop
4: end if
5: while new CQP can be created do
6:
Choose p1 ∈ S maximizing the utility measure
′
7:
Let S be the set of plans that can be combined with p1
8:
for each p2 ∈ S ′ (in the order of their utility measure) do
9:
Let p3 be a (dependent) join product of p1 and p2
10:
if p3 is not viable then
11:
Ignore p3
12:
end if
13:
if ∃ p4 ∈ S s.t.p4 covers p3 and cost(p4 )<cost(p3 ) then
14:
p3 = p4
15:
end if
16:
if ∃ p5 ∈ S s.t.p5 = p3 then
17:
if cost(p3)>cost(p5) then
18:
Ignore p3
19:
else
20:
In each p6 ∈ S, using p5 replace p5 by p3 and recalculate cost of p5
21:
end if
22:
end if
23:
Let S = S ∪ p3
24:
if ∃ p7 ∈ S s.t.p3 covers p7 and cost(p3 )<cost(p7 ) then
25:
replace everywhere p7 by 0(p3 ), recalculate cost
26:
end if
27:
end for
28: end while
29: Calculate the monetary cost of each CQP
30: if S Contains the lowest monetary cost then
31:
return the optimal plan from S
32: end if
6 Monetary cost estimation
This section describes how the CroRDF system estimates the cost of a CQP. Assume
that each acquisition rule has a fixed cost that can be set by the CroRDF system.
Although cost may vary with different acquisition rules, we simplify the assumption
14
that the cost of each acquisition rule is not dependent on the specific predicates.
We convert the cost estimation into a PossiNum estimation, which is the number of
possible results of generating to satisfy the overall query target. Therefore, the cost
estimation formula is as follows:
estimation cost =
X
X
cij × fij
qi ∈T P fij ϵFi
(5)
(1 ≤ i ≤ n, 1 ≤ j ≤ mi )
where T P is the set of triple patterns in the SPARQL query, qi is a triple pattern,
Fi is a set of candidate acquisition rules generated by qi , fij is the PossiNum of the
j-th acquisition rule in Fi , and cij is the cost of the acquisition rule corresponding
to fij . To estimate the PossiNum of the triple pattern, we should fully consider the
associations and restrictions among the triple patterns.
6.1 PossiNum estimation
When executing a SPARQL query, CroRDF generates a BGP graph composed of triple
patterns. A CQP corresponds to an ordered BGP graph that indicates the order in
which the triple pattern is executed. After generating the best CQP, that is, after the
questions occurs, we need to estimate the cost of questions. Therefore, the PossiNum
estimation algorithm can be regarded as a graph exploration and traversal process
that considers the association among triple patterns. Based on the resolution rule
turns ratio and predicate density, the whole process starts from the extended query
target. It estimates the result tuples that each triple pattern needs to deliver to the
next triple pattern and computes the PossiNum of each triple pattern until the entire
BGP graph traversal is complete until returns the calculation result.
6.2 Important parameters
In the PossiNum estimation, the resolution rule turns ratio and predicate density can
be applied to estimate the PossiNum.
6.2.1 Resolution rules
The quality of the crowd is uneven, leading to many answers to the same question. In
order to ensure the consistency of the answer, we propose resolution rules. Resolution
rules are applied to eliminate the ambiguity and inconsistency of crowdsourcing result
triples. The results are returned to the knowledge base. The form of the resolution rule
is Rule(S → O, predicate), where S and O represent the subject and the object (S can
be empty). Predicate is the predicate involved in the rule. The number of inputs can be
used for the query cost estimation. The resolution rules involved in the query process
include distinct, majority, average, etc. If the answers of the two workers are the same,
we only choose one. If each worker’s answer is different, we choose the answer of most
people. If each worker’s answer is not unique, we can take the average as the answer.
In the example of the hospital system, there may be some resolution rules as follows:
15
Distinct(∅ → hospital, Is): Take different values.
Average − 3(doctor → score, Has rate): Calculate the average of three scores.
M ajority − 3(doctor → hospital, W orkIn): Take most items of the three results.
6.2.2 Resolution rule turns ratio
The resolution rule turns ratio can estimate the average number of output tuples for
each input tuple. For example, the resolution rule Average-n represents the average
value of n values and the turns ratio is 1/n; Majority-3 represents the majority of 3
results and the turn ratio is between 1/3 and 1/2, when the two results are inconsistent,
the turns ratio is 1/2. If all three results are the same, the turns ratio is 1/3.
6.2.3 Predicate density
The predicate density in an acquisition rule implies a probability value that owns
the predicate for all possible RDF resources. The predicate density is related
to the predicate category, such as for the acquisition rules Is(?, doctor) and
W orkIn(?doctor, \BeiyiHospital”), whose possible predicate densities are 1 and 0.1,
respectively.
6.3 Calculate the PossiNum
First, we define four types of relationships between triple patterns, as shown in Table
1. The crowdsourcing process for each triple pattern has a direction, which refers to the
direction between the source and target, represented by src and tgt, respectively. The
source and target differ from the subject and object. The right arrow ‘→’ represents
the matching direction from subject to object, whereas the left arrow ‘ ←′ indicates
←
from object to subject. For example, for q2, src represents the source is the object of
→
the triple, whereas for q2, src indicates the subject of the triple.
Table 1 Relationships between the triple patterns
TP Relationship
Description (example in Fig. 3)
R1: src − src
q1 and q3
R2: tgt − src
q3 and q4
R3: src − tgt
→
→
→
→
→
←
q1 and q2
←
R4: tgt-tgt
←
q1 and q3
Now, we explain the TriplePossEst PossiNum estimation algorithm in terms of the
four types of relationships between triple patterns. The basic process unit of the algorithm is a single triple pattern. In the implementation process, two input parameters
are involved:
target: The number of target tuples to be output for one triple pattern.
binding: The candidate set of association values between the triples.
16
According to the input parameters and a CQP, the TriplePossEst algorithm estimates the PossiNum for a specific triple pattern, and the output is passed as the
target input of the next triple pattern. Then, the total estimated cost of all tuples
is calculated cumulatively. Four local variables are referenced in each triple pattern
estimation:
f ets: The acquisition rule set of a triple pattern.
preds: The predicate set with the density of the involved triple pattern.
res sel: The resolution rule set and its turns-ratio. poss: The PossiNum of the
current triple pattern.
Algorithm 5 illustrates the basic process of the TriplePossEst algorithm. The input
is the CQP, including the process order and crowdsourcing direction of TP. The output
is the estimated PossiNum of CQP, which is the number of possible result tuples
needed for all acquisition rules in the CQP.
Algorithm 5 TriplePossEst
Require: Crowdsourcing Query Plan (CQP)
Ensure: Estimation PossiNum (EstPoss)
1: target ⇐ n − N or 1
2: binding ⇐ GraphExplore(DataBase)
3: for tpi in T P do
4:
T ripleP ossEstCore(target, binding, tpi )
5:
target ⇐ poss
6:
associate attribute ⇐ Relation(tpi , tpi + 1)
7:
binding ⇐ binding(associate attribute)
8:
GraphExplore(DataBase)
9:
EstP oss ⇐ EstP oss + poss
10: end for
11: return bestP lan
First, according to the query result in the Search phase, the algorithm initializes
the parameters target and binding (lines 1 and 2). For MinTuples n, the parameter
target is initialized as the number of result tuples required to satisfy the query target
n; for MaxCost c, target is initialized to 1. Then, the algorithm calls the TriplePossEstCore algorithm to calculate the PossiNum of each triple pattern, updates the input
parameters of the next triple pattern, and sums the PossiNum estimation cumulatively (lines 4-7). Algorithm 6 illustrates the TriplePossEstCore algorithm procedure,
which aims to estimate the PossiNum of the current triple pattern. The inputs are
the current triple pattern tp, the number of results to be output, and the value set
associated with the previous tp. If the current tp is the first one in the CQP, the
binding set is initialized by the TriplePossEst algorithm. The output is the PossiNum
of the current tp. The initialization is processed in lines 1-5 to obtain the following
information about tp: f ets, preds, and res sel. The fets set determines the association
type between the current tp and the previous tp and initializes the PossiNum. Then,
according to the association types, three cases are handled separately. The first case
17
(lines 6-14) is applied to the R1 and R2 association types. In this case, the binding
set limits the range of the src of tp. Therefore, when the acquisition rules in the fets
set are crowdsourced in a certain order, it is unnecessary to crowdsource the variable
node values that match the binding set to collect new data. In terms of the matching
type between the acquisition rules and the candidate set of the association values of
tp, the algorithm estimates the number of other possible crowdsourcing.
The second case (lines 15-23) aims to handle the R3 and R4 association types. In
this case, the binding set limits the range of the tgt of tp. Similarly, the PossiNum
of each acquisition rule is calculated in terms of different matching types. When the
acquisition rule obtains the unassociated values before knowing the associated values,
the algorithm must re-calculate the number of acquisition rules required to obtain the
associated values based on the estimated cost and the binding set (lines 13 and 14 and
lines 19 and 20). The third case occurs when tp is the first triple pattern in the CQP
or when there is no association between the two tps. The PossiNum of acquisition
rules can be estimated directly based on the density of predicates and the turns-ratio
of resolution rules (lines 24-27). Finally, the PossiNum values of all acquisition rules
are combined as the output result.
For the target MinTuples n, the parameter target is assigned to the number of
results still to be crowdsourced, considering the partial query results generated by the
existing knowledge. For the target MaxCost c, the principle of the algorithm is to
return as many results as possible within the range of cost c, based on returning at
least n query results (n is the system default). Therefore, we complete the PossiNum
estimation process in three steps. First, it sets the parameter target to the number of
partial result tuples from the Search phase and calculates the cost used to return the
missing values in the partial result tuples. If the number limit is satisfied or the budget
has been exceeded, the process returns the results directly and is ended; otherwise, it
proceeds to the next step. Second, the process sets the target to 1 to calculate the cost
for returning one result tuple. Third, according to the cost c, it repeats the calculation
until the budget is exhausted and then returns the number of tuples in the result.
6.4 This is an example for second level head—subsection head
We illustrate the PossiNum estimation algorithm in Section 6.3 with two simple examples. Taking the query in Section 3.2. We assume that the predicates
P ositionalT itle =′ P ROF ESSOR′ and Has level = 3 have a density of 0.2 and
that the other predicates have a density of 1. The turns ratios of the resolution rules
Distinct, Average − 3, and M ajority − 3 are 1.0, 0.3, and 0.4, respectively. The cost
of each acquisition rule is assumed to be $0.05. The resolution rules involved in CQPs
A and B are:
Distinct(∅ → doctor, Is),
Average − 3(doctor → score, Has rate),
M ajority − 3(doctor → hospital, W orkIn),
M ajority − 3(hospital → f ield, M ajorIn),
and M ajority − 3(hospital → level, Has level).
Fig. 6 shows the PossiNum estimation process of plan A. First, we
consider the impact of the existing data. In our case, the partial query
18
Algorithm 6 TriplePossEstCore
Require: target, binding, tp
Ensure: EstimationP ossiN um(EstP oss)
1: f ets, preds, res sel ⇐ Initialize(tp)
2: r type ⇐ Relationship(tp) with the previous tp
3: poss ⇐ target − |binding.existingpartialdata(tp)|
4: posss ⇐ poss, poss, . . . , poss
5: f 1, . . . , f n ⇐ sort(f ets)
6: if r type = R1 or R2 then
7:
tp.src ⇐ binding
8:
for f i in f ets do
9:
if M apping(f i.src, tp.src) then
10:
for pred in preds ∪ res sel(f i) do
11:
posss[i] ⇐ posss[i]/pred.density posss[i] ⇐ posss[i]/c
12:
end for
13:
else if M apping(f i.tgt, tp.src) then
14:
c ⇐ (1 − |tp.src|/posss[i]) ∗ tp(associate attributeassrc).preds.density
15:
posss[i] < −posss[i]/c
16:
else if rt ype = R3 or R4 then
17:
tp.tgt ⇐ binding
18:
for f i in f ets do
19:
if M apping(f i.tgt, tp.tgt) then
20:
c ⇐ (1 − |tp.tgt|/posss[i]) ∗ tp(associate attribute as
tgt).preds.density
21:
posss[i] ⇐ posss[i]/c
22:
else if N oM apping(f i, tp.tgt) then
23:
for pred in res sel(f i) do
24:
posss[i] ⇐ posss[i]/pred.density
25:
end for
26:
else if r type = N U LL then
27:
for f i in f ets do
28:
for pred in preds ∪ sel(f i) do
29:
posss[i] ⇐ posss[i]/pred.density
30:
end for
31:
end for
32:
end if
33:
end for
34:
end if
35:
end for
36: end if
37: return poss ⇐ sum(posss)
19
results
are
the
tuples
of{wang1, JishuitanHospital, orthopedics, 8}and
{wang1, BeiyiHospital, ?, 9}. Therefore, the target parameter is initialized with
4, and the binding set is {wang1} (the first processed triple pattern is q2). Then,
T ripleP ossEstCore(4, {wang1}) is called to process q2. For the acquisition rule,
P ositionalT itle(?doctor, P ROF ESSOR), all results satisfy the predicate, and
q2.poss = 4 − 1 = 3. Then,T ripleP ossEstCore(3, {wang1, 9} ∪ binding(doctor))is
called to process q1. Since the predicate density is 1, the resolution rule
Average − 3 turns ratio is 0.3, and the acquisition rule is Has rate(doctor, ?score),
q1.poss = 3/0.3 = 10.
Fig. 6
PossiNum estimation process of plan A
Similarly, q3.poss = 3/0.4 = 7.5. In case of the lack of a ‘field’ value in the partial
results, q4.binding = {wang1, 9, BeiyiHospital} ∪ binding(hospital). The PossiNum
calculation of q4 should consider supplying the missing data; therefore, q4.poss =
(7.5 + 1)/0.4 = 21.25. Similarly, q5.poss = 7.5/0.4 = 18.75. The final estimated
PossiNum is3 + 10 + 7.5 + 21.25 + 18.75 = 60.5, and the estimated cost is $0.0560.5 =
$3.025.
Plan B: Fig. 7 shows the PossiNum estimation process of plan B. The difference
from plan A is the crowdsourcing direction of q2. The initialization is the same as in
plan A. When calling T ripleP ossEstCore(4, {wang1}) to process q2, q2.poss = 2(4 −
1)/0.2 = 30 owing to the density of the predicate P ositionT itle =′ P ROF ESSOR′ .
Then, T ripleP ossEstCore(30, {wang1, 9} ∪ binding(doctor)) is called to process q1,
q1.poss = 30/0.3 = 100. Similarly, q3.poss = 30/0.4 = 75, q4.poss = (75 + 1)/0.4 =
190, and q5.poss = 75/0.4 = 187.5. The final estimated PossiNum is 30 + 100 + 75 +
190 + 187.5 = 585, and the estimated cost is $0.05585 = $29.25.
As shown above, plan A costs less than plan B. Therefore, to optimize the query
cost, plan A is a better choice than plan B.
20
Fig. 7
PossiNum estimation process of plan B
7 Experimental evaluation
7.1 Accuracy of the cost estimation
To evaluate the accuracy of the CroRDF cost model, we designed three experiments
to compare the actual cost with the estimated cost: no data in the knowledge base
(Experiment 1), partial data (Experiment 2), and partial data with different logic
query plans (Experiment 3). For Experiment 1, we adopted a real crowdsourcing platform (Amazon Mechanical Turk) to execute different CQPs and acquire the actual
crowdsourcing cost for comparison with the experimental result. To perform repeated
experiments and not generate actual cost, we built a crowdsourcing simulator that
returns results by selecting from a predefined set of values. We could set the simulator to either always return correct answers or return wrong answers with a certain
probability.
Experiment 1: No data. For the SPARQL query in Section 3.2, we considered
the query target MinTuples 5 by adopting the following two CQPs: Plan A Assume
that the knowledge base has data, as shown in Fig. 4, we take the query in Section 3.2
← → → → →
as an example. We take the following two CQPs as an example: A: {q2,q1,q3,q4,q5}
→ → → → →
and Plan B {q2,q1,q3,q4,q5}. The acquisition rules of plans A and B are shown in Fig.
5. Assume that the cost of each acquisition rule is $0.05 and that the crowdsourcing
start situation is no data. The actual costs of the two crowdsourcing plans are $4.5
and $45.25, respectively.
The experimental parameter settings were the same as those in the example in
Section 6.4. The estimated results were $4.835 and $48.33, respectively. Fig. 8 illustrates the comparison between the estimated costs and the actual costs of the two
plans. As shown in the figure, the overall estimated costs were very close to the actual
costs, although there were still minor errors (7.4% and 6.8%, respectively) for two
21
Plan B
Plan A
Fig. 8 Accuracy of the cost estimation without data
main reasons. First, our turns ratio and density settings were not sufficiently accurate. Second, our PossiNum estimation algorithm uses some simple assumptions. For
the acquisition rule P (?var, CON ST ), it is assumed that the results always satisfy a
constant restriction, but this is often not the case.
Experiment 2: Partial data. Because of the crowdsourcing cost and latency arising from the repeated experiments performed on a real crowdsourcing platform, we
adopted a crowdsourcing simulator to simulate the crowdsourcing platform to collect
data for the following experiments. We mainly considered the target MinTuples n and
observed the estimated results and actual results under different existing data distributions. Consider two different types of SPARQL queries (star structure and chain
structure):
Query I:
select?doctor, ?hospital, ?position, ?score
W here{q1 :?doctorW orkIn?hospital, q2 :?doctorP ositionalT itle?position, q3 :
?doctrHas rate?score}.
→ → →
The query plan is {q1,q2,q3}, and the acquisition rules are {q1 :
Is(?, doctor), W orkIn(doctor, ?hospital); q2 : P ositionT itle(doctor, ?position); q3 :
Has rate(doctor, ?score)}.
Query II:
select?doctor, ?hospital, ?f ield
W here{q1
:?doctorHas rate?9, q2
:?doctorW orkIn?hospital, q3
:
?hospitalM ajorIn?f ield}.
← → →
The query plan is {q1,q2,q3}, and the acquisition rules are {q1 :
Has rate(?doctor, 9); q2 : W orkIn(doctor, ?hospital); q3 : M ajorIn(hospital, ?f ield)
}.
Suppose that the resolution rule of the hospital, position, and field is M ajority −
3. The resolution rule of the doctor is Distinct. The resolution rule of the score
22
Fig. 9 Accuracy of the cost estimation with partial data
is Average − 3. The turns ratios are 0.4, 1 and 0.3, respectively. Fig. 9 shows the
comparison of the estimated and actual results when N results were obtained. In the
experiment, we set three different initial states of the existing data and randomly
selected 100 and 200 different values. The crowdsourcing query was performed based
on the partial result tuples. The results showed that the estimated costs were very
close to the actual costs. Under the two data distributions, the average relative errors
of query I and query II were 10%, 34.95% and 14.18%, 13.17%, respectively.
7.2 The superiority of the best-first algorithm
Experiment 3: When looking for the lowest monetary cost CQP, we use two methods.
One is the native algorithm, which traverses all CQPs in turn and calculates the cost
of each CQP in turn. In order to improve the efficiency of search, we improved the
native algorithm and proposed the optimal best-first search algorithm. We calculated
the time to find the first CQP with the lowest cost. We observe that the time to first
CQP for the best-first search algorithm is almost slower growth as the size of the
BGPs increases in Figure.10. Figure. 11 shows the ratio between the time to find the
first CQP and the time for exhaustive search.
Fig. 10 The time of finding the first CQP with the lowest coast
23
Fig. 11 The ratio between the time to first CQP and exhaustive search
According to Figure. 11, we compare the ratio between the time to the first CQP
and exhaustive search. Obvious, The result of the best-first search is better than the
native algorithm.
8 Conclusion
This paper presented CroRDF to complete RDF queries via crowdsourcing with a
crowdsourcing query plan optimizer that finds the optimal CQP based on the estimated monetary cost. According to the characteristics of the RDF data and the query
requirements, we defined the data model and extended the SPARQL query statement.
We proposed best-first search algorithm based on triple pattern sequences and acquisition rule selection and a monetary cost estimation algorithm. Through the comparison
of actual data and simulation data, the accuracy of our cost estimation algorithm and
the validity of the plan enumeration algorithm were verified.
In future work, we will study how to optimize multiple SPARQL crowdsourcing
queries that integrate a reasoning module and extract the common query substructure
to turn multiple queries into one query for crowdsourcing to effectively reduce the
crowdsourcing cost.
Declarations
Availability of supporting data. Not Applicable.
Competing interests.
The authors declare that they have no competing interests.
Availability of supporting data. Not Applicable.
24
Author contributions. All authors contributed to the study conception and
design. Depeng Dang prepared material preparation, collected and analyzed data.
Jingfan Yang and Hu Gao wrote the first draft of the manuscript, Tingting Fang prepared figures, and all authors commented on previous versions of the manuscript. All
authors read and approved the final manuscript.
Acknowledgments. Depeng Dang is the corresponding authors of this paper. This
research is supported by The National Key Research and Development Program of
China under Grant No.2020YFC1523303;the National Natural Science Foundation of
China under Grant No. 61672102, No.61073034, No. 61370064 and No. 60940032;
the National Social Science Foundation of China under Grant No.BCA150050; the
Program for New Century Excellent Talents in the University of Ministry of Education of China under Grant No. NCET-10-0239; the Open Project Sponsor of Beijing
Key Laboratory of Intelligent Communication Software and Multimedia under Grant
No.ITSM201493;and the Science Foundation of Ministry of Education of China and
China Mobile Communicaions Corporation under Grant No. MCM20130371.
Ethical Approval. Not Applicable.
References
Acosta M, Simperl E, Flöck F, et al (2018) Hare: An engine for enhancing answer
completeness of sparql queries via crowdsourcing. In: Companion Proceedings of the
The Web Conference 2018, pp 501–505, https://doi.org/10.1016/j.websem.2017.07.
001
Ali W, Saleem M, Yao B, et al (2022) A survey of RDF stores & SPARQL engines
for querying knowledge graphs. The VLDB Journal 31(3):1–26. https://doi.org/10.
1007/s00778-021-00711-3
Balakrishnan V, Bhaskar N (2020) Optimized database using crowdsourcing. In: Inventive Communication and Computational Technologies: Proceedings of ICICCT 2019,
Springer, pp 1217–1221, https://doi.org/10.1007/978-981-15-0146-3 118
Ceschia S, Roitero K, Demartini G, et al (2022) Task design in complex crowdsourcing experiments: Item assignment optimization. Computers & Operations Research
148:105995. https://doi.org/10.1016/j.cor.2022.105995
Chaudhuri S, Shim K (1993) Query optimization in the presence of foreign functions.
In: VLDB, Citeseer, pp 529–542, https://doi.org/10.5555/645919.672647
Chittilappilly AI, Chen L, Amer-Yahia S (2016) A survey of general-purpose crowdsourcing techniques. IEEE Transactions on Knowledge and Data Engineering
28(9):2246–2266. https://doi.org/10.1109/TKDE.2016.2555805
Demartini G, Trushkowsky B, Kraska T, et al (2013) Crowdq: Crowdsourced query
understanding. In: CIDR
25
Deng Z, Xiang Y (2021) Multistep planning for crowdsourcing complex consensus tasks. Knowledge-Based Systems 231:107447. https://doi.org/10.1016/j.knosys.
2021.107447
Doan A, Ramakrishnan R, Halevy AY (2011) Crowdsourcing systems on the worldwide web. Communications of the ACM 54(4):86–96. https://doi.org/10.1145/
1924421.1924442
Fan J, Zhang M, Kok S, et al (2015) Crowdop: Query optimization for declarative
crowdsourcing systems. IEEE transactions on Knowledge and Data Engineering
27(8):2078–2092. https://doi.org/10.1109/TKDE.2015.2407353
Franklin MJ, Kossmann D, Kraska T, et al (2011) Crowddb: answering queries with
crowdsourcing. In: Proceedings of the 2011 ACM SIGMOD International Conference
on Management of data, pp 61–72, https://doi.org/10.1145/1989323.1989331
Hettiachchi D, Kostakos V, Goncalves J (2022) A survey on task assignment in crowdsourcing. ACM Computing Surveys (CSUR) 55(3):1–35. https://doi.org/10.1145/
3494522
Huang SW, Fu WT (2013) Enhancing reliability using peer consistency evaluation in
human computation. In: Proceedings of the 2013 conference on Computer supported
cooperative work, pp 639–648, https://doi.org/10.1145/2441776.2441847
Huang Y, Nazir S, Wu J, et al (2021) An efficient decision support system for the
selection of appropriate crowd in crowdsourcing. Complexity 2021:1–11. https://
doi.org/10.1155/2021/5518878
Liu Z, Li K, Zhou X, et al (2022) Multi-stage complex task assignment in spatial crowdsourcing. Information Sciences 586:119–139. https://doi.org/10.1016/j.ins.2021.11.
084
Malhotra A, Majchrzak A, Bonfield W, et al (2020) Engaging customer care employees
in internal collaborative crowdsourcing: Managing the inherent tensions and associated challenges. Human Resource Management 59(2):121–134. https://doi.org/10.
1002/hrm.21952
Noel-Storr A, Dooley G, Elliott J, et al (2021) An evaluation of cochrane crowd found
that crowdsourcing produced accurate results in identifying randomized trials. Journal of Clinical Epidemiology 133:130–139. https://doi.org/10.1016/j.jclinepi.2021.
01.006
Parameswaran AG, Park H, Garcia-Molina H, et al (2012) Deco: declarative crowdsourcing. In: Proceedings of the 21st ACM international conference on Information and knowledge management, pp 1203–1212, https://doi.org/10.1145/2396761.
2398421
26
Park H, Widom J (2014) Crowdfill: collecting structured data from the crowd. In:
Proceedings of the 2014 ACM SIGMOD international conference on Management
of data, pp 577–588, https://doi.org/10.1145/2588555.2610503
Park H, Garcia-Molina H, Pang R, et al (2012) Deco: A system for declarative crowdsourcing. Proceedings of the VLDB Endowment 5(12):1990–1993. https://doi.org/
10.14778/2367502.2367555
Pérez J, Arenas M, Gutierrez C (2009) Semantics and complexity of sparql. ACM
Transactions on Database Systems (TODS) 34(3):1–45. https://doi.org/10.1145/
1567274.1567278
Preda N, Kasneci G, Suchanek FM, et al (2010) Active knowledge: dynamically
enriching rdf knowledge bases by web services. In: Proceedings of the 2010 ACM
SIGMOD International Conference on Management of data, pp 399–410, https:
//doi.org/10.1145/1807167.1807212
Saralioglu E, Gungor O (2022) Crowdsourcing-based application to solve the problem
of insufficient training data in deep learning-based classification of satellite images.
Geocarto International 37(18):5433–5452. https://doi.org/10.1080/10106049.2021.
1917006
Shaukat K, Shaukat U (2016) Comment extraction using declarative crowdsourcing (coex deco). In: 2016 International Conference on Computing, Electronic
and Electrical Engineering (ICE Cube), IEEE, pp 74–78, https://doi.org/10.1109/
ICECUBE.2016.7495258
Wahl J, Füller J, Hutter K (2022) What’s the problem? how crowdsourcing and textmining may contribute to the understanding of unprecedented problems such as
covid-19. R&D Management 52(2):427–446. https://doi.org/10.1111/radm.12526
Wang M, Huang Q, Wu N, et al (2022) Rdf subgraph matching by means of star
decomposition. Journal of Internet Technology 23(7):1613–1621. https://doi.org/10.
53106/160792642022122307015
Yaegashi M, Saito S, Nakano T, et al (2022) Do you know how humans sound?
exploring a qualification test design for crowdsourced evaluation of voice synthesis
quality. In: 2022 Asia-Pacific Signal and Information Processing Association Annual
Summit and Conference (APSIPA ASC), IEEE, pp 1–6, https://doi.org/10.23919/
APSIPAASC55919.2022.9979812
Zeng K, Yang J, Wang H, et al (2013) A distributed graph engine for web scale
rdf data. Proceedings of the VLDB Endowment 6(4):265–276. https://doi.org/10.
14778/2535570.2488333
27