Why Pay?: Exploring How Financial Incentives
are Used for Question & Answer
Gary Hsieh, Robert E. Kraut, Scott E.
Hudson, CHI 2010
Research Methods
• Data collection
– Mahalo Answers
• Rating questions in Mahalo
– Randomly selected 800 questions
• 400 were user-paid, 400 were not.
– Rating (1) question type (fact, opinion, advice,
non-question; non-mutual exclusive), and (2)
question value (sincerity, urgency, difficulty)
– Amazon M-Turk: 401 raters; each rated 19
questions on average
Results
• Question asking
– Factual vs. is paid?
– Difficulty level vs. reward levels
• Question answering
– Rewards/Opinion/Sincerity vs. answer length
– Rewards/Opinion/Difficulty vs. answer count
– Opinion/Difficulty vs. answer quality
• Archival use
– Higher value; higher archival value
Review
• Askers value longer answers, quick responses, and reliable
sources
• Answerers learn over time (as such, receiving higher
ratings, slightly earning more)
• Answerers tend to specialize into several topics (which may
lower actual hourly wage)
• Financial incentives improves quantity but not quality of
answers (and reputation of repliers is related to quality)
• Tangible (monetary) and intangible incentives have positive
impact on the number of answers per answerer
• Question type matters (factual vs. conversational)
• How much a question is paid can be used as an indicator of
how valuable certain Q&A exchanges are
Q&A over Social Networks
KSE 652
Uichin Lee
Aardvark: The Anatomy of a LargeScale Social Search Engine
Damon Horowitz, Aardvark
Sepandar D. Kamvar, Stanford University
WWW '10
Original slides by Hailong Sun, April 13, 2010
Web Search vs. Social Q&A
• Google is about traditional Web search
– Give me keywords, I will provide contents
– Search for the most suitable contents
• While Aardvark is about social search
– Users can ask questions in natural language, not keywords
– Content is generated “on-demand”, tapping the huge
amount of information in peoples’ heads (i.e., everyone
knows something)
– The system is fueled by the goodwill of its users
Problem in Aardvark
Search
Engine
• How to find the user who can best answer a
given question?
Aardvark Architecture
2
1
Question?
3
Edit question?
Determine the appropriate
topics for the question
3
(Routing Suggestion Request)
-- find a list of candidate
answerers (and rank them)
4
Ask one by one
Social graph indexing
User’s topic parsing
VarkRank: Relevance and Connectedness
Question
User
User
User
Topic
User
User
User
P(ui|t)
Used for measuring relevance score of
a user’s question (query dependent)
User
User
User
P(ui|uj)
Used for measuring connectedness
score between users (query
independent)
VarkRank: Relevance and Connectedness
• Given a question q, the probability that a user ui can
answer it (relevance score)
• Score that user ui can answer a question from uj: (query
dependent user’s query relevance score * query
independent user connectedness score)
– Query independent user quality score = p(ui|uj): i.e., user i
delivers a satisfying answer to user j (simply measured using
connectedness)
Relevance Scores (Expertise)
• How to find experts on a given topic?
• Expertise score: p(ui|t), and w/ Bayes’ law, we have:
• For each user, we need to profile a user’s interest in a
given topic by using the following information sources:
–
–
–
–
–
3+ topics provided by a user
Topics provided by friends of a user
Online profiles
Online unstructured data
Status message updates
13
Connectedness Scores
• Connection strengths between people i.e., p(ui|uj) are computed
using a weighted cosine similarity over this feature set (normalized)
• Utilize existing social networks
– Facebook, Twitter, LinkedIn…
• Feature set measures similarities in demographics/behavior
–
–
–
–
–
–
–
–
Social connection (common friends and affiliations)
Demographic similarity
Profile similarity (e.g., common favorite movies)
Vocabulary match (e.g., IM shortcuts)
Chattiness match (frequency of follow-up messages)
Verbosity match (the average length of messages)
Politeness match (e.g., use of “Thanks!”)
Speed match (responsiveness to other users)
Analyzing Questions: TopicMapper
• Question classification:
–
–
–
–
Question or not?
Inappropriate question?
Trivial question?
Location sensitive question?
• Map a question to a topic (weighted linear sum of the
following features)
– Keyword matches w/ a user’s profile topics?
– Classifies the question text into a taxonomy of roughly 3000
popular question topics (using an SVM trained on an annotated
corpus of several million questions)
– Extracting salient phrases from questions and find semantically
similar user topics
Ranking Algorithm
• Topic and connectedness matching  availability
• Routing Engine prioritizes candidate answerers
– Optimize the chances that the present question will be answered
– Yet, preserving the available set of answerers (i.e., the quantity of
“answering resource” in the system) as much as possible by spreading
out the answering load across the user base
• Considering factors: currently online users (e.g., via IM presence
data, iPhone usage, etc.), user’s daily activity history, and user’s
response history (lowering scores of non-responsive users)
• Conversation Manager serially inquiring whether candidates would
like to answer the present question; and iterating until an answer is
provided and returned to the asker.
User Interface
IM
Email
Twitter
Aardvark
Deployment Status
• Aardvark is actively used
– Users: from 2,272 to 90,361
– 55.9% active users, 73.8% passive
users
– 3,167.2 questions/day
– 3.1 queries/month
• Mobile users are particularly active
–
–
–
–
Average 3.6322 sessions/month
Comparison: Google
Desktop v.s. mobile users: 3
Mobile users: 5.68 sessions/month
Categories of Questions in Aardvark
• Questions are highly contextualized
– Average query length: 18.6 words
• 2.2~2.9 for Web search
– 45.3% are about context
• Questions often have a subjective element
– What are the things/crafts/toys your children have made that made them
really proud of themselves?
Answers
• Answers
– 57.2% received answers in less than 10 minutes
– A question receives 2 answers averagely
– The quality of answers are good
• 70.4% are “good”; 14.1% are “OK”; 15.5% are “bad”
Distribution of questions and answering times
Distribution of questions and number of answers received
Topic Distribution
• People are indexable
– 97.7% have 3+ topics
21
Comparative Evaluation with Google
• Experimental setup
– Randomly select a group of questions
– Insert a tip “do you want to help Aardvark run an
experiment?”
– Recording response time and quality of answers from
Google and Aardvark
• Experimental results
– Aardvark: 71.5% answered; rating: 3.93 (σ=1.23)
– Google: 70.5% answered; rating: 3.07 (σ=1.46)
• Aardvark is more suitable for subjective questions
What Do People Ask Social Networks?
Meredith Ringel Morris, MSR
Jaime Teevan, MSR
Katrina Panovich, MIT
http://www.yelp.com
good restaurants in atlanta
Questions About People’s Questions
• What questions do people ask?
– How are the questions phrased?
– What are the question types and topics?
– Who asks which questions and why?
• Which questions get answered?
– How is answer speed and utility perceived?
– What are people’s motivations for answering?
What Is Known About Question Asking
•
•
•
•
Collaborative search [Morris & Teevan]
Searching v. asking [Evans et al.; Morris et al.]
Expertise-finding [vark.com; White et al.; Bernstein et al.]
Online question answering (Q&A) tools
– Question type [Harper et al.: conversational v. informational]
– Response rate [Hseih & Counts: 80%]
– Response time [Zhang et al.: 9 hours; Hseih & Counts: 3 hours]
– Motivation [Raban & Harper; Ackerman & Palen; Beenan et al.]
Survey of Asking via Status Messages
• Survey content
– Used a status message to ask a question?
• Frequency of asking, question type, responses received
• Provide an example
– Answered a status message question?
• Why or why not?
• Provide an example
• 624 participants
– Focus on Facebook and Twitter behavior
Questions About People’s Questions
• What questions do people ask?
– How are the questions phrased?
– What are the question types and topics?
– Who asks which questions and why?
• Which questions get answered?
– How is answer speed and utility perceived?
– What are people’s motivations for answering?
Questions: Phrasing
• Questions short (75 characters, 1 sentence)
• 18.5% of phrased as a statement
I need a recommendation on a good all purpose pair of sandals.
• Often scoped
– 1 out of 5 directed to “anyone”
Anyone know of a good Windows 6 mobile phone that won’t
break the bank?
– Network subset
Hey Seattle tweeps: Feel like karaoke on the Eastside tonight?
Questions: Types
Type
%
Example
Recommendation
29%
Building a new playlist – any ideas for good running
songs?
Opinion
22%
I am wondering if I should buy the Kitchen-Aid ice
cream maker?
Factual
17%
Anyone know a way to put Excel charts into LaTeX?
Rhetorical
14%
Why are men so stupid?
Invitation
9%
Who wants to go to Navya Lounge this evening?
Favor
4%
Need a babysitter in a big way tonight… anyone??
Social connection
3%
I am hiring in my team. Do you know anyone who
would be interested?
Offer
1%
Could any of my friends use boys size 4 jeans?
Questions: Topics
Topic
%
Example
Technology
29%
Anyone know if WOW works on Windows 7?
Entertainment
17%
Was seeing Up in the theater worth the money?
Home & Family
12%
So what’s the going rate for the tooth fairy?
Professional
11%
Which university is better for Masters? Cornell or
Georgia Tech?
Places
8%
Planning a trip to Whistler in the off-season.
Recommendation on sites to see?
Restaurants
6%
Hanging in Ballard tonight. Dinner recs?
Current events
5%
What is your opinion on the recent proposition that
was passed in California?
Shopping
5%
What’s a good Mother’s Day gift?
Philosophy
2%
What would you do if you had a week to live?
Questions: Who Asks What
Type
Recommendation
Topic
old
men
Opinion
Twitter
Entertainment
women
Factual
Rhetorical
Invitation
Technology
Home & Family
Professional
young
Places
Favor
Restaurants
Social connection
Current events
Offer
Shopping
Philosophy
Facebook
Questions: Motives for Asking
Topic
%
Example
Trust
24.8%
I trust my friends more than I trust strangers.
Subjective
21.5%
Search engine can provide data but not an opinion.
Thinks search
would fail
15.2%
I’m pretty search engine couldn’t answer a question of
that nature.
Audience
14.9%
Friends with kids, first hand real experience.
Connect
12.4%
I wanted my friends to know I was asking the question.
Speed
6.6%
Quick response time, no formalities.
Context
5.4%
Friends know my tastes.
Tried search
5.4%
I tried searching and didn’t get good results.
Easy
5.4%
Didn’t want to look through multiple search results.
Quality
4.1%
Human-vetted responses.
Questions About People’s Questions
• What questions do people ask?
– How are the questions phrased?
– What are the question types and topics?
– Who asks which questions and why?
• Which questions get answered?
– How is answer speed and utility perceived?
– What are people’s motivations for answering?
Answers: Speed and Utility
• 94% of questions received an answer
• Answer speed
– A quarter in 30 minutes, almost all in a day
– People expected faster, but satisfied with speed
– Shorter questions got more useful responses
• Answer utility
– 69% of responses helpful
Answers: Speed and Utility
Type
Topic
Recommendation
Opinion
Technology
Fast
Factual
Rhetorical
Entertainment
Home & Family
Unhelpful
Professional
Invitation
Places
Favor
Restaurants
Social connection
Current events
Offer
Shopping
Philosophy
Answers: Motives for Answering
Motive
%
Example
Altruism
37.0
Just trying to be helpful.
Expertise
31.9
If I’m an expert in the area.
Question
15.4
Interest in the topic.
Relationship
13.7
If I know and like the person.
Connect
13.5
Keeps my network alive.
Free time
12.3
Boredome/procrastination.
Social capital
10.5
I will get help when I need it myself.
Obligation
5.4
A tit-for-tat.
Humor
3.7
Thinking I might have a witty response.
Ego
3.4
Wish to seem knowledgeable.
Answers About People’s Questions
• The questions people ask
– Short, directed to “anyone”
– Subjective questions on acceptable topics
– Social relationships important motivators
• The questions that get answered
– Fast, helpful responses, related to length and type
– Answers motivated by altruism and expertise
QUESTIONS?
Meredith Ringel Morris
Jaime Teevan
Katrina Panovich
M. R. Morris, J. Teevan, and K. Panovich. What Do People Ask Their Social
Networks, and Why? A Survey Study of Status Message Q&A Behavior. CHI
2010.
M. R. Morris, J. Teevan, and K. Panovich. A Comparison of Information
Seeking Using Search Engines and Social Networks. ICWSM 2010 (to appear).