uTest
Crowdsourced Testing
AGENDA
1. INTRO
2. WHY WE NEED CROWDSOURCING FOR TESTING?
3. GENOME
4. UTEST MULTIVARIATE ANALYSIS
5. UTEST & CS CHALLENGES
6. UTEST PLATFORM NAVIGATION
uTest is the world’s largest provider of
in-the-wild testing services
200,000+ testers from 200 countries & territories
60,000+
test cycles
Functional, Load,
Usability, L10N &
Security
3,000+ customers
Web, mobile
& desktop
apps
Enterprises, SMBs & universities
WHY WE NEED CROWDSURCING FOR TESTING?
Testing inside the lab is vital. But customers and their
challenges do not necessarily live there.
ANDROID DISTRIBUTION
Sum,
N ou
re
o
op
Mar 6.0
shm
all
llip
O
gat
Lo
0
8.1 Oreo
7 .0
5 .1
8.
ow
10.0 10
9 .0
P ie
O th e r
1.Apps consumed under endless range of
environments
-Exponential spike in devices, OSes &
browsers
-Imperfect connectivity
-Infinite number of locations
2.Users have less patience than ever for buggy
apps
-Switching costs lower
-App quality issues carry higher costs
3.In-the-lab testing alone cannot meet the
demands of modern apps
TYPES OF
TESTING
SERVICES
• Scalable, on-demand access to professional testers with the geographies and
real-world infrastructure used by your customers and end users, functional
• On-demand test subjects and usability experts to provide comprehensive UX
and user interaction testing services across most demographics and configs
• Live and synthetic load and performance testing and analysis
• White hat security experts providing automated and manual penetration testing
from anywhere in the world
• Real-world localization validation using native speakers from over 190
countries
Better Applications
–In-the-wild testing – live
testers, real devices, real-world
conditions
–Unmatched coverage across large
matrix of OS/browser/location
Lower Total Cost of Testing
–Subscription based and project
based contracts
–Lower total cost of testing than
near-shore boutiques or offshore
shops
Faster Time to Market
–Test cycles created in minutes;
underway in hours; completed
in days
–Test cycle can be executed
overnight or during weekends
How
Customers
Benefit
Scalability
–200K + professional testers
from around the world
–Scale size of test team up or
down at will
HOW IT WORKS?
Specify testing needs
OS, Browser, Carrier, Device
Location, Language & Industry
Customers
Testing Community
Selected testers invited to
project
Testers report
in real-time
uTest pays testers
Customer approves
testing results
Genome:
To write a genome for a CS
system such as uTest answering
four fundamental questions is
crucial.
1. What?
What is being done? Or what is the goal of uTest?
A. Create projects platform for companies and providing them
with real-world user feedbacks. - Corresponds to Create
gene.
B. how commentaries and feedbacks can improve a product,
what features should be added or removed? - Corresponds to
Decide gene.
2.
Who?
Who undertakes the activity?
As explained earlier uTest’s TTLs (test team leaders
or team managers) and customers are in charge of creating
product testing projects. – Corresponds to hierarchy gene.
Providing commentaries and feedbacks is the main
job and it is designed to be done by community members.
– corresponds to Crowd gene.
3.
Why?
What motivates people to participate? uTest platform consists of two
group of people:
A. Management (hierarchy): in these platform hierarchies are
customers and management teams and they are essentially
motivated by money. Corresponds to Money gene.
B. Crowd: As stated in the introduction participants can improve their
testing skills, earn money and join a global community, so testers
can be motivated by both monetary and non-monetary incentives.
Corresponds to Money and Love genes.
4.
How?
A. Create
uTest customers and TTLs create a new project. Corresponds to
Hierarchy gene.
A. Decide
Every tester has his/her opinion on what changes are necessary
and how a product can be improved. Corresponds to Individual
decision gene.
What
Create - projects
Decide- Evolution
Who
Hierarchy
Crowd
Why
Money
Money, Love
How
Hierarchy
Individual decision
UTEST
MULTIVARIATE
ANALYSIS
•
•
•
•
Nature of collaboration
Type of target problem
Degree of manual effort
Role of human users
Nature of collaboration and type of target problem
UTest is an independent crowdsourcing system, meaning that it was explicitly designed to gather specific
users to perform a set of specific tasks. In case of uTest, the explicit cs system gathers users with a general
understanding of a certain product or service to review, evaluate and provide commentaries about their
experiences with it. These evaluations can help a company to gain new insights from a customer’s perspective
which in turn can lead up to a better product.
Degree of manual effort and Role of human users
UTest has created policies and procedures that determine how management teams and the crowd interact,
additionally it defines a code of conducts for these teams and testers.
According to uTest’s code of conduct, a certain product and its Cycle Overview – a document that assigns tasks,
and instructs on how testers must complete a certain task. – are provided by customers and management teams,
and eligible testers, can choose to contribute to the project by evaluating the product according to the given
instructions.
In an implicit CS system that humans serve as perspective providers, users generally do relatively little, but
uTest as an explicit standalone CS system heavily relies on the contributions made by crowd therefore users do
the main job and platform owners and customers only regulate and coordinate the efforts made by testers.
Recruiting and Retaining a Right Crowd
Compensation method and Policy
Keeping the Vision Clear
Maintaining Quality
Intellectual Property
Human Rights
UTEST CS CHALLENGES
Recruiting and Retaining a Right Crowd
Compensation method and Policy
Keeping the Vision Clear
Maintaining Quality
Intellectual Property
Human Rights
UTEST CS CHALLENGES
EXPERT
SOURCING
TEST SPECIFIC
COMMUNIY
TEST SPECIFIC
CROWDS
OPEN CROWDS
Recruiting and Retaining a Right Crowd
Recruiting and Retaining a Right Crowd
1. Community profiling
Technical: OS, browser, mobile devices,
carriers
Geographic: City, country, languages
Demographic: Age, gender, education,
hobbies
2. Community ratings & micro-ratings
By testing type
By industry
3. Precise matching
Between each project & each tester
Compensation method and Policy
forms of compensation
Monetary
Reputation
Performance-based ratings based upon a dozen factors:
Participation Level:
Lifetime & recent participation: # active test
cycles, #
reported issues
Quality of Participation
Approval percentage for bugs, test cases and usability
surveys
Accuracy of bug type & severity classifications
Ability to write test cases & create automated test scripts
Compensation method and Policy
•
•
•
Referenceable customer successes
By company size
By industry
Ability to adapt to your legacy systems
and processes
Ability to satisfy legal requirements
IP protection
NDA
Ability to do the job
Consistent
Predictable
Professional
• Confidentiality is a serious matter with
material consequence.
• Crowdsourcing is often used for prelaunch products
Best Practices:
• Ask upfront about preventative measures
Precautions… NDAs? What else?
Is the policy all-inclusive? e.g. incl.
social media, message boards, etc.
What are the consequences for breaches?
Intellectual Property
• Intellectual property (IP) is a legit issue,
esp. for the new & innovative
Copywriting
Design
Animation
Code writing
• IP policies vary within crowdsourcing orgs
Best Practices:
• Establish the ground rules: What’s their IP
policy?
How is it communicated to the crowd?
How is it enforced?
FIN