Evaluation
Eyal Ophir
CS 376
4/28/09
Readings
Methodology Matters (McGrath, 1994)
Practical Guide to Controlled Experiments
on the Web (Kohavi et al., 2007)
Methodology Matters
Methodology Matters
 Methods for Research in the Behavioral and
Social Sciences
 Different methods have strengths and
weaknesses
 Tradeoff between:
Generalizability
Precision
Realism
 Credibility requires consistency, convergence
across methods
Study Design
Find baserates, correlations, or differences
Randomization of selection, assignment to
conditions
Statistical significance
Validity (internal, statistical, construct,
external)
Measures
Self report
Trace measures
Observation (by a visible or hidden
observer)
Archival records (public or private)
Manipulation
Selection
Direct intervention
Induction (indirect intervention:
confederates, deception)
Case Study: Multitasking UI
Users play two simultaneous instantiations
of a game
Does making the two instantiations
visually different make it easier to switch
back and forth?
Case Study
Case Study
Case Study
•
Tradeoffs: Generalizability,
Precision, Realism
•
Design: baserates, correlations,
differences
•
Random selection, assignment
•
Validity: internal, statistical,
construct, external
•
Measures: self-report, trace
measures, observation, archival
records
•
Manipulation: selection,
intervention, induction
General Question
Has social psychology resisted formal
theory, and if so, why?
Practical Guide to
Controlled Experiments
on the Web
Web Experiments
OEC: Overall Evaluation Criterion
Web Experiments
Hypothesis testing and sample size
Confidence, power
Reducing the standard error
Sufficiently large sample size
OEC with inherently low variability
Reduce variability by excluding irrelevant cases
Web Experiments
Extensions for Online Experiments
Treatment ramp-up
Automation
Software Migration
Web Experiments
Limitations of web experiments
No explanation of mechanism
Focus on short term effects
Primacy/newness
Must implement treatments
Web Experiments
Implementation
Randomization
Pseudorandom with caching
Hash and partition
Assignment
Traffic splitting
Server-side
Client-side
Lessons learned (i.e.- tips for
the researcher):
Analysis
Mine the Data
Time matters
Multi-factor experiments
Lessons Learned
Trust and Execution
Run A/A tests (test your system)
Ramp-up and abort
Correct sample size
Assign 50% to treatment
Beware day of week effects
Lessons Learned
Culture and Business
Agree on OEC upfront
Beware “harmless” features
Weigh performance vs. maintenance cost
Data-driven (vs. opinion-driven) culture
Extended Case Study
Assume the game UI from the first case
study was an actual gaming site
The website is interested in promoting
multiple simultaneous games between
users, but users complain that it’s difficult
to manage multiple games
Design a web-based study informed by the
reading to test the new design
Case Study
•
OEC
•
Sample size, reducing error
•
Ramp-up, automation
•
Mechanism explanation, short vs.
long-term effects,
primacy/newness
•
Randomization/assignment
•
Mine the data, multi-factor
experiments
•
A/A tests, sample size, day of
week effects
Data-Oriented Culture
Pros?
Cons?
How can we best use user tests to inform
design and innovation?
Trade-offs of experimentation vs. intuition
Why the OEC? What are good measures
for non-commerce sites?
Do online tests maximize all McGrath’s
parameters?