Field Studies for Pervasive Computing

PERVASIVE COMPUTING USER STUDIES A.J. Bernheim Brush Who am I?     Ph.D. in Computer Science Researcher at Microsoft Research Technology for families, workgroups (HCI/CSCW/Ubicomp) “Love” studies Why do a user study?  Bad Reasons  You don’t have anything else to do…  You think it’s a requirement to get your paper accepted  It might be fun to see how people use your stuff  Good Reasons  You’re designing stuff for people to use. Wouldn’t it be nice to know how they might use it?  There is a new domain or behavior you want to observe The Reality….  User Studies are a lot of work  Really, more work than you ever expected  No, really I’m not kidding  Understanding your GOAL is critical Studying current behavior: What are people doing now?  Proof of concept: Does my novel technology work for people?  Experience using a prototype: How does using my prototype change people’s behavior or allow them to do new things  WARNING: INTERACTIVE EVENT AT THE END START THINKING ABOUT ONE OF YOUR OWN PROJECTS Outline       Introduction Types of Studies Study Design Example Ten Mistakes to Avoid Your Turn Goal Types Many types of studies Ethnography Survey Interviews Focus Groups Logging Learn about a domain Design inspiration “Discount Usability” Heuristic Evaluations Lab studies Iterative Testing Fix & understand your prototype Lab studies Field Studies Logging Does it work? Can people use it? How does it compare to other designs/prototypes? Goal Types Many types of studies Ethnography Survey Interviews Focus Groups Logging Learn about a domain Design inspiration “Discount Usability” Heuristic Evaluations Lab studies Iterative Testing Fix & understand your prototype Lab studies Field Studies Logging Does it work? Can people use it? How does it compare to other designs/prototypes? Surveys     Easy to get large number of people Design guidance Evaluation of deployed system Surprisingly hard to do well….  Phrasing of questions  Biased responses  Pilot your survey! “Discount Usability” (Jakob Nielsen) If you are building prototype very useful to get feedback from users early and often  Low-cost, Quick, Iterative, Small N, Identify big problems  Lo-fi prototypes     Heuristic evaluation   Paper version can be very helpful People feel ok telling you to change stuff No feedback on responsiveness etc. Experts review the interface based on list of heuristics Cognitive walk-through  Determine tasks, review and ask questions for each task WARNING: Not typically a research contribution Lab Studies  Bring participants into a lab  Minimize   variability Hypothesis testing Independent variables between conditions  Interface A vs. Interface B  Control condition?  Measure dependent variable  Speed of use, … (Quantitative)  Preference, …. (Qualitative) http://www2.sta.uwi.edu/usability/facilities.htm Between vs. Within Subjects Designs  Between subjects  Participants each in different condition E.g., everyone randomly assigned to a group  ANOVA, t-test   Within subjects  I PREFER DUE TO PARTICIPANT VARIABILITY Each participants experiences all conditions E.g., expertise, practice or individual differences accounted for  RM ANOVA, paired t-test   Have to worry about repetition and carryover effects  Counterbalancing, Latin square, etc. Statistics  Descriptive Statistics (count, mean, …) Nominal – categories: frequency only  Ordinal – Ranked preference: frequency, median  Interval – numbers: frequency, sum, median, mean..    Watch out for outliers Inferential statistics Are results statistically significant between groups?  T-test, ANOVA, paired t-test, etc.  Significance values   If p < 0.01, there is a 99% chance that the data collected represents a real difference in the population rather than a sampling error Field study   In-situ (“not on your turf”) Trading “control” for realism  Think  carefully if this is important All types  current behavior,  proof-of-concept  prototype WARNING: Often good idea to do lab study before field study Outline       Introduction Types of Studies Study Design Example Ten Mistakes to Avoid Your Turn How do I choose?  You might not… LINC: An Inkable Digital Calendar  What is your goal/research contribution? Research Question/Goal Your Research Question is critical Bad: How will families use SPARCS? Better: Do sharing suggestions promote sharing? There are very few right decisions, instead decisions you need to justify Study Design   What type of study? What will your participants do during the study?  Give      them hardware? Give them tasks? Why type of participants you should recruit? What data will you collect? How long will the study be? Where can you skimp during the study…. What absolutely has to work (if it’s a prototype) Human Subjects  Ethical treatment of people in the study Respect—remember that they are doing YOU a favor  Participants can stop at any time       Consent Forms Privacy Statements Compensation Your organization should have some review process THIS IS IMPORTANT! What will they will be doing during the study  How will you report on what they did  http://www.hhs.gov/ohrp/irb/irb_chapter3.htm PILOT!!!!   Allow time to a pilot study, Run through the entire methodology with volunteer participants.  Uncover system problems  Uncover experimental design problems  Uncover problems with materials IMPORTANT IMPORTANT IMPORTANT Participant Profile  What types people do you want to participate?  All the same? Participant Profile  What types people do you want to participate?  All the same?  All different? Participant Profile  What types people do you want to participate?  All the same?  All different?  People with extreme characteristics vs. “normal” people Participant Profile  What types people do you want to participate?  All the same?  All different?  People with extreme characteristics vs. “normal” people  Consider  Age  Gender  Technology  …. experience How many participants?  This can be a difficult question  Between or within subjects  What claims you are making  What is feasible  Some will drop out! Length   How long should the study be? Another difficult question….  Novelty  What are you asking participant to do?  Long term use vs. feasibility Data Collection    How will you understand if you answered your research question? Quantitative Data (logs, timing, # errors, …) Qualitative Data (interviews, surveys, …) TRIANGULATE between multiple sources  Logging  You must have a plan going in about how you will use the log data  Risk of forgetting to log something important  Logging too much can create an analysis nightmare  Make a list of questions you expect to answer with log data  How many times did they upload a photo?  How many days did they use your prototype? Logging Question How/Notes 1. How often was SPARCS running, when did it stop? # of logfiles, LogfileStart(0) LogfileClosed 2. Is automatic previewing setup? When Settings:IsPreviewing 3. Is automatic publishing setup? When Settings:IsAutoPublishing 4. How many people’s feeds were they subscribed to? SPARCS_Feed ADD as well as number of unique Create View SPARCS_Feed, add or remove tags Calendar_Type, Settings:CalendarSource tag 5. When did they add or subtract feeds? 6. What type of calendar feed are they using Qualitative Data  Surveys Pre-survey  Post condition  Post-survey   Experience Sampling Methodology Small set of question  Event triggered, random….     Diaries Interviews (semi-structured, structured) Observation Analyzing Qualitative Data    Affinity Diagramming Coding of Comments Inter-rater reliability What if it doesn’t work?  There are many ways a study can fail  Technical problems  They don’t like it   Nobody but you cares about usability problems Brace yourself for this  Figure out what has to work and skimp other places  Comparison between prototypes  Pilot studies Outline       Introduction Types of Studies Study Design Example Ten Mistakes to Avoid Your Turn Example CareNet Consolvo, Roessler, Selton Intel Research Seattle CareNet Methodology      4 “care networks” (3F, 1M) 13 people (4 elders and 9 care givers) 2-3 people had CareNet displays 3 weeks Compensation  With CareNet - $150  Elders $75- $300 (depend on data provided)  Sept. – Dec. 2003 CareNet Data Collection       Interviews, before and after (semi-structured, recorded) Questionnaires: half-way, end Photos 3-6 times per day by phone to collect data for display Glowing display People will take your technology apart, even when you tell them not too. Lots of other examples… ButterflyNet Yeh, Liao, Klemmer, Guimbretière, Lee, Kakaradov, Stamberger, Paepcke Stanford, Maryland UbiFit Consolvo, McDonald, Toscos, Chen, Froehlich, Harrison, Klasnja, LaMarca, LeGrand, Libby, Smith, Landay Intel Research Seattle SuperBreak Morris, Brush, Meyers Microsoft Research Wearable Jersey Display Page and Moere University of Sydney Outline       Introduction Types of Studies Study Design Example Ten Mistakes to Avoid Your Turn #10 Not enough people involved    It takes a village Huge time commitment, 24 hour support for field studies Do  Include multiple people #9 Not being prepared   Don’t want to realize at the end of the study that you forgot to do something important Do  Study design document  Research Question  Participant Profile  Methodology (within/between etc)  Timeline  Data collection  Pilot studies WARNING: If you do not plan, you plan to fail! #8 Not enough time for logistics      Everything takes time…. (more than you think) Recruiting Installation (e.g. 16 people X 2 hours = ) Support Do  Allow plenty of time for the study  Have enough people #7 Seeing what you want to see   We all want our prototypes to be popular Do:  Think carefully about how you discuss the technology with participants  Avoid leading questions  Stay close to the data and find multiple support for conclusions  Use neutral language, “Tell me more,” “ummm”  Don’t get defensive #6 Being judgmental    Users are always “right” They may say things that are offensive, objectionable, etc. Do:  Leave your opinions at home  Collect feedback  Use neutral language #5 Not monitoring usage   Don’t want to find out at the end of the study that people were not using the prototype Do:  “phone home” messages  Server logs  Have a plan about when and how you might intervene #4 Not collecting a variety of data    Hard to understand logs without interviews Hard to know if what someone says in an interview matches their use without logs Do  Interviews  Surveys  Logs  Support for your findings in multiple ways gives you (and reviewers) more confidence #3 Prototype is not robust enough    Technical challenges are a distraction You are making a “product” Do  Usability studies  Pilots with friendly folks #2 Making inappropriate claims    One of the biggest and most common mistakes Your study is one data point Do:  Being careful with your language  Include a “limitations” section  Tell a clear story (don’t have to tell every finding) #1 Not having a clear research question     Hard to explain the choices you made Hard to explain your findings May end up focusing on usability problems Do  Have a clear research question Your Turn….   Get into groups of 2-3 Choose an example prototype …   Determine a research question Come up with a study design  Participants  Length  Data to collect  Timeline for study Your thoughts Acknowledgments Ed Cutrell MSR India Sunny Consolvo Intel Research Seattle Beverly Harrison Intel Research Seattle User Studies Are Fun!    Always learn something Often surprising! Inspire you References:     Brush, Ubiquitous Computing Field Studies in Ubiquitous Computing Fundamentals. Green & Salkind Using SPSS for the Windows and Macintosh: Analyzing and Understanding Data (nth Ed.) APA Publication Manual  HCI textbooks  Preece, J., Rogers, Y., and Sharp, H. Interaction design: beyond human-computer interaction. 2002. John Wiley & Sons, Inc A.J. Brush, ajbrush@microsoft.com

Field Studies for Pervasive Computing

Related documents

Products

Support

Field Studies for Pervasive Computing

Related documents

Add this document to collection(s)

Add this document to saved

Suggest us how to improve StudyLib