Housekeeping
• Seminar Topics doodle poll on facebook page –
sign up by tomorrow night (Fri) please, and I’ll
make the schedule this weekend
• Option to propose a novel topic. One slot on the
last day – I may free up another couple on other
days depending on interest levels
• Post-mortem of 1st MP1 Protocol testing
• Preparation
• Volunteers
• Those rating sheets
• Critiques
Agenda
• Today:
• Finish off contextual interviews
• Diary Studies
• Discussion of diary studies reading (assigned for
Tues)
• Questionnaire handout (start in class – complete
for Tuesday)
• Next Tuesday:
• Finish off Questionnaires, Methodology Matters
Contextual inquiry
 Interviewees are interviewed in their context,
when doing their tasks, with as little
interference from the interviewer as possible.
 Allows probing of “why?”
 Can be real-time or record interesting actions
for later discussion
“Typical” 4 phased approach
 Traditional interview
 Get an overview, establish trust, start recording
 Switch to a master-apprentice relationship
 Tell them what you want to observe
 Make sure to establish when ok to interrupt
 Observe, ask questions
 Take notes
 Balance need to understand with impact of
interruptions
 Summarization
 Go over observations and your understanding with
participant
 Make sure that you go it right
Other ways of providing
context
 If natural observation not possible, can ask
them to demonstrate specific tasks of
interest
 Can provide task scenarios and ask them to
perform
 “Think aloud” aloud protocols
Other ways of getting
observational data
 Logging
 Screen recording (check out Camtasia)
 Trace data
CSCI 4163/6610, Winter 2014
DIARY STUDIES
Advertising Diary
 Did you remember to fill it out?
 When did you do it?
 As encountered?
 In a batch?
 Were the instructions clear?
 What was easy/hard about it?
 Do you think it captured your exposure to
advertising?
Diary studies…
 Participants collect data about events
 As they happen
 In the context of the event (in situ)
 Can think of like a small, longitudinal
questionnaire
 Often used to prompt interview discussion
(similar to observations in a contextual
inquiry)
 Can help understand rare/infrequent events
Data collection methods
 Survey style forms
 Paper (little training required, but hard to monitor)
 Online entry forms
 Google docs
 Photographs, video
 Digital/disposable camera, mobile phone
 Audio recording
 Voice recorder, mobile phone
 Aggregating data online
 Blogs (text, audio, video) , Twitter
 Rich and timely information
Type of data recorded
 When
 Date/time
 Duration
 Activity/task
 What
 Activity/task
 feelings/mood
 Context (environment/setting)
When is data recorded?
 Randomly
 In response to prompts
 At specific intervals
 Based on activity
General types of diary
studies
 Unstructured
 Ask participants to report on everyday activities
 Trying to elicit general themes
 Structured
 Ask participants to report on everyday activities by
answering specific questions about the activity
 Combination of question types
 Can also be used as a form of usability tests and
problem reports
 Ask them to complete a task and report results,
identify bugs, etc.
Variations based on purpose
Feedback (now)
Elicitation (later)
 Record everything in situ  Record aspects or triggers of an
 No follow up
 Focus on the “what”
data
 Mostly structured
 Can be burdensome on
participants and
researchers




activity
Follow up with an interview to gain
more information
Data captured is used to elicit the
“why” and the how”
Mostly unstructured
Can be problems with participant
recall
Cultural Probe
 Diary study++
 Used to not only record
activities, but capture more of “felt life”
 Stimulate thought as well as capture experiences


http://www.hcibook.com/e3/casestudy/cultural-probes/ - pack contained a small
disposable camera and a listening glass that participants used to listen at walls and
doors and write what they heard. It also contained a small solid state recorder
packaged in a sleeve that said 'dream recorder'.
Was used by designers at the Royal College of Art, London, to study the way people
see their own homes. The results were used to enable designers to get a 'feel' of the
meaning of home for many people.
 Experience sampling method (measure feelings,
moods)
 Technology Probes (prototypes, experimental artifacts)
Coordination Challenges
 Keeping participants on track
 Periodic reminders
 Feedback about the level of detail in responses
 Progressive incentives, surprise incentives/gifts
(may or may not be allowable by BREB)
 Adapting to changes
 Start analysis as soon as first results arrive
 May need to re-evaluate the diary format if the
data being captured is not what you expected!
Advantages
 Time efficient for researchers
 Lower cost than direct observations
 Can have a broad geographic distribution
 Supports contextual reports over time
 Natural environment
 Can give rich data about contexts of use
Disadvantages
 Participant fatigue
 Missed responses (frequency data is generally
unreliable – lower bound)
 Study drop outs
 “after the fact” reporting to catch up
 Behaviour adjustment
 Participants need reminders
 Can be expensive for long durations
 Volume of collected data can be
overwhelming
Study protocol
 Introductory letter
 Consent form
 Needs to motivate their participation
 Very specific instructions
 Can be helpful to give an example of a filled out
form (take care not to prime/limit)
 Diary form / collection aids
Forms of analysis
 Track temporal patterns
 Look for semantic patterns in visual data
 Combine with interview analysis
Diary reading
 When Participants Do the Capturing: The Role
of Media in Diary Studies
 What was their motivation?
 What were their research questions?
 What was their approach?
 What were their findings?
 What were the strengths of their study design?
 What were the weaknesses?
 Do you trust their findings?
Question writing exercise
 1. Answer the questionnaire (these are BAD
questions!)
 2. Take a few minutes to think about what
made the questions bad/hard to answer/hard
to analyze
 3. look at 6-17 on the questionnaire design
checklist
 For each checklist item, identify which questions
are examples of poor design
 4. On Tuesday, we will discuss this as a class