IN4MATX 231
Human-Computer Interaction
Observing Users
Presenters:
Ritesh Subramanian
Tanmay Goel
Outline





Benefits and challenges of different types of
observations.
How to observe as an onlooker, a
participant, and an ethnographer.
How to collect, analyze and present
observational data.
Indirect observation using diary studies and
logging.
Qualitative and Quantitative analysis.
What and when to observe



Observation is usable at all stages during
product development.
Goals and questions determine the paradigms
and techniques used.
Observers can be:
– Onlookers
– Participants
– Ethnographers

The degree of immersion that evaluators adopt
varies across a broad outsider-insider spectrum.
Types of Observation
Observation
Outsider
looking on
Insider
Controlled
Environment
(e.g. Lab)
Field
Environment
(e.g. Natural)
“Quick and Dirty”
In usability testing
“Quick and Dirty”
In field testing
N.A.
Participant
observation (e.g. in
ethnography)
Approaches to Observation

“Quick and Dirty” observation
– It can occur anywhere, anytime
– Least formality involved

Observation in usability testing
– Video and interaction logs capture all the user
operations
– One-way mirrors or remote TV screen
– Data is used to analyze what users are doing
and provide insight into users’ affective reactions
Approaches to Observation

Observation in field studies
– The observer may be a passive observer, participant
observer or an ethnographer.
– Goal is to cause as little disruption as possible.
– Passive or outside observer – An observer in a class of
boys and girls whose primary job to keep track of how
much time a particular technology is used by each gender.
– Participant observer – Participates in social conventions of
a group, combines participation with maintenance of
professional distance for unbiased observation.
– Ethnography – Takes weeks, months or longer. Inside
information of community activity is obtained.
How to observe:
In controlled environment

Practical issues that need to thought of in advance:
– Decide where users are located so that equipment can be
setup in advance
– Decide about modes of data capture e.g. video, interaction
logs
– Equipment testing to get expected performance (audio
and video)
– Get legal user consent
– Prepare a script to guide users through the set of
questions
– It is important to make users feel comfortable
– Problem with this approach: Observers do not know what
users are thinking.
Controlled Environment –
Think-Aloud Technique




The user is trying to perform a certain task.
Observer wonders, what is going on, what is
he thinking, why did he do that, etc.
Eternalize thought process.
A big problem is occurrences of silence
during think-aloud process.
Maybe have two people work together and
talk to each other.
Controlled Environment –
Think Aloud Technique
Checklist of things to plan
before going into the field










State the initial goal and questions clearly
Select a framework to guide activities in the field
Decide how to record events
Think about how to gain acceptance and trust of users under
observation
Be prepared to go through these notes to weed out ambiguities
Highlight and separate personal opinion from what happens
Be prepared to refine and refocus the study for same or new
user groups
Think about how to handle sensitive issues
Consider working as a team; conform notes with another
member
Consider checking notes with an informant and plan to look at
the situation from a different perspective
How to observe (contd.):
In the field


Frameworks to guide observation
– The person. Who?
– The place. Where?
– The thing. What?
The Goetz and LeCompte (1984) framework:
– Who is present?
– What is happening?
– When does the activity occur?
– Where is it happening?
– Why is it happening?
– How is the activity organized?
In the field

The Robinson (1993) framework
– Space. What is the physical space like?
– Actors. Who is involved?
– Activities. What are they doing?
– Objects. What objects are present?
– Acts. What are individuals doing?
– Events. What kind of event is it?
– Goals. What do they to accomplish?
– Feelings. What is the mood of the group and of
individuals?
How to observe (contd.):
Participant observer or ethnography

Checklist for doing ethnography
– Identify the problem or goal and ask good questions to be
answered by the study
– The most important part of fieldwork is being there to
observe, ask questions and record what is seen and heard
– Collect variety of data. E.g. Notes, still pictures, audio and
video
– Be prepared to move backwards and forwards between
the broad picture and specific questions
– Analyze the data using a holistic approach in which
observations are contextualized
Data Collection Techniques

Notes plus still camera
– The least technical way of collecting data
– Transcription the first step in data analysis
– Photographs, sketches, etc. can be easily collected

Audio recording plus still camera
– Less intrusive than video
– More flexible and mobile
– One drawback is transcribing the data which can be
onerous if many hours have to be transcribed
Data Collection Techniques

Video
– Advantage of both audio and video
data; but more intrusive
– Attention becomes focused on what
is seen through the lens
– Analysis of video can be long and
time consuming
– Sound may get muffled when
recording in noisy conditions
Table source: Preece, Rogers, Sharp: Interaction
Design: Beyond Human-Computer Interaction. Wiley
Comparison
Table source: Preece, Rogers, Sharp: Interaction
Design: Beyond Human-Computer Interaction. Wiley
Comparison (contd.)
Indirect observation: tracking
user activities

Diaries
– Diaries provide a record of what users did, when they did
it and what they thought about their interactions with
technology
– Useful when users and scattered and unreachable
– Inexpensive and require no special equipment or expertise
– They rely on participants being reliable and remembering
to complete them
– Participants usually remember incidents for better or for
worse
Indirect observation: tracking
user activities

Interaction logging
– Includes analysing data from key presses, mouse
movements etc.
– Usually synchronized with video and audio logs
– Time stamped to calculate how much time a user spends
on a particular task
– E.g. Explicit counters that record visits to a website. If you
want to find out if adding a bulletin board increases the
number of visits, comparison of traffic before and after is
useful
– Unobtrusive and large volumes of data can be collected
Interaction Logging- Webtrends
Log Analyzer
Interaction Logging
Ethical Concerns: They don't know we are
watching. Shall we tell them?




If we tell users about logging they may react or change
their behavior
It depends on the context, and how much personal
information is collected
e.g. Companies tell about data logged for quality
assurance purposes
Concerns arise when personal information is used to infer
financial or health information
Analyzing, interpreting and
presenting the data

Three main types of data are explored
 Qualitative data - interpreted & used to tell the ‘story’
about what was observed.
 Qualitative data - categorized using techniques such as
content analysis.
 Quantitative data - collected from interaction & video logs.
Presented as values, tables, charts, graphs and treated
statistically.
Qualitative analysis to tell a
story






Review the data after each observation to identify key
themes
Record themes in a coherent yet flexible form
– E.g. Laptops, audio recordings
Record the date and time of each data session
As themes emerge, check your understanding with
informants or people you observe
Iterate until the story reflects observation
Report findings to a group in the form of a presentation
or written report
Analyzing and reporting
ethnographic data





Look for key events within a group that drive a group’s
activity
Look for patterns or behavior among different players
Compare sources of data against each other
Report findings in a convincing and honest way
e.g. Software tools such as Ethnograph allow ethnographers
to code their notes so that they can be searched, sorted and
retrieved. The information can be printed out as a tree
showing relationships of occurrences.
Ethnograph
Ethnograph
Qualitative analysis for
categorization

Looking for incidents or patterns
– Look for critical incidents, such as times when the user
was obviously stuck. (marked by silence, looks of
confusion)
– E.g. Jurgen Koenemann-Belliveau et al(1994) used this
approach to compare the efficacy of two versions of a
Smalltalk programming manual for novice programmers.
– They were able to identify specific problems by tracing
through sequence of incidents and thereby achieving a
holistic understanding.
Qualitative analysis for
categorization

Analyzing data into categories
– Content analysis provides a reliable and systematic way of
coding content into meaningful sets of mutually exclusive
categories
– Categories should be orthogonal i.e. no overlap
– Appropriate granularity must be selected
– e.g. Training two people to use categories and letting them
analyze the same data sample. If huge discrepancy in the
analyses is observed, then either the training was insufficient or
categories need to be redefined
– Inter-research reliability rating- Percentage of agreement
between two researchers, defined as ratio of number of items
categorized in the same way to the total number of items
observed
Qualitative analysis for
categorization

Analyzing Discourse
– Focus on the dialog, i.e. meaning of what was said rather
than content
– Discourse analysis is strongly interpretive
– Language is viewed as a constructive tool and discourse
analysis provides a way of focusing upon people’s use of
language to construct versions of their worlds
– E.g. Analyzing discourse on the Internet (e.g. in chat
rooms, bulletin boards) has started to influence designers’
understanding about user needs in these environments
Quantitative data analysis and
feedback

Quantitative data analysis
– Video data collected is usually annotated to calculate
performance times
– The data stream from interaction logs is used to calculate
performance times
– Data is analyzed using means, standard deviations, t-tests etc.

Feeding the findings back into the design
– Written reports make for easy reading and a good reference
document
– Include anecdotes, pictures, quotations and video clips to bring
study to life and stimulate interest
– Verbal presentations can be a powerful mode of feedback
Summary








Observation in usability study tends to be objective
In contrast, in participant observation the evaluator works
with the user to understand their activities and beliefs
Ethnography uses a technique wherein the ethnographer
immerse themselves in the culture they study
Observational data collection depends on the paradigm used
Combination of video, audio, diaries and logs can be used to
collect observational data
Analyzing video and data logs can be tedious due to the sheer
volume of data
Evaluators flag events in real time and return to examine
them later.
Identifying key events is an effective approach