CSCW Evaluation
Techniques
Presented by:
Christopher Edwards
Overview of Presentation
Evaluation Techniques
 Understanding Ethnography
 Using Ethnography in CSCW
 Understanding Ethnomethodology
 Ethnomethodology and CSCW
 Technomethodology
 Conclusion

Evaluation Overview
Olson and Olson.
 What are we Evaluating?
 Evaluations Techniques…
 Internal/External Validity
 Conclusion

Characteristics of Groups

Individuals Differ
in:






Skills
Ability
Knowledge
Personalities
Motivations
Agendas
Characteristics of
Organizations






System comprised of
people and
technology
Social Technology
Physical Technology
Comprised of multiple
actors
Dependency on
Communication
Information
Processing Entities
Characteristics of Task




Tasks involve different
types of material
Physical, Digital or
Ethereal
Ease or Difficulty of
Task
Differ on Core Activity

Subtasks – Tightly
Coupled/Loosely
coupled
Characteristics of
Environment

Physical
Environment



Distance between
Group members
CSCW technologies
designed to overcome
Contextual Time
• When in the day the
interaction occurs
• Effects on Distant
Group member
Characteristics of
Technology

Increasingly Varied

Technologies to
Support Conversation
• Auditory/Visual
• Back
channels/gestures

Technologies to
Support Shared Work
• Objects that support
work
• Fit of Tool to Material
Process
 Technology
Deployment
 How and Why
 Process Analysis
 Why
outcomes were affected
•Progress of Task
•Communication process
•Examined through time scales
Outcomes
The initial outcome of using
technology
 Quality of work
 Measure outcomes at every level

• Group Outcome
• Organizational Productivity
Conceptual Framework for
CSCW Studies
Group
Organization
Task
Environment
Technology
Process
Outcomes
Tools used to evaluate
CSCW technologies
The Survey




Set of questions
Fixed Alternatives
Statistically
Analyzed
Wording of
questions
problematic
Interview

Structure of
interview



Formal and
structured
Unstructured
Analysis can be
complicated
Experimental

Controlled Setting
Specific Task
 Conditions
 Assignment of Participants
 Useful for making inferences about
causality

Case Study


Examines a single
or small number of
cases
Exploratory
research
Ethnography


Method adopted
from Anthropology
Describing Culture


Used originally to
describe other
cultures
Misunderstood
method
Many other Methods
Diaries
 Analytic Field Studies
 Quasi Experimental
 Longitudinal Studies
 Historical Studies

Internal and External
Validity
HIGH
Level of
Internal
validity
Laboratory
Experiments
Field
Experiments
Surveys
Ethnographies
LOW
Level of External Validity
HIGH
Conclusion to the Overview
Different factors influence use and
evaluation of CSCW software
 Framework of CSCW studies
 Evaluation Techniques
 Validity of Techniques

Short Break

Reconvene in 3 minutes

*Upcoming – Understanding
Ethnography


Using Ethnography
Ethnography and CSCW
Ethnography
Understanding Ethnography
 Sociology Adoption
 Using Ethnography
 CSCW Ethnography in Design
(Hughes)

Understanding Ethnography


Ethnography is
loosely applied to
qualitative
research
Home is originally
from Anthropology

Aim to describe
cultural
interpretation
Understanding Ethnography
Understanding culture “from an
insiders point view”
 Three sources of data

Participant Observation
 Interviews
 Collection of representative artifacts

Sociology Adoption
Originally used to study distant
cultures
 Chicago School of Sociology

Studies focused on exploration of
groups in urban settings
 Cultural comparisons in USA
 Family of Ethnographic Techniques

Class Participation Time




In Pairs (Saul and
myself included)
Everyone gets a
Handout
For a total of 5
Minutes (2.5 Minutes
each)
Each member of the
pair (one at a time)
asks the other
questions from the
sheet
Debrief of Class
participation
What answers were given….
 Obviously not a long term
ethnography study
 Depending on your relationship to
this lab – differing perspectives
 Understanding Grouplab culture (to
some extent)

Ethnography and CSCW

Prominence of Ethnography in CSCW
Insufficient attention to social context
 New problems for design of
collaborative character of work and
activities
 Ethnography and system design

• Problem of scale
• Pressure of time
• Role of the ethnographer
Concurrent Ethnography
Design is influenced by on-going
ethnographic study
 Sequenced process

Ethnographic
Study
Debriefing Meetings
System Prototype
Systems
Development
Concurrent Ethnography in
action
London Air Traffic Control Centre
 Four week Ethnography
 Each stage of fieldwork was
intended to target designers issues
 Small research team
 What ethnography provided

Quick and Dirty
Ethnography



Brief Ethnographic Studies
Duration relative to the size of the task
Selecting aspects of work setting of
importance to design
Outline of project
Meetings
Short
Focus
Studies
Debriefing Meetings
Scoping Document
Quick and Dirty
Ethnography in Action
Ethnographic investigation of
software engineers
 Challenges of Large scale setting
 Working in Industrialized
Environments
 Acceptance into the setting (*Key to
Ethnographic research)

Evaluative Ethnography

Ethnography used to verify
formulated design decisions
Initial outline
Design or
Specification
Short
Ethnographic
study
Debriefing Meetings
Amended Design
Or Specification
Evaluative Ethnography in
action
Fieldwork in Building Society
 Using research for IT developments
 Routine of work
 Finding what customers wanted
 Outlined limitations of model that
had been proposed

Re-examination of previous
studies
Previous studies are re-examined to
inform design
 Ethnography used for many decades
 Many studies related to work and
occupation
 Can be informative

Re-examination in action
Inform preliminary design of Shared
Object Service
 Using previous Ethnographic studies
on:

Social work, police work and invoice
processing in a multi-site fast food
company
 What common service should support

Summary of Ethnography
Understanding Ethnography
 Ethnography and CSCW
 Uses of Ethnography

Concurrent
 Quick and Dirty
 Evaluative
 Re-examination

Big Break Time 5 Minutes

Reconvene in 5 Minutes

Upcoming – Ethnomethodology
Understanding Ethnomethodology
 Ethnomethodology in CSCW

Ethnomethodology
Understanding Ethnomethodology
 Confusing Ethnography and
Ethnomethodology
 Ethnomethodology in system design
 Incorporation of Sociology and
Computer Science?
‘Technomethodology’

Understanding
Ethnomethodology
Ethnomethodology literally means
“People’s Methods”
 A Shift from ‘other’ Sociological
Methods
 Social Life is potentially Chaotic
 Social Actors
 Members methods for making sense

Understanding
Ethnomethodology
Garfinkel “Documentary Method”
 Example of Documentary Method
 Garfinkel “Indexicality”
 Disrupt Technique


Example in class
Understanding
Ethnomethodology
We can observe other members
methods of construction
 Development of Conversation
Analysis

Confusing Ethnography and
Ethnomethodology
Ethnography is a form of
investigative fieldwork
 Ethnography focuses on the
“Member’s Point of View”
 Ethnomethodology is a specific
analytical technique

Confusing Ethnography and
Ethnomethodology
Confusion arises because:
 Ethnomethodologist is likely to use
ethnographic techniques
 ‘Analytic mentality’-selection of
phenomena and topics for
investigation

Ethnomethodology in HCI
and CSCW
Observations of work activities and
interactions help design process
 Understanding temporal
organization of activities and
interactions and implications to
design

Learning from
Ethnomethodologists
Division of Labour
 Field Observation conducted by
ethnomethodologists
 Act as proxy for end users
 Hand off requirements to computer
science people

Ethnomethodology for
Critique and Design

Ethnomethodology has provided:
Critique of the design
 Failure to support the work
 Technology doesn’t allow people to
engage in their work
 Outlines organization of work and
communication in the real world

Two Paradoxes

Paradox of system design


Large scale activity
Paradox of technomethodology
Transformational nature of technology
 Analysis of practice not invention

Technomethodology
Develop a stance in which
ethnomethodology and computer
science play equally significant roles
 Foundational relationships

Technomethodology
Trying to exploit generalizations
from ethnomethodology
 Abstractions from both disciplines
 Means by which such working
practices arise
 Dialogical interfaces

Conclusion
Understanding Ethnomethodology
 Confusing Ethnography and
Ethnomethodology
 Ethnomethodology and system
design
 Technomethodology

Conclusion of Presentation
Overview of Evaluation Techniques
used in CSCW
 Ethnography
 Ethnomethodology
 Technomethodology

My thoughts
CSCW has been unfortunately
caught in a battle between
sociological techniques
 CSCW design should incorporate
work context issues

It was this clear….