Qualitative
Data (collection,
analysis & presentation)
Dr. Anne Adams
Overview
•
Qualitative Data – how to collect it
•
Qualitative Analysis – how to do it
•
How to present qualitative data
•
Tools to support you
Qualitative / Quantitative Approach
Research is like fishing
Quantitative methods
• You find the best river for the fish you want,
you have one line, a specific bait for a
specific type of fish.
Qualitative methods
• You may want to catch tuna so you fish in
certain parts of the sea BUT on the whole
you throw your nets out to sea and catch
everything including the things you want and
don’t want.
Qualititative methods
• In-depth Interviews,
• Focus groups,
• Observational / ethnographic studies,
• Open ended data (system logs)
• Data transcribed
Documentation
 Course materials,
 Standards,
 Previous literature (conference /
journal, web pages)
 Internal reports
 External reports (e.g. HEFCE)
Research Diaries (Research Logs / Field Notes)
1) Research conducted & decisions made
2) Thought & creative research process
 Faraday’s ‘Field Notes’:
"The main interest of the Diary lies quite outside the range
of propositions and experimental proofs. It centres round
the methods of Faraday's attack, both in thought and in
experiment: it depends on the records of the workings of his
mind as he mastered each research in turn, and on his
attitude not only to his own researches but also to scientific
advance in general" (Bragg 1932: v).
Critical Incidents

Used in conjunction with other methods

Used as a data collection & analysis approach

Identify critical moment / factor
1.
Ask respondent to recall
2.
Observe, record interactions identify incidents
3.
Review documentation, logs identify incidents
Observational Diaries
1) Interaction patterns
2) Attitudinal changes (e.g. becoming
scared of the computer)
3) Behavioural changes
4) Sequencing (e.g. incremental changes
experienced over time)
Critical Incidents (observations)
Timeframe
Technology
interactions
Social
interactions
barriers
enablers
Clinician 1
10.01
Nurse 1
10.02
Receptionist 1
09.30
Recorded patient
arrival
Discussion
about
information
required for
appointment
All verbal
nothing
printed or
electronic
provided
Social
details
about the
consultant
provided
Patient 1
9.15
None
Discussion with
receptionist
All verbal
no further
information
provided
Key points
about
consultant
provided
A&E White-boards
– Collaboration & unconscious social cues
– Interaction roles (e.g. pen holders) & acceptable sharing
Long looks
Glances
Entering
and
Annotating
Penholder
Other Staff Total of
Whiteboard Whiteboard overall
Interactions Interactions observation
s
28%
48%
38%
26%
46%
51%
1%
39%
23%
Broome & Adams (’05) BCS HC, (’06) IMM
Hospital spaces & technology use
Neurologist
patient interaction
interaction pattern:
pattern:
consultant // patient
Piece of paper
from the diary
Paperbased diary
Computer
Desk
Computer
Window
Nurse A
Desk
Patient
Consultant
Consultant
Patient
Patient
Window
Nurse B
Nurse A
Desks
Wi
nd
ow
Computer
Examination
bed
Patient
Window
Consultant
Patient interactions
Examination bed
Clinician interactions
Adams & Blandford, HCI’08
EPSRC / ESRC / NHS
funded
Interviews
 Structure (semi-structured/ structured)
 Style (expert / novice)
 Setting (natural, office)
 Recording the data (audio – quotes, written notes
distract, phone interviews).
 Biasing – talking (<15%), your opinion (NO)
SEE: http://oro.open.ac.uk/11909/
Focus Groups
 6 or 7 participants
 Moderate to keep focus & obtain all perspectives
Questionnaires (open ended)
 Design (quantitative / qualitative)
 Purpose (obtain background info & recruit)
 Open not closed to allow ‘participants’ accounts of
experiences, feelings, observations
 E.G. :
 Give examples of using mobile devices for learning purposes
 Detail how you felt about those experiences
 Provide information about where you have seen others using
mobile devices for learning.
User Trials
– Test current applications with a sample of students.
– Experiments where you test one type of course
presentation with another.
– Video students completing a sample assessment /
piece of coursework to see where they actually get stuck
rather than where they think they get stuck.
– Video and observe interactions between students and /
or students and tutors.
Log Analysis
 First Class interactions
 Eluminate / flash-meeting logs
 Analyse (quantitatively and qualitatively)
 Interaction and usage patterns
 Tracking individuals
 Language used
Participant Video
 Gaining in popularity as a research method
 The OU Participatory Video Group
 Give participants video’s they capture data on
what is important with regard to your focus of
research.
 Can be as focused or as open-ended as you
want.
Qualitative Analysis Background
• Conversational / Discourse Analysis
• Thematic Analysis / Grounded Theory
• Content Analysis / Critical Incident Analysis
• counting
• imposing established frameworks
• “Both qualitative and quantitative approaches
share a common concern with theory as the
goal of research” (Henwood & Pidgeon, 1992
p.101)
Qualitative Analysis Methods
• Conversational / Discourse Analysis
• Thematic Analysis / Grounded Theory
• Content Analysis / Critical Incident Analysis
• counting
• imposing established frameworks
• “Both qualitative and quantitative approaches
share a common concern with theory as the
goal of research” (Henwood & Pidgeon, 1992
p.101)
QUANT / QUAL Comparison
Quantitative approaches
Qualitative approaches
'Simple' numeric data
'Complex' rich data
Measurement
Meaning
Explanation
Understanding
Prediction
Interpretation
Generalisable account
Contextual account
Representative population sample
Purposive/ representative
perspective sample
Hypothesis-testing
Exploratory
Claims objectivity
Accepts subjectivity
Closed system
(experimental control)
Open system
(ecological validity)
Qualitative Analysis Methods
• Conversational / Discourse Analysis
• Thematic Analysis / Grounded Theory
• Content Analysis / Critical Incident Analysis
• counting
• imposing established frameworks
“Both qualitative and quantitative approaches share
a common concern with theory as the goal of
research” (Henwood & Pidgeon, 1992 p.101)
TELLING STORIES through DATA
http://www.youtube.com/watch?v=jbkSRLYSojo
IBM Data Visualisations at ‘many eyes’:
http://www-958.ibm.com/software/data/cognos/manyeyes/
Details of GT approach used here found at:
http://oro.open.ac.uk/11911/
GT Application
•
•
Data in whatever form is :Broken down, conceptualised, and put
back together in new ways.
•
Analysis Stages – 3 levels of coding :



open,
axial,
selective (with process effects)
Open coding
1.
Concepts are identified.
2.
Concepts are grouped into
categories
3.
Properties and dimensions
of the category identified
Open coding: detailed
•
Concepts are:- Conceptual labels placed on
discrete happenings, events, and other instances
of phenomena
•
Categories are:- where concepts are classified
and grouped together under a higher order – a
more abstract concept called a category.
•
Properties are:- characteristics pertaining to a
category
•
Dimensions are:- Location (values) of properties
along a continuum
Open coding: example
• “ When I want to have a personal
conversation, I encrypt the
message. I think that makes the
email private. Stops people from
listening in”
Open coding: analysis
•
“ When I want to have a personal
conversation (private interaction), I encrypt
the message (security measure). I think that
makes the email private (Securing privacy).
Stops people from listening in
(Surveillance).”
•
Concepts are:- private interaction,
security measures, securing privacy,
surveillance
Categories are:- Interaction, privacy,
security
•
Open coding (5)
Category Class Property
Dimension Dimensional Range
surveillance
frequency
Being observed
scope
often ........never
more ........less
intensity
high.........low
duration
long .........short
Axial Coding (1)
• High level phenomena identified.
• Phenomena conditions identified (causal,
context, intervening).
• Phenomena action / interaction
strategies and consequences identified
Axial Coding (2)
• Phenomena are:- central ideas, events.
• CONDITIONS
• Causal conditions are:- events that lead to
occurrence or development of a phenomenon.
• Context:- The specific set of properties (and
location on a dimensional range) that pertain to a
phenomenon.
• Intervening conditions:- broader structural
context.
Axial Coding (3)
• “ When I want to have (context) a
personal conversation (phenomenon),
I encrypt the message (strategy). I
think that makes the email private
(consequence).”
Selective Coding (1)
• Define the core category & high-level story line.
• Relate subsidiary categories by its properties
• Relate categories at the dimensional level
• Iterative validation of relationships with data
• Identify category gaps
32
Selective Coding (2)
• Core category is: The central phenomenon
around which all the other categories are
integrated.
• Story is: A descriptive narrative about the central
phenomenon of the study.
• Story line is: The conceptualisation of the story the core category.
• Ways to represent this ‘STORY LINE’
(Conditional Matrix, Process Effects)
Process Effects
• Process is the linking of AI sequences over time
Action / Interaction
Strategy
CHANGE TO
CONDITIONS affecting A/I
CHANGING
CONDITIONS
CONSEQUENCES of
response
(over time)
RESPONSE
from A/I
Authentication
A
B
Perceived
Perceived
POOR
Security
HIGH
Security
L Perceived
O
threats
W
L Information
O
importance
W
Password disclosure
Adams & Sasse (’99) ACM Multimedia
35
Increased
Privacy and Process
IS
Users
IR
Contexts
Trust Privacy
secure
(based on
assumptions)
IU
Technology makes assumptions inaccurate
Emotive
Increased perceived privacy
invasions
Decreased organisational trust
36
Rejection
Initial Problems
• Lines between each type of coding are artificial
–Data presented at dimensional level
–Action / interactions & conditions present.
• “ I find computers always break down for me when
I have a lot of things to do. So I try not to use them
when I have a lot to do. Which slows everything
down a bit”
37
Solution
• Code both open and axially together
• Qualitative analysis tools
–NVivo
–Atlas TI
• Analyse without loosing the detail
38
Problems & Solutions
• P: Complex method to apply
• S: Ease up on yourself, take the best
approach for you …. Paper-based coding with
colour pens used by social-scientists
(immersing yourself in the data)
• P: Focus of research
• S: Data collection and analysis tightly
interwoven
39
Qualitative Analysis SUMMARY
• Powerful for appropriate issues
• Application Complex
• Rewarding – ‘Convincing
Theories’
40
Good Quality Research
•
Not Divide but to compliment
•
Exploratory (discovery) – reductionistic
(justification)
•
Henwood / Pidgeon – good quality
research
•
7 golden rules of good quality research
TOOLS
•Atlas Ti, NVivo
Atlas TI – multiple media
Can be used for:
• Textual data – transcripts in
ASCII or ANSI character code
table
• Graphical data – BMP, TIFF,
Kodak Photo CD
• Audio Data – WAV
Hermeneutic unit editor
• Primary document Pane
–document
–line number
–margin area
• HU editor components
–Main menu
–main tool bar
–drop-down list (prim. doc, quotes, codes,
memo)
–primary doc tool bar
NVivo
CONCLUSIONS