Research Methods in
Human-Computer Interaction
Chapter 11Analyzing
Qualitative Data
©2010 John Wiley and Sons
www.wileyeurope.com/college/lazar
Chapter 11
Overview
•
•
•
•
•
•
Introduction
Stages of qualitative analysis
Grounded theory
Content analysis
Analyzing text content
Analyzing multimedia content
©2010 John Wiley and Sons
www.wileyeurope.com/college/lazar
Chapter 11
Stages of qualitative analysis
• Identify components of the substance
• Study properties and dimensions of each
component
• Understand and make inference about the
substance
©2010 John Wiley and Sons
www.wileyeurope.com/college/lazar
Chapter 11
Grounded theory
• An inductive research method
©2010 John Wiley and Sons
www.wileyeurope.com/college/lazar
Chapter 11
Procedures of grounded theory
•
•
•
•
open coding
development of concepts
grouping concepts into categories
formation of a theory
©2010 John Wiley and Sons
www.wileyeurope.com/college/lazar
Chapter 11
Grounded theory
• Advantages
– a systematic approach to analyzing qualitative,
mostly text-based, data,
– generating theory out of qualitative data that can
be backed up by ample evidence of the coding
– Interplay between data collection and analysis
©2010 John Wiley and Sons
www.wileyeurope.com/college/lazar
Chapter 11
Grounded theory
• Disadvantages
– Researcher can be overwhelmed by the details of
the data
– The theory generated is hard to evaluate
– Findings may be subject to bias
©2010 John Wiley and Sons
www.wileyeurope.com/college/lazar
Chapter 11
Content analysis
• A more specific view: a systematic, replicable
technique for compressing many words of text
into fewer content categories based on explicit
rules of coding
• A broader view: any technique for making
inferences by objectively and systematically
identifying specified characteristics of
messages
©2010 John Wiley and Sons
www.wileyeurope.com/college/lazar
Chapter 11
Content categories
©2010 John Wiley and Sons
www.wileyeurope.com/college/lazar
Chapter 11
Preparing for content analysis
•
•
•
•
Define the data set
Define the population
Clean up the data
Understand the context of the data
©2010 John Wiley and Sons
www.wileyeurope.com/college/lazar
Chapter 11
Analyzing text data
• A priori coding
– Identify coding categories
– Coding
– Reliability check
• Emergent coding
–
–
–
–
–
–
Multiple coders identify coding categories based on subset of data
Consolidate category list
Code a subset of data
Reliability check
Repeat the process until satisfactory result is met
Code the rest of the data
©2010 John Wiley and Sons
www.wileyeurope.com/college/lazar
Chapter 11
Identify coding categories
•
•
•
•
Theoretical framework
Researcher-denoted concepts
In-vivo codes
Building a code structure (nomenclature)
©2010 John Wiley and Sons
www.wileyeurope.com/college/lazar
Chapter 11
Coding the text
• Look for key items
©2010 John Wiley and Sons
www.wileyeurope.com/college/lazar
Chapter 11
Coding the text
• Ask questions about the data
• Making comparisons
– Between different coding category
– Between different participant group
– Between existing data and previous literature
• Using computer software
©2010 John Wiley and Sons
www.wileyeurope.com/college/lazar
Chapter 11
Ensure high quality analysis
• Validity
– constructing a multi-faceted argument in favor of
your interpretation of the data
– Constructing a database
– Data source triangulation
– Interpretation should account for as much as
possible of the data
– Alternative interpretations may also help
©2010 John Wiley and Sons
www.wileyeurope.com/college/lazar
Chapter 11
Ensure high quality analysis
• Reliability check
– Stability
• also called intra-coder reliability
• examines whether the same coder rates the data in the
same way throughout the coding process
– Reproducibility
• also called inter-coder reliability or investigator
triangulation
• examines whether different coders code the same data
in a consistent way
©2010 John Wiley and Sons
www.wileyeurope.com/college/lazar
Chapter 11
Reliability measures
• Percent agreement
Number of cases coded the
same way
% Agreement =
Total number of cases
• Cohen’s Kappa:
K = (Pa − Pc)/(1 − Pc)
©2010 John Wiley and Sons
www.wileyeurope.com/college/lazar
Chapter 11
Reliability check
• Agreement matrix
©2010 John Wiley and Sons
www.wileyeurope.com/college/lazar
Chapter 11
Reliability check
• Interpretation of Cohen’s Kappa
©2010 John Wiley and Sons
www.wileyeurope.com/college/lazar
Chapter 11
Subjective vs. objective coder
• Subjective coders
– Knowledge and experience can help interpret the
data
– Less training required
– May cause inflated reliability
• Objective coders
– Less likely to cause inflated reliability
– Lack of knowledge affect the ability to understand
the data
– More training required
©2010 John Wiley and Sons
www.wileyeurope.com/college/lazar
Chapter 11
Analyzing multimedia content
• The supporting techniques and methods are less mature
compared to text data analysis
• Approaches:
– Manual analysis:
• highly labor intensive and time consuming
• More accurate
– Completely automated analysis:
• Faster, less amount of work
• Highly inaccurate
– Partially automated approach
• Combines the advantages of the manual process and the
completely automated process
©2010 John Wiley and Sons
www.wileyeurope.com/college/lazar
Chapter 11
End-of-chapter
• Summary
• Discussion questions
• Research design exercise
©2010 John Wiley and Sons
www.wileyeurope.com/college/lazar
Chapter 11