IS 4800 Empirical Research Methods
for Information Science
Class Notes January 13, 2012
Instructor: Prof. Carole Hafner, 446 WVH
hafner@ccs.neu.edu Tel: 617-373-5116
Course Web site: www.ccs.neu.edu/course/is4800sp12/
http://www.classictvads.com/smoke_1.shtml
http://www.buzzfeed.com/jpmoore/x-ridiculous-vintage-cigarette-ads
Classification schemes for research methods
1. Epistemological – what kind of claims can be made
■ Descriptive/Interpretive
■ Correlational/Relational
■ Causal
2. Methodological: how we go about it (process)
■
Quantitative vs. qualitative; survey, experiment, etc.
3. Level of Generality (by purpose)
■ Scientific research aims to discover general principles or
test a general theory
■ Evaluation research aims to support decision making or test
a hypothesis in a specific context
Evaluation Research:
Evaluating a product, solution, or method
■ Purpose: provide useful feedback about some application
or technology (how well it achieves its goals, its value or
impact, etc.)
■ Can use quantitative, qualitative or mixed methods
■ Supports a decision about what should be done
■ Best if using real-world (vs. laboratory) data
■ Pilot studies of systems or technologies enable
evaluation research
Results of Evaluation Research
■ Choose which of several products to purchase
■ Decide whether or not to go forward with full
deployment of a new system or process
■ Decide whether more or different training is needed
■ Choose which features of a product design should be
kept and which ones dropped
■ Decide whether a policy or the governance structure that
supports it needs to be changed
■ Decide whether a new or modified policy would be
effective
■ etc.
Steps in evaluation research
■ Formulation of research questions/hypotheses
■ Selection of research strategy (quant. or qual.)
■ The usual three elements:
■ Population to observe
■ Characteristics to observe or measure
■ How to analyze the data
■ Utilization: what will be done, depending on the
conclusions ?
(an important question about medical tests ??)
Quantitative Research Claims
• Quantitative research: the claims are supported by
objective data and (usually) statistics
• Descriptive studies – how are characteristics
distributed in a population ?
• Correlational studies – how are two characteristics
related?
• Inferential studies – do differences in one
characteristic lead to differences in another
characteristic?
Quantitative Research Claims: examples
■ People who use internet for research on medical issues ask more
(average of 3.5) questions when they visited their doctor, while
those who did not (average of 1.5 questions).
■ Employees at company ABC work faster with Word than with
WizziWord.
Qualitative Research Claims
■ Qualitative research: claims are supported by observations
and appeals to common sense (more “anecdotal”)
■ Descriptive: observations are organized into themes and patterns.
(Ex: Rogers theory of innovation diffusion).
■ Exploratory: May identify hypotheses for later quantitative
studies
■ Explanatory: offers the possibility of explaining why things
happen in addition to what happens
Each study by itself does not “prove” anything but if more and more
studies show the same thing, it can be convincing.
Qualitative Research Claims:
hypothetical example
■ Involvement in competitive sports among children:
■ When segregated by sex, there is a tendency for the boys to reject girls as
friends and playmates in non-sport activities
■ When boys and girls participate together, there is an greater friendship and
interaction between boys and girls in school and in other activities
Methodological classification
■ Quantitative methods (and type of claims)
■ Observational (descriptive and/or correlational)
■ Survey (descriptive and/or correlational)
■ Experiment (causal)
■ Quasi-experiment (weaker causal claims)
■ Qualitative methods
■ Ethnography (descriptive)
■ Case study (descriptive; explanatory)
■ Grounded theory (descriptive; exploratory)
■ Mixed methods
Quantitative Methods: Observational Studies
• A population is identified; measures are defined
• Data is collected by observing/recording, or from existing
databases or other artifacts
• Descriptive statistics and graphs reveal how the measures
are distributed
• Statistics may also reveal that there is a relationship
between measures
Quantitative Methods: Observational Studies
Example: Distribution by major of CCIS students
Number/proportion of CCIS students in a
combined major
Relationship of gender to the above
Conclusions:
42% of CCIS students choose combined major
The data do (do not) reveal a relationship between
gender and the choice of major
Quantitative Methods: Surveys
• A population and a sampling technique are identified
• A data collection instrument is created (usually a
questionnaire) with questions representing self-report
measures of interest.
• Members of the sample are contacted and asked to
answer the questions
• Descriptive statistics and graphs reveal how the data are
distributed
• Graphs or correlation statistics may reveal that there is a
relationship
Quantitative Methods: Surveys
Example: 100 randomly selected recent NU graduates are
asked to complete a survey about their post-graduation
employment
Conclusion: NU students with STEM degrees were more
likely to have a full-time job within 6 months of
graduating than students in other majors. (cannot claim
causality)
Quantitative Methods: Experiments
• A treatment variable (independent variable) is defined
with at least two different values (levels), and an outcome
measure (dependent variable) is also defined
• A criterion level of statistical significance is selected
• Participants are selected and assigned to treatment groups
(randomly or using some balancing method)
• The treatments are applied to the groups, and the
outcomes are observed
• Inferential statistics are used to determine if the data
satisfy the criterion to conclude that different treatment
levels lead to different outcomes.
Quantitative Methods: Quasi-experiments
• A treatment variable (independent variable) is defined
with at least two different values (levels), and an effect
measure (dependent variable) is also defined
• A criterion level of statistical significance is selected
• Existing groups of participants are selected that are
assumed or hoped to be comparable
• The treatments are applied to groups, and the outcomes
are observed
• Inferential statistics are used to determine if the data
satisfy the criterion to conclude that different treatment
levels lead to different outcomes.
Advantages/disadvantages of quantitative
methods
+: Systematic rules and procedures already worked
out, and can be followed
+: Traditional, accepted as “proof”
-: Closed-ended questions may lead to ignoring
important factors and relationships
-: Quantitative methods cannot handle phenomena
that are difficult to turn into variables
Qualitative Methods: Ethnography
• Purpose: Identify the social structures and behavior
patterns of a culture (subgroups and roles; how status
is achieved; how work gets done; distribution of
resources; how, when and why people communicate)
• Observations are made in the natural setting
• The researcher often joins the society to experience
directly how its members behave and interact
Qualitative Methods: Case Studies
• Purpose: Explore in depth an activity or project
• Purpose: Discover and tell a story with a beginning,
middle and end
• Multi-modal data collection (observation, interviews,
artifacts . . .)
• Identifies lessons learned that explain successful and
unsuccessful outcomes
• A frequently-used method for IT empirical research
Qualitative Methods: Case Studies
Example: A failed software development project is studied
to find out what lessons can be learned to avoid similar
results in future projects
21
Advantages/disadvantages of qualitative
methods
+: more innovative and creative
+: capable of addressing issues that do not lend
themselves to being described by variables
-: justification of conclusions from data is often
less clear
-: makes some physical scientists uncomfortable
22
Mixed Methods
Pragmatic philosophy – find out whatever you can
using whatever methods are possible
Involves both qualitative and quantitative elements
 (at least 2 stages of research)
Advantages/Disadvantages
+: combines structure and flexibility
-: requires more time and resources
Examples of Mixed Method Designs
Pattern 1: quantitative study followed by indepth interviews to get insight on the reasons
for the observed relationships. Typical in
usability research.
Pattern 2: exploratory qualitative study
followed by a survey or experiment to confirm
and generalize the results
24
Collecting data for quantitative studies
Quantitative research skills
Identify/define the variables (and their coding)
Design the “instrument” – a measurement
process or technique
Types of instruments:
a questionnaire (paper or Web-based)
logs of “click” data
form for observer to fill in (experiment or field study)
Field testing/validating instruments is part of quantitative
methods
Real-world vs. laboratory data collection is a key distinction
Collecting data for qualitative studies
Qualitative research skills
Careful observation, being a good listener
Gaining the trust of informants
Linguistic analysis of documents and conversations
Ability to recognize patterns in social behavior and in
documents
Types of instruments:
Paper and pencil – making good field notes
Audio and video recording
Contemporaneous vs. retrospective information gathering is
an important distinction
V. Purposes and Products
■ Reasons for doing empirical research
■ Disconfirm or support a theory
■ Solve a design or technical problem
■ Evaluate a product or solution
■ Inform practice (develop “best practices”)
■ Understand why people do what they do
■ Predict the future
■ Contribute to a better world
■ Advance personal goals
V. Purposes and Products
■ Products of Empirical Research
■ Evidence of a relationship
■ Disconfirm a theory (or some aspect of it)
■ Comparison of existing products or techniques
■ A new or improved product or technique
■ In-depth study of a particular outcome
■ Explanation of unanticipated outcomes
■ A new theory
■ A new improved research tool or technique
■ A critical analysis (pros and cons)