Methodology Matters: Doing Research
in the Social and Behavioral Sciences
Joseph E. McGrath
Gary Suh
Vesna Memisevic
1
Fall 2007
Informatics 231 Human-Computer
Interaction
Outline
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
2
Introduction
Some Basic Features of the Research Process
Substantive Domain
Conceptual Domain
Methodological Domain
Research Methods as Opportunities and Limitations
Research Strategies: Choosing a Setting For a Study
Quadrant I: The Field Strategies
Quadrant II: The Experimental Strategies
Quadrant III: The Respondent Strategies
Quadrant IV: The Theoretical Strategies
Some Strategic Issues
Fall 2007
Informatics 231 Human-Computer
Interaction
Outline
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
3
Study design
Comparison techniques
Baserates
The Correlation Question
The Difference Question
Randomization and “True experiments”
Sampling, allocation and statistical inference
Validity of findings
Potential classes of measures in social psychology
Strengths and Weaknesses of Types of Measures
Techniques for Manipulating Variables
Selection, direct intervention, induction
Conclusion
Fall 2007
Informatics 231 Human-Computer
Interaction
Introduction


4
Doing research – the systematic use of some set of
theoretical and empirical tools to try to increase our
understanding of some set of phenomena or events
Some of the tools with which
researchers in the social and
behavioral sciences go about
doing research (strategy,
tactics, and operations)
Fall 2007
Informatics 231 Human-Computer
Interaction
Some Basic Features of the Research
Process
Always involves bringing together three sets of
things:
(a) some content that is of interest,
(b) some ideas that give meaning to that
content, and
(c) some techniques or procedures by means
of which those ideas and content can be
studied.
5
Fall 2007
Informatics 231 Human-Computer
Interaction
Some Basic Features of the Research
Process
These three sets of things more formally, as three
distinct, though interrelated domains:
(a) The Substantive domain, from which we draw
contents that seem worthy of our study and
attention;
(b) The Conceptual domain, from which we draw
ideas that seem likely to give meaning to our
results; and
(c) The Methodological domain, from which we draw
techniques that seem useful in conducting that
research.
6
Fall 2007
Informatics 231 Human-Computer
Interaction
Some Basic Features of the Research
Process
7
Fall 2007
Informatics 231 Human-Computer
Interaction
Substantive Domain


8
Phenomena and Patterns of phenomena are
the object of our study
The phenomena of interest involve the states
and actions of some human systems and the
conditions and processes that give rise to
and follow from those states and actions.
Fall 2007
Informatics 231 Human-Computer
Interaction
Substantive Domain

9
Example – an individual
casting a vote in a
county election
Fall 2007
Informatics 231 Human-Computer
Interaction
Conceptual Domain



10
Properties of the states and actions of those
human systems that are the focus of study
Relations refer to any of a variety of possible
ways in which two or more elements can be
connected
Examples – causal connections, logical
relations, chronological relations
Fall 2007
Informatics 231 Human-Computer
Interaction
Conceptual Domain

11
Examples – attitude,
cohesiveness, power, social
pressure, status
Fall 2007
Informatics 231 Human-Computer
Interaction
Methodological Domain
Basic sets of elements or “tools” by which
social and behavioral scientists
systematically gather empirical information:
 Measuring
 Manipulating
 Controlling
 Distributing Impact
12
Fall 2007
Informatics 231 Human-Computer
Interaction
Methodological Domain

Measuring
–
–
13
For assessing the state or magnitude of some property of
some actors-behaving-in-context so that the researcher can
determine what value or level that feature has for each
“case” to be studied
Examples – questionnaire, rating scale, personality test,
instruments for observing and recording communications,
techniques for assessing the quality of some products
resulting from individual or group task performance
Fall 2007
Informatics 231 Human-Computer
Interaction
Methodological Domain

Manipulating
–
–
14
Techniques for manipulating some property of an
actor-behavior-context
Experimental manipulation – making a feature
have one particular predetermined value or level
for certain “cases” to be studied and another
specific preordained value or level for certain
other “cases,” so that the effect of differences in
that property can be assessed by comparing
those two sets of “cases”
Fall 2007
Informatics 231 Human-Computer
Interaction
Methodological Domain

Manipulating
–
–
15
Social psychologists have tried to manipulate features of the
systems they study by a number of techniques, such as:
(a) giving instruction to participants
(b) imposing constraints on features of the environment
(c) selecting materials for use
(d) giving feedback about prior performances
(e) using experimental confederates
(More is said about techniques for manipulating variables
near the end of this chapter)
Fall 2007
Informatics 231 Human-Computer
Interaction
Methodological Domain

Controlling
–
–
A set of techniques for controlling the impact of features that
are important but that you are not going to measure or
manipulate in a particular study
These include:



16
Techniques for experimental control – you make certain
features take the same predetermined value for all cases in
the study
Techniques for statistical control – you try to nullify the effects
of variations in a given property within a study by “removing”
those variations by statistical means
Techniques for distributing the impact – so that such impact
can be taken into account in interpretation of results
Fall 2007
Informatics 231 Human-Computer
Interaction
Methodological Domain

Distributing Impact
–
–
17
Techniques for distributing the impact of a
number of features of the system and its contextwithout directly manipulating or controlling any
one of them-so that such impact can be taken into
account in interpretation of results
Randomization – the most prominent means;
refers to procedures for the allocation of “cases”
among various conditions within the study
Fall 2007
Informatics 231 Human-Computer
Interaction
Research Methods as Opportunities
and Limitations
(a)
(b)
(c)
(d)
18
Methods enable but also limit evidence.
All methods are valuable, but all have weaknesses
or limitations.
You can offset the different weaknesses of various
methods by using multiple methods.
You can choose such multiple methods so that they
have patterned diversity; that is so that strengths of
some methods offset weaknesses of others.
Fall 2007
Informatics 231 Human-Computer
Interaction
Research Methods as Opportunities
and Limitations
The fundamental principle, in behavioral and
social science is that credible empirical
knowledge requires consistency or
convergence of evidence across studies
based on different methods.
19
Fall 2007
Informatics 231 Human-Computer
Interaction
Research Strategies: Choosing a
Setting For a Study

20
When you gather a batch of research evidence, you
are always trying to maximize three desireable
features or criteria:
A. Generalizability of the evidence over the
populations of Actors.
B. Precision of the measurement of the behaviors
that are being studied (and precision of control over
extraneous factors that are not being studied).
C. Realism of the situation or Context within which
the evidence is gathered, in relation to the contexts
to which you want your evidence to apply.
Fall 2007
Informatics 231 Human-Computer
Interaction
Research Strategies: Choosing a
Setting For a Study


21
Although you always want to maximize all
three of these criteria, A, B and C
simultaneously, you cannot do so.
This is the fundamental dilemma of the
research process.
Fall 2007
Informatics 231 Human-Computer
Interaction
Research Strategies: Choosing a
Setting For a Study
22
Fall 2007
Informatics 231 Human-Computer
Interaction
Quadrant I: The Field Strategies
23
Fall 2007
Informatics 231 Human-Computer
Interaction
Quadrant I: The Field Strategies



24
The two research strategies in quadrant I are the
Field Study and the Field Experiment.
Field study – the researcher sets out to make direct
observations of “natural”, ongoing systems, while
intruding on and disturbing those systems as little as
possible.
Field experiment – also works within an ongoing
natural system as unobtrusively as possible, except
for intruding on that system by manipulating one
major feature of that system.
Fall 2007
Informatics 231 Human-Computer
Interaction
Quadrant I: The Field Strategies

25
The essence of both of the strategies in
quadrant I, the field study and the field
experiment, is that the behavior system
under study is “natural”, in the sense that it
would occur whether or not the researcher
were there and whether or not it were being
observed as part of a study.
Fall 2007
Informatics 231 Human-Computer
Interaction
Quadrant II: The Experimental
Strategies
26
Fall 2007
Informatics 231 Human-Computer
Interaction
Quadrant II: The Experimental
Strategies


27
Laboratory experiment – the investigator deliberately
concocts a situation or behavior setting or context,
defines the rules for its operation, and then induces
some individuals or groups to enter the concocted
system and engage in the behaviors called for by its
rules and circumstances.
Experimental simulation – the researcher attempts to
achieve much of the precision and control of the
laboratory experiment but to gain some of the
realism (or apparent realism) of field studies.
Fall 2007
Informatics 231 Human-Computer
Interaction
Quadrant II: The Experimental
Strategies


28
The two strategies in Quadrant II, in contrast
to those of Quadrant I, involve concocted
rather than natural settings.
The laboratory experiment and the
experimental simulation are strategies that
involve “actor-behavior-context” systems that
would not exist at all were it not for the
researcher’s interest in doing the study.
Fall 2007
Informatics 231 Human-Computer
Interaction
Quadrant III: The Respondent
Strategies
29
Fall 2007
Informatics 231 Human-Computer
Interaction
Quadrant III: The Respondent
Strategies


30
Sample survey – the investigator tries to
obtain evidence that will permit him or her to
estimate the distribution of some variables,
and/or some relationships among them,
within a specified population
Examples – public opinion surveys on voting
intentions, political preferences, buying
intentions
Fall 2007
Informatics 231 Human-Computer
Interaction
Quadrant III: The Respondent
Strategies


31
The strategies of Quadrant III concentrate on the
systematic gathering of responses of the participants
to questions or stimuli formulated by the
experimenter, in contrast to the observation of
behaviors of the participants within an ongoing
behavior system
Studies are usually done under “neutral” conditions
of room temperature, lighting, chair comfort to nullify
any effects of the behavior setting or context on the
judgments that are the topic of study.
Fall 2007
Informatics 231 Human-Computer
Interaction
Quadrant IV: The Theoretical
Strategies
32
Fall 2007
Informatics 231 Human-Computer
Interaction
Quadrant IV: The Theoretical
Strategies


33
Formal theory – the researcher focuses on
formulating general relations among a
number of variables of interest
Computer simulation – a complete and
closed system that models the operation of
the concrete system without any behavior by
any system participants
Fall 2007
Informatics 231 Human-Computer
Interaction
Quadrant IV: The Theoretical
Strategies


34
The inclusion of these two strategies reminds
us of the importance of the theoretical side of
the research process.
One of the more powerful general strategies
for research is the simultaneous use of one
of the theoretical strategies (say, the
formulation of a general theory) and one of
the empirical strategies (for example, a
laboratory experiment).
Fall 2007
Informatics 231 Human-Computer
Interaction
Some Strategic Issues
1.
2.
35
Does the material, as presented, properly reckon
with the strengths and weaknesses of the research
strategies it encompasses?
To what extent is the research evidence on each
problem or topic based on use of only a single
research strategy, and therefore limited by the
weaknesses of that strategy; and to what extent is
that body of evidence based on use of multiple,
complementary strategies, with agreement or
convergence among the findings attained via the
different strategies?
Fall 2007
Informatics 231 Human-Computer
Interaction
Study design
1.
2.
3.
36
Gathering observations
Aggregation and partitioning
Comparison on the data set
Fall 2007
Informatics 231 Human-Computer
Interaction
Study design

Comparison depends on:




37
What’s included in the study (what phenomena, what
properties, what model of treatment for variables)
What system works (which substantive system we
study, paradigms, strategies)
What conceptual relations have been posted (which
properties are linked)
What comparison techniques are available
Fall 2007
Informatics 231 Human-Computer
Interaction
Comparison Techniques

Three base forms:



38
Baserates (how often?)
Correlations (are properties related;
do they occur together?)
Differences (comparison or
difference)
Fall 2007
Informatics 231 Human-Computer
Interaction
Base rates


39
How often Y occurs in the general case, as a basis for deciding
whether the rate of Y in some particular case is “notably” high
or low
Problems - difference in interpretation - various political,
economic and social issues
Fall 2007
Informatics 231 Human-Computer
Interaction
The Correlation Question




40
Is there covariation in the values of two properties or features of
system?
Correlations: High – Low; Positive – Negative; Zero
Linear or nonlinear relation between two or more variables
Can asses conceptual relations that imply covariation between two or
more variables, but cannot asses any conceptual relations that are
causal in their implications
Fall 2007
Informatics 231 Human-Computer
Interaction
The Difference Question




41
Whether Y is present under condition where X is present and whether
Y is absent when X is absent?
Samples are separated in two groups, one with members which have
performance X, and another with members who don’t have
performance X.
Some set of tasks is given to both of groups, and average task
performance is compared for both groups.
Groups must be comparable on factors that might affect task
performances!
Informatics 231 Human-Computer
Interaction
Randomization and True Experiments



42
Randomization = using random
assignment procedure to allocate cases to
conditions (in order to remove artifacts and
observations that happen by chance)
In order for the study to be called a “true
experiment”, study must include some
randomization
Random allocation procedure doesn’t
guarantee an equal distribution of any of
the potential factors among the conditions
being compared
Fall 2007
Informatics 231 Human-Computer
Interaction
Randomization and True Experiments

43
Possible problems:
–
Reduce the scope of study, as some variables
are hold constant, and therefore experimental
variables will occur only at a few levels
–
Reduce the realism of context of your study,
designed the tasks serves us - not the
participants' - purposes.
Fall 2007
Informatics 231 Human-Computer
Interaction
Sampling, Allocation and Statistical
Inference


44
The way we choose cases which will be included in our study
(from larger population of potential cases) effect credibility of
the evidence resulting from the study
How to choose right nature of the “random sample” population?
Fall 2007
Informatics 231 Human-Computer
Interaction
Sampling, Allocation and Statistical
Inference

45
We need to use
random sample in that
way that we can apply
results of the study to
the population of which
cases constitute a
random sample.
Fall 2007
Informatics 231 Human-Computer
Interaction
Sampling, Allocation and Statistical
Inference



46
Sampling case procedure: which cases from larger population
will be included in study.
Allocation case procedure: which condition every given cases
(already selected as a part of the study) will be assigned to
Random refers to procedure, not outcome!
Fall 2007
Informatics 231 Human-Computer
Interaction
Sampling, Allocation and Statistical
Inference



47
Size of the samples?
The larger the number the more distributed those
cases will approach the idealized random distribution
Uneven distribution doesn’t occur often if only chance
is operating.
Fall 2007
Informatics 231 Human-Computer
Interaction
Validity of Findings

Four different types of
validity (Cook & Campbell):




48
Statistical conclusion validity
Internal validity
Construct validity
External validity
Fall 2007
Informatics 231 Human-Computer
Interaction
Validity of Findings

Statistical conclusion validity:
–
–

Internal validity:
–
–
–
49
Difference arisen just by chance?
Relationship between cause and effect
How close can you come to
asserting that the present of X
caused the altered level of Y values?
Difference in Y associated with a
difference in X does not necessarily
imply a causal role for X
Are there other factors which may
have been covary with X and they, rather
than X, might have produced the change in Y
Fall 2007
Informatics 231 Human-Computer
Interaction
Validity of Findings

Construct validity:
–
–
–

External validity:
–
–
50
How well defined are the theoretical ideas in our study?
How clearly understood are the conceptual relations
being explored?
How well are mapping of concepts and relations
How confident you can be that your
findings will hold upon replication or
how general are your our findings.
How confident you can be that
you can make predictions ?
Fall 2007
Informatics 231 Human-Computer
Interaction
Potential Classes of Measures In
Social Psychology


Different techniques to measure presence or values of the
specific features of the human system
Participants awareness:



Natural or unnatural behavior
Reactivity of measures
Who makes which record of behavior:

Participants



Investigators


Observations (visible or hidden observer)
Third party (for non research purposes)

51
Knowingly make (self reports)
Unwittingly make (trace measures)
Archival records (private or public behavior)
Informatics 231 Human-Computer
Fall 2007
Interaction
Strengths and Weaknesses of
Types of Measures
Self reports
Observations by
visible observer
Observations by
hidden observer
Archival records
of public behavior
Archival
records of
private behavior
Trace measures
Reactivity
High potential
(Some participants are aware that
their response will be recorded, that
may influence how they response –
they might try to make good
impression, to give socially desirable
answers or to try to help investigation)
High
Low
High
Low
None
Versatility
Versatile to potential contents and to
the population to which they would
apply to
Can only be used on
overt behavior (not
feelings,
expectations, etc)
Med-Low
Low
Low
Low (many
concepts not
available)
Costs
Low
High
High
Medium
Medium
High
(sometimes)
Dross Rates
(discarded
info.)
Low
High (per observer
hour bases)
High(per observer
hour bases)
High
High
Very high (links
to concept
sometimes very
week)
Time
Low
High
High
Medium
Medium
High
Vulnerable to
observer error
Ethical concerns,
Vulnerable to
observer error
No possibility to
cross validate
No possibility to
cross validate
Errors
52
Fall 2007
Informatics 231 Human-Computer
Interaction
Techniques for Manipulating
Variables


Techniques for manipulating variables are
not very well specified
Three classes of techniques:




53
Selection
Direct intervention
Inductions
Manipulating variables are use to create an
experimental manipulation of features of a
situation
Fall 2007
Informatics 231 Human-Computer
Interaction
Selection


54
Advantage: Selection is the most convenient means to make
sure that all cases of a given conditions are alike on a certain
variable and that all cases of another comparison condition
differ o that variable
Disadvantage: conclusion drawn is more or less unreliable
unless all other potential factors are removed (uncertainty in
nature of variable manipulated (additional meaning))
Fall 2007
Informatics 231 Human-Computer
Interaction
Direct intervention

Advantages:
–
–
–
–

Disadvantages:
–
–
55
Surest way of achieving a definite and specific variable manipulation
Permits random allocation (distributing impact of other factors and
variables, that we are not studying)
Nor likely to be costly or time consuming
Low dross rate
Applicable only for relatively overt
and tangible variables
May suffer from reactivity effects
and experimental demands
Fall 2007
Informatics 231 Human-Computer
Interaction
Induction
Three major forms

1.
2.
3.

Advantages:
–
–

Fairy versatile in context of variables
Normally low in cost and time
Disadvantages:
–
–
–
56
Use of misleading instruction to the participants
Use of false feed back
Use of experimental confederates
Involves deception (raises ethical concerns)
Risk of detection
Risk of experimental demands
Fall 2007
Informatics 231 Human-Computer
Interaction
Conclusion



57
Results depend on methods. All methods
have limitations. Hence, any set of results is
limited.
It is not possible to maximize all desirable
features of method in any one study;
tradeoffs and dilemmas are involved.
Each study (each set of results) must be
interpreted in relation to other evidence
bearing on the same questions.
Fall 2007
Informatics 231 Human-Computer
Interaction
Questions?
58
Fall 2007
Informatics 231 Human-Computer
Interaction