i
INF 397C
Introduction to Research in Library and
Information Science
Fall, 2003
Day 5
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
1
The Scientific Method
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
2
i
• More than anything else, scientists are
skeptical.
• P. 28: Scientific skepticism is a gullible
public’s defense against charlatans and
others who would sell them ineffective
medicines and cures, impossible
schemes to get rich, and supernatural
explanations for natural phenomena.”
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
3
Research Methods
i
S, Z, & Z, Chapters 1, 2, 3, 7, 8
Researchers are . . .
- like detectives – gather evidence, develop a
theory.
- Like judges – decide if evidence meets
scientific standards.
- Like juries – decide if evidence is “beyond a
reasonable doubt.”
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
4
Science . . .
i
• . . . Is a cumulative affair. Current
research builds on previous research.
• Scientific Method:
– Empirical (acquires new knowledge via
direct observation and experimentation)
– Systematic, controlled observations.
– Unbiased, objective.
– Operational definitions.
– Valid, reliable, testable, critical, skeptical.
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
5
CONTROL
i
• . . . Is the essential ingredient of science,
distinguishing it from nonscientific
procedures.
• The scientist, the experimenter,
manipulates the Independent Variable
(IV – “treatment – at least two levels –
“experimental and control conditions”)
and controls other variables.
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
6
More control
i
• After manipulating the IV (because the
experimenter is independent – he/she
decides what to do) . . .
• He/she measures the effect on the
Dependent Variable (what is measured –
it depends on the IV).
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
7
Key Distinction
i
• IV vs. Individual Differences variable
• The scientist MANIPULATES an IV, but
SELECTS an Individual Differences
variable (or “subject” variable).
• Can’t manipulate a subject variable.
– “Select a sample. Have half of ‘em get a
divorce.”
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
8
Operational Definitions
i
• Explains a concept solely in terms of the
operations used to produce and measure it.
–
–
–
–
–
–
Bad: “Smart people.”
Good: “People with an IQ over 120.”
Bad: “People with long index fingers.”
Good: “People with index fingers at least 7.2 cm.”
Bad: Ugly guys.
Good: “Guys rated as ‘ugly’ by at least 50% of the
respondents.”
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
9
Validity and Reliability
i
• Validity: the “truthfulness” of a measure. Are
you really measuring what you claim to
measure? “The validity of a measure is
supported to the extent that people do as well
on it as they do on independent measures that
are presumed to measure the same concept.”
• Reliability: a measure’s consistency.
• A measure can be reliable without being valid,
but not vice versa.
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
10
Theory and Hypothesis
i
• Theory: a logically organized set of
propositions (claims, statements, assertions)
that serves to define events (concepts),
describe relationships among these events,
and explain their occurrence.
– Theories organize our knowledge and guide our
research
• Hypothesis: A tentative explanation.
– A scientific hypothesis is TESTABLE.
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
11
Goals of Scientific Method
i
• Description
– Nomothetic approach – establish broad generalizations and
general laws that apply to a diverse population
– Versus idiographic approach – interested in the individual,
their uniqueness (e.g., case studies)
• Prediction
– Correlational study – when scores on one variable can be
used to predict scores on a second variable. (Doesn’t
necessarily tell you “why.”)
• Understanding – con’t. on next page
• Creating change
– Applied research
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
12
Understanding
i
• Three important conditions for making a
causal inference:
– Covariation of events. (IV changes, and the
DV changes.)
– A time-order relationship. (First the scientist
changes the IV – then there’s a change in
the DV.)
– The elimination of plausible alternative
causes.
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
13
Confounding
i
• When two potentially effective IVs are
allowed to covary simultaneously.
• Poor control!
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
14
Intervening Variables
i
• Link the IV and the DV, and are used to
explain why they are connected.
• Here’s an interesting question: WHY did
the authors put this HERE in the
chapter?
– Because intervening variables are important
in theories.
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
15
A bit more about theories
i
• Good theories provide “precision of
prediction”
• The “rule of parsimony” is followed
– The simplest alternative explanations are
accepted
• A good scientific theory passes the most
rigorous tests
• Testing will be more informative when
you try to DISPROVE (falsify) a theory
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
16
Populations and Samples
i
• Population: the set of all cases of
interest
• Sample: Subset of all the population that
we choose to study.
Population
Parameters
Sample
Statistics
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
17
Ch. 3 -- Ethics
i
• Read the chapter.
• Understand informed consent, p. 57 – a person’s
expressed willingness to participate in a research
project, based on a clear understanding of the nature
of the research, the consequences of declining, and
other factors that might influence the decision.
• Odd quote, p. 69 – Debriefing should be informal and
indirect.
• Know that UT has an IRB:
http://www.utexas.edu/research/rsc/humanresearch/
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
18
Ch. 7 – Independent Groups
Design
i
• Description and Prediction are crucial to the
scientific study of behavior, but they’re not
sufficient for understanding the causes. We
need to know WHY.
• Best way to answer this question is with the
experimental method.
• “The special strength of the experimental
method is that it is especially effective for
establishing cause-and-effect relationships.”
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
19
Good Paragraph
i
• P. 196, para. 2 – Discusses how
experimental methods and descriptive
methods aren’t all THAT different – well,
they’re different, but related. And often
used together.
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
20
Good page – P. 197
i
• Why we conduct experiments
• If results of an experiment (a well-run
experiment!) are consistent with theory,
we say we’ve supported the theory.
(NOT that it is “right.”)
• Otherwise, we modify the theory.
• Testing hypotheses and revising theories
based on the outcomes of experiments –
the long process of science.
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
21
Logic of Experimental Research
i
• Researchers manipulate an independent
variable in an experiment to observe the
effect on behavior, as assessed by the
dependent variable.
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
22
Independent Groups Design
i
• Each group represents a different
condition as defined by the independent
variable.
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
23
Random . . .
i
• Random Selection vs. Random Assignment
– Random Selection = every member of the
population has an equal chance of being selected
for the sample.
– Random Assignment = every member of the
sample has an equal chance of being placed in the
experimental group or the control group.
• Random assignment allows for individual differences
among test participants to be averaged out.
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
24
Let’s step back a minute
i
• An experiment is personkind’s way of
asking nature a question.
• I want to know if one variable (factor,
event, thing) has an effect on another
variable – does the IV influence the DV?
• I manipulate some variables (IVs),
control other variables, and count on
random selection to wash out the effects
of all the rest of the variables.
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
25
Block Randomization
i
• Another way to wash-out error variance.
• Assign subjects to blocks of subjects,
and have whole blocks see certain
conditions.
• (Very squirrelly description in the book.)
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
26
Challenges to Internal Validity
i
• Testing intact groups. (Why is the group a group?
Might be some systematic differences.)
• Extraneous variables. (Balance ‘em.) (E.g.,
experimenter).
• Subject loss
– Mechanical loss, OK.
– Select loss, not OK.
• Demand characteristics (cues and other info
participants pick up on) – use a placebo, and doubleblind procedure
• Experimenter effects – use double-blind procedure
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
27
Role of Data Analysis in Exps.
i
• Primary goal of data analysis is to
determine if our observations support a
claim about behavior. Is that difference
really different?
• We want to draw conclusions about
populations, not just the sample.
• Two ways – stat and replication.
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
28
Two methods of making
inferences
i
• Null hypothesis testing
– Assume IV has no effect on DV; differences we
obtain are just by chance (error variance)
– If the difference is unlikely enough to happen by
chance (and “enough” tends to be p < .05), then we
say there’s a true difference.
• Confidence intervals
– We compute a confidence interval for the “true”
population mean, from sample data. (95% level,
usually.)
– If two groups’ confidence intervals don’t overlap, we
say (we INFER) there’s a true difference.
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
29
What data can’t tell us
i
• Proper use of inferential statistics is NOT
the whole answer.
– Scientist could have done a trivial
experiment.
– Also, study could have been confounded.
– Also, could by chance find this difference.
(Type I and Type II errors – hit this for real in
week 5.)
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
30
This is HUGE.
i
• When we get a NONsignificant difference, or
when the confidence intervals DO overlap, we
do NOT say that we ACCEPT the null
hypothesis.
– Hinton, p. 37 – “On this evidence I accept the null
hypothesis and say that we have not found
evidence to support Peter’s view of hothousing.”
• We just cannot reject it at this time.
• We have insufficient evidence to infer an effect
of the IV on the DV.
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
31
Notice
i
• Many things influence how easy or hard
it is to discover a difference.
– How big the real difference is.
– How much variability there is in the
population distribution(s).
– How much error variance there is.
– Let’s talk about variance.
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
32
Sources of variance
i
• Systematic vs. Error
– Real differences
– Error variance
• What would happen to the standard deviation if our
measurement apparatus was a little inconsistent?
• There are OTHER sources of error variance, and the
whole point of experimental design is to try to minimize
‘em.
Get this: The more error variance, the harder for real
differences to “shine through.”
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
33
One way to reduce the error
variance
i
• Matched groups design
– If there’s some variable that you think MIGHT
cause some variance,
– Pre-test subjects on some matching test that
equates the groups on a dimension that is relevant
to the outcome of the experiment. (Must have a
good matching test.)
– Then assign matched groups. This way the groups
will be similar on this one important variable.
– STILL use random assignment WITHIN the groups.
– Good when there are a small number of possible
test subjects.
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
34
Another design
i
• Natural Groups design
– Based on subject (or individual differences)
variables.
– Selected, not manipulated.
– Remember: This will give us description,
and prediction, but not understanding
(cause and effect).
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
35
We’ve been talking about . . .
i
• Making two groups comparable, so that
the ONLY systematic difference is the IV.
– CONTROL some variables.
– Match on some.
– Use random selection to wash out the
effects of the others.
– What would be the best possible match for
one subject, or one group of subjects?
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
36
Themselves!
i
• When each test subject is his/her own
control, then that’s called a
– Repeated measures design, or a
– Within-subjects design.
(And the random groups design is called a
“between subjects” design.)
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
37
Repeated Measures
i
• If each subject serves as his/her own
control, then we don’t have to worry
about individual differences, across
experimental and control conditions.
• EXCEPT for newly introduced sources of
variance – order effects:
– Practice effects
– Fatigue effects
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
38
Counterbalancing
i
• ABBA
• Used to overcome order effects.
• Assumes practice/fatigue effects are
linear.
• Some incomplete counterbalancing
ideas are offered in the text.
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
39
Which method when?
i
• Some questions DO lend themselves to
repeated measures (within-subjects) design
– Can people read faster in condition A or condition
B?
– Is memorability improved if words are grouped in
this way or that?
• Some questions do NOT lend themselves to
repeated measures design
– Do these instructions help people solve a particular
puzzle?
– Does this drug reduce cholesterol?
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
40
Hinton typo
i
• P. 62, para. 1: “. . . population standard
deviation, µ, divided by . . . .”
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
41
Some questions we’d like to ask
Nature
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
42
Midterm
i
• Emphasize
–
–
–
–
How to lie with statistics – concepts
To know a fly – concepts
SZ&Z – Ch. 1, 2, 7, 8
Hinton – Ch. 1, 2, 3, 4, 5
• De-emphasize
– SZ&Z – Ch. 3
– Other readings
• Totally ignore for now
– SZ&Z – Ch. 14
– Hinton – Ch. 6, 7, 8
R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu
43