i
INF 397C
Introduction to Research in Library and
Information Science
Fall, 2009
Day 5
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
1
3 things today
i
1. Y’all teach me what Dr. Rice Lively said
2. The Scientific Method
3. An exercise to drive home experimental
design
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
2
More than anything else . . .
i
• . . . scientists are skeptical.
• P. 15 – “Scientific skepticism is a gullible
public’s defense against charlatans and
others who would sell them ineffective
medicines and cures, impossible
schemes to get rich, and supernatural
explanations for natural phenomena.”
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
3
Research Methods
i
S, Z, & Z, Chapters 1, 2, 3, 7, 8
Researchers are . . .
- like detectives – gather evidence, develop a
theory.
- Like judges – decide if evidence meets
scientific standards.
- Like juries – decide if evidence is “beyond a
reasonable doubt.”
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
4
Science . . .
i
• . . . Is a cumulative affair. Current
research builds on previous research.
• The Scientific Method:
– is Empirical (acquires new knowledge via
direct observation and experimentation)
– entails Systematic, controlled observations.
– is unbiased, objective.
– entails operational definitions.
– is valid, reliable, testable, critical, skeptical.
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
5
CONTROL
i
• . . . is the essential ingredient of science,
distinguishing it from nonscientific
procedures.
• The scientist, the experimenter,
manipulates the Independent Variable
(IV – “treatment” – at least two levels –
“experimental and control conditions”)
and controls other variables.
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
6
More control
i
• After manipulating the IV (because the
experimenter is independent – he/she
decides what to do) . . .
• He/she measures the effect on the
Dependent Variable (what is measured –
it depends on the IV).
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
7
Key Distinction
i
• IV vs. Individual Differences variable
• The scientist MANIPULATES an IV, but
SELECTS an Individual Differences
variable (or “subject” variable).
• Can’t manipulate a subject variable.
– “Select a sample. Have half of ‘em get a
divorce.”
• Consider an Individual Difference, or
Subject Variable, as a TYPE of IV.
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
8
Operational Definitions
i
• Explains a concept solely in terms of the
operations used to produce and measure it.
–
–
–
–
–
–
Bad: “Smart people.”
Good: “People with an IQ over 120.”
Bad: “People with long index fingers.”
Good: “People with index fingers at least 7.2 cm.”
Bad: Ugly guys.
Good: “Guys rated as ‘ugly’ by at least 50% of the
respondents.”
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
9
Validity and Reliability
i
• Validity: the “truthfulness” of a measure. Are
you really measuring what you claim to
measure? “The validity of a measure . . . the
extent that people do as well on it as they do
on independent measures that are presumed
to measure the same concept.”
• Reliability: a measure’s consistency.
• A measure can be reliable without being valid,
but not vice versa.
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
10
Theory and Hypothesis
i
• Theory: a logically organized set of
propositions (claims, statements, assertions)
that serves to define events (concepts),
describe relationships among these events,
and explain their occurrence.
– Theories organize our knowledge and guide our
research
• Hypothesis: A tentative explanation.
– A scientific hypothesis is TESTABLE.
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
11
Goals of Scientific Method
i
• Description
– Nomothetic approach – establish broad generalizations and
general laws that apply to a diverse population
– Versus idiographic approach – interested in the individual,
his/her uniqueness (e.g., case studies)
• Prediction
– Correlational study – when scores on one variable can be
used to predict scores on a second variable. (Doesn’t
necessarily tell you “why.”)
• Explanation – con’t. on next page
• Application
– Applied research
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
12
Explanation
i
• Three important conditions for making a
causal inference:
– Covariation of events. (IV changes, and the
DV changes.)
– A time-order relationship. (First the scientist
changes the IV – then there’s a change in
the DV.)
– The elimination of plausible alternative
causes.
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
13
Confounding
i
• When two potentially effective IVs are allowed to
covary simultaneously.
– Poor control!
• Remember week 1 – Men, overall, did a better job of
remembering the 12 “random” letters. But the men
had received a different “clue” (“Maybe they’re the
months of the year.”)
• So GENDER (what type of IV? A SUBJECT variable,
or indiv. differences variable) was CONFOUNDED with
“type of clue” (an IV).
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
14
Populations and Samples
i
• Population: the set of all cases of
interest
• Sample: Subset of all the population that
we choose to study.
Population
Sample
Parameters
Statistics
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
15
Intervening Variables
i
• Link the IV and the DV, and are used to
explain why they are connected.
• Here’s an interesting question: WHY did
the authors put this HERE in the
chapter?
– Because intervening variables are important
in theories.
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
16
A bit more about theories
i
• Good theories provide “precision of
prediction”
• The “rule of parsimony” is followed
– The simplest alternative explanations are
accepted
• A good scientific theory passes the most
rigorous tests
• Testing will be more informative when
you try to DISPROVE (falsify) a theory
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
17
Ch. 3 -- Ethics
i
• Read the chapter.
• Understand informed consent, p. 70 – a
person’s expressed willingness to participate in
a research project, based on a clear
understanding of the nature of the research,
the consequences of declining, and other
factors that might influence the decision.
• Know that UT has an IRB:
http://www.utexas.edu/research/rsc/humanrese
arch/
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
18
Ch. 7 – Independent Groups
Design
i
• Description and Prediction are crucial to the
scientific study of behavior, but they’re not
sufficient for understanding the causes. We
need to know WHY.
• Best way to answer this question is with the
experimental method.
• “The special strength of the experimental
method is that it is especially effective for
establishing cause-and-effect relationships.”
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
19
Good sentence
i
• P. 223, para. 2, last sent. – “The special
strength of the experimental method is
that it is especially effective for
establishing cause-and-effect
relationships.”
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
20
Good page – P. 223-224
i
• Why we conduct experiments
• If results of an experiment (a well-run
experiment!) are consistent with theory,
we say we’ve supported the theory.
(NOT that it is “right.”)
• Otherwise, we modify the theory.
• Testing hypotheses and revising theories
based on the outcomes of experiments –
the long process of science.
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
21
Logic of Experimental Research
i
• Researchers manipulate an independent
variable in an experiment to observe the
effect on behavior, as assessed by the
dependent variable.
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
22
Independent Groups Design
i
• Each group represents a different
condition as defined by the independent
variable.
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
23
Random . . .
i
• Random Selection vs. Random Assignment
– Random Selection = every member of the
population has an equal chance of being selected
for the sample.
– Random Assignment = every member of the
sample (however chosen) has an equal chance of
being placed in the experimental group or the
control group.
• Random assignment allows for individual differences
among test participants to be averaged out.
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
24
Let’s step back a minute
i
• An experiment is personkind’s way of asking
nature a question.
• I want to know if one variable (factor, event,
thing) has an effect on another variable – does
the IV systematically influence the DV?
• I manipulate some variables (IVs), control
other variables, and count on random
selection to wash out the effects of all the rest
of the variables.
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
25
Block Randomization
i
• Another way to wash-out error variance.
• Assign subjects to blocks of subjects,
and have whole blocks see certain
conditions.
• (Very squirrelly description in the book.)
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
26
Challenges to Internal Validity
i
• Testing intact groups. (Why is the group a group?
Might be some systematic differences.)
• Extraneous variables. (Balance ‘em.) (E.g.,
experimenter).
• Subject loss
– Mechanical loss, OK.
– Select loss, not OK.
• Demand characteristics (cues and other info
participants pick up on) – use a placebo, and doubleblind procedure
• Experimenter effects – use double-blind procedure
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
27
Role of Data Analysis in Exps.
i
• Primary goal of data analysis is to
determine if our observations support a
claim about behavior. Is that difference
really different?
• We want to draw conclusions about
populations, not just the sample.
• Two different ways – statistics and
replication.
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
28
Two methods of making
inferences
i
• Null hypothesis testing
– Assume IV has no effect on DV; differences we
obtain are just by chance (error variance)
– If the difference is unlikely enough to happen by
chance (and “enough” tends to be p < .05), then we
say there’s a true difference.
• Confidence intervals
– We compute a confidence interval for the “true”
population mean, from sample data. (95% level,
usually.)
– If two groups’ confidence intervals don’t overlap, we
say (we INFER) there’s a true difference.
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
29
What data can’t tell us
i
• Proper use of inferential statistics is NOT
the whole answer.
– Scientist could have done a trivial
experiment.
– Also, study could have been confounded.
– Also, could by chance find this difference.
(Type I and Type II errors – hit this for real in
week 9.)
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
30
This is HUGE.
i
• When we get a NONsignificant
difference, or when the confidence
intervals DO overlap, we do NOT say
that we ACCEPT the null hypothesis.
• We just cannot reject it at this time.
• We have insufficient evidence to infer an
effect of the IV on the DV.
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
31
Notice
i
• Many things influence how easy or hard
it is to discover a difference.
– How big the real difference is.
– How much variability there is in the
population distribution(s).
– How much error variance there is.
– Let’s talk about variance.
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
32
Sources of variance
i
• Systematic vs. Error
– Real differences
– Error variance
• What would happen to the standard deviation if our
measurement apparatus was a little inconsistent?
• There are OTHER sources of error variance, and the
whole point of experimental design is to try to minimize
‘em.
Get this: The more error variance, the harder for real
differences to “shine through.”
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
33
One way to reduce the error
variance
i
• Matched groups design
– If there’s some variable that you think MIGHT
cause some variance,
– Pre-test subjects on some matching test that
equates the groups on a dimension that is relevant
to the outcome of the experiment. (Must have a
good matching test.)
– Then assign matched groups. This way the groups
will be similar on this one important variable.
– STILL use random assignment to the groups.
– Good when there are a small number of possible
test subjects.
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
34
Another design
i
• Natural Groups design
– Based on subject (or individual differences)
variables.
– Selected, not manipulated.
– Remember: This will give us description,
and prediction, but not understanding
(cause and effect).
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
35
We’ve been talking about . . .
i
• Making two groups comparable, so that
the ONLY systematic difference is the IV.
– CONTROL some variables.
– Match on some.
– Use random selection to wash out the
effects of the others.
– What would be the best possible match for
one subject, or one group of subjects?
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
36
Themselves!
i
• When each test subject is his/her own
control, then that’s called a
– Repeated measures design, or a
– Within-subjects design.
(And the independent groups design is
called a “between subjects” design.)
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
37
Repeated Measures
i
• If each subject serves as his/her own
control, then we don’t have to worry
about individual differences, across
experimental and control conditions.
• EXCEPT for newly introduced sources of
variance – order effects:
– Practice effects
– Fatigue effects
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
38
Counterbalancing
i
• ABBA
• Used to overcome order effects.
• Assumes practice/fatigue effects are
linear.
• Some incomplete counterbalancing
ideas are offered in the text.
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
39
Which method when?
i
• Some questions DO lend themselves to
repeated measures (within-subjects) design
– Can people read faster in condition A or condition
B?
– Is memorability improved if words are grouped in
this way or that?
• Some questions do NOT lend themselves to
repeated measures design
– Do these instructions help people solve a particular
puzzle?
– Does this drug reduce cholesterol?
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
40
Some questions we’d like to ask
Nature
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
41
Exercise
i
• Break up into groups of 5 or 6.
• Pick a question.
• Design an experiment to ask nature that question –
and find an answer.
• What are the:
– IV(s)?
– DVs
– Controls
• How will you select test participants and assign them
to groups?
• How will conduct the study?
• Take 45 minutes to come up with this. We’ll
reconvene and share experimental designs.
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
42
Some (as yet untested) online
practice problems
i
• http://webster.edu/~woolflm/zscores.html
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
43
Midterm
i
• Emphasize
–
–
–
–
How to lie with statistics – concepts
To know a fly – concepts
SZ&Z – Ch. 1, 2, 7, 8
Hinton – Ch. 1, 2, 3, 4, 5
• De-emphasize
– SZ&Z – Ch. 3
– Other readings
• Totally ignore for now
– SZ&Z – Ch. 14
– Hinton – Ch. 6, 7, 8
R. G. Bias | School of Information | UTA 5.424 | Phone: 512 471 7046 | rbias@ischool.utexas.edu
44