1
Tonight
? Causal Inference and Internal Validity
? Sampling
HD FS 503
Research Methods in Human
Development & Family Studies
Susan Hegland
January 28, 2002
2
3
The Research Chain
Links to previous studies
Explanation, rationale, theory, point of view
Questions, hypotheses, models
Design
Chapter 7
Participants Situations
Causal Inferences and Internal
Validity
IV
Observation
DV
Procedure
Analysis & results
Conclusion
Links to next study
4
Internal Validity (Linking Power)
?
Explanation credibility:
Internal Validity: Hoffman-Reim
• previous research? critical variables ? causal
relationships in hypotheses?
?
Translation fidelity
Apply criteria for internal validity to the
Hoffman-Reim study: Has the author
provided sufficient support to ensure:
? Explanation credibility?
? Translation fidelity?
? Demonstrated result?
? Rival explanations eliminated?
? Credible Results?
• Participants?
• Instruments (test reliability & validity?)
• Procedures
?
Demonstrated result
• Statistics appropriate?
• Percentage of variance convincing?
?
Rival explanations eliminated
?
Credible result
?
What about Zimbardo et al.? Hoffman-Reim?
• Controls for alternate explanations?
• Was hypothesis supported?
5
6
1
A Source of Potential Confusion
Complexities in the Concept of Cause
? Internal Validity
• Linking power within the study components
? Which is the cause in the causal chain?
? External Validity
? Causes as inferences
• Generalizing power to the population
? Validated
causal propositions
escape disconfirmation
? NEVER confirmed!
? Test Validity (see chapter 18)
•
•
•
•
Measurement reliability and validity
Does the test produce consistent results?
Does the test measure the construct?
Does the test predict what it is supposed to?
7
Does laziness
lead to school failure?
Necessary & Sufficient
Conditions
Is the cause sufficient to
produce the effect?
Is the cause
necessary
to produce
the effect?
8
Yes
No
Yes
The one and
cause!!!
A
contingent
condition
No
An
alternative
condition
A
contributing
condition
? Is laziness the one and only one cause of
school failure?
? Is laziness an alternative condition?
? Is laziness a contributing condition?
? What’s the direction of causality here?
? Does school failure lead to laziness?
? Is there a causal chain here?
9
10
Drug abuse
Contingent causes?
? Availability of drugs in community
? Prior use
Contributing causes?
? Poverty
? Peer use
? Boredom
Alternative causes?
? Prior addiction
Simple patterns
?A ?
B
?
Infant attachment patterns cause adult
attachment patterns
? Sensitive, accepting, cooperative, accessible
parenting causes secure infant attachment
? Rarely appropriate!
? We live in a multivariate world
11
12
2
Multiple Causes
Multiple Causes
Positive
Preschool
Experience
Family
Environment
Personality
Work
Kindergarten
Achievement
Marital
Relations
Parenting
Social
Network
Family
Resources
Child
Characteristics
Ramey & Ramey, 1997
Belsky, 1984
13
Causal Chain?
Positive parent
interactions
More family
interactions
Positive
mother-child
communication
14
Support for causal explanation
Causal explanations are hard to establish in social
sciences; they require:
? Precedence of cause in time
? Presence of effect does not reflect a chance
deviation
? Explanation and evidence are congruent
? Effect follows a detailed prediction
? Effect follows the pattern of a manipulated cause
? Rival explanations eliminated
Note: study this material carefully—it WILL be on
the exam!
Positive parentChild interactions
Shared family
views
Less teen sex
15
Triangulation: establishing
multiple supports for conclusions
16
The heart of description and explanation
is the relationships between
? attributes (values) and
? variables
Landmark paper: Baron & Kenny, 1986:
? Many researchers confuse
? moderating and
? mediating relationships
? Different roles, different statistical tests!
? Multiple sightings from multiple angles
? Multiple-measure, multiple-method
procedures
? Multiple measures
• Self-esteem and depression measures
? Multiple method
• Parents and teachers rating social skills
• Self-report and observers rating competence
17
18
3
Complex views of the world
Moderating relationships
? Preexisting
condition
? Affects when X causes Y
? Simple patterns are rare
? Multiple cause patterns are more common
• Under what circumstances
• Moderator variables establish when, or
under what conditions
• Mediator variables establish how: what’s the
mechanism of influence
• Baron & Kenny, 1986
? Examples of moderating variables
Sex
Age (developmental level)
Marital status
Financial resources
Culture
Marital status
Coping skills
Social support network
? Tested as an interaction (i.e., sex x treatment)
19
Showing a moderating
relationship
Mother’s
responsiveness
20
Another complicated study
Child’s
attachment
Parent
educational
level
Child
reading
skills
Quality of
marital
relationship
21
What’s the cause here?
22
A mediating relationship
? Is parent education the cause?
? Or does some other variable mediate the
relationship between parent education and
children’s reading?
Parent
educational
level
23
Child
reading
skills
24
4
A mediating relationship
Mediating relationships
? Describe how cause affects outcome
Parent-child
language
interactions
? When added to the model,
? “cause” is no longer related to the outcome
Child
reading
skills
Parent
educational
level
? tested in a hierarchical regression:
see if
mediator (i.e., language) accounts for some
of the variance originally accounted for the
the predictor (i.e., parent education)
Baron & Kenny 1986
25
26
Is school size the cause?
Register editorial
School size and academic achievement
? Potential Moderators? Mediators?
? Moderators
Mean ACT Scores
23
• Parent education?
• Parent income?
22
21
? Mediators
20
19
•
•
•
•
18
17
< 250
250-399 400-599 600-999
10002499
25007499
7500+
Enrollment
Quality of teachers (i.e., salaries)?
Number of advanced courses?
Ratio of extracurricular activities?
Percentage of kids who take the ACT?
? How could you test?
27
28
IV and DV may be related but
You won’t find it because
? Too much error
? Too small a sample
? Inappropriate measures
? Wrong sample
? Confounding variables
? Too stringent statistical criterion
(alpha level):
Type II error
Mediators and Moderators
Day care and aggression
? Possible mediators?
? Possible moderators?
Family therapy and quality of marital
communication
? Possible mediators?
? Possible moderators?
Your studies?
29
30
5
32
IV and DV may be unrelated but
You conclude they are related because
? treatment effects confounded with
systematic error (e.g., response set, social
desirability)
? confounding variables
? sampling error
Too loose a statistical criterion
? (alpha level)
? Type I error
Sampling
31
Teen sex prevention program and
teen sexuality prevalence
Sampling and Representation
? 300 high school students, how would you
Sampling procedures
? Strategies for selecting a small number of
units from a population
? Enable researchers to make reliable
inferences about the population
choose them?
? What steps would you go through to select
them?
? Why?
33
Depression in freshmen:
incidence and relationship to
stress
34
Required sample size is based on:
?
Certainty of inference desired
• What’s the cost of drawing the wrong conclusion?
?
Precision of inference desired
• Population estimates must be very accurate
• Or base N on prior studies (significant results!)
? Homogeneity of population on characteristic
? Goal: 200 first-year female students
? Random sample: N = 133
• Stratified by parents’ education and race
• Representative of freshman class
• multiple cultures?
• many differences in parenting behaviors!
? Add a convenience sample
• 67 cooperative females from dormitory
?
? Are you comfortable throwing these
Size of effect vs normal sampling variation
• small effect and large sampling variation ?
samples together? Why or why not?
very large study
35
36
6
Sample Size and Statistical Significance
?
Treatment effect: big or small?
Error: big or small?
Statistical tests like t and F compare
difference due to treatment (IV)
difference due to error
?
Treatment: potential cause controlled by IV
Error:
25
Number of Participants
?
• Variation in DV NOT explained by IV
• Residual
• noise
?
More error means
bigger (treatment) effect is needed
(more treatment effect; more powerful treatment)
or bigger sample is needed
?
20
15
Treatment
Control
10
5
0
0
10
20
30
40
50
60
70
80
90
100
Scores
In your review, include sample size!
37
38
To increase likelihood of finding
a significant effect…..
Treatment effect: big or small?
Error: big or small?
?
Maximize group differences due to treatment
?
Minimize group differences due to error
• Intensify treatment
50
Number of participants
45
Treatment
Control
40
35
• Choose more homogeneous sample
• Eliminate potential confounds
30
25
• Exclude some potential moderating variables
20
?
15
Example: Zimbardo:
10
• only males
5
• only those susceptible to hypnosis
• eliminate sources of systematic error
(e.g., response sets, practice effects)
0
0
10
20
30
40
50
60
70
80
90
100
Scores
39
Sampling
40
Validity (accuracy) in sampling
? External validity:
? The process of selecting the people you
want to study
? Want your sample to represent a larger
population
? Only an issue if you hope to generalize your
findings!
? In general, use probability, random
sampling
41
(Generalizing Power)
systematically (and randomly) sample from
population to which you hope to generalize
findings
? Internal validity:
(Linking Power)
randomly assign participants to treatments
42
7
Representative samples
?
All members of the population have an equal
chance of being selected
• protects against bias
• allows generalizability of findings
? Each sample member is independent, or unrelated,
to the others
? Similarity among members in one treatment is due
to treatment
• Similarity is not due to other factors
• Systematic error in sample looks like
treatment (or independent variable) effects!
Study of depression in freshman
students
? Go to dorms and find who’s there at 8 PM
on Saturday night in November
? First 100 students interviewed
? Conclusion: Nearly 50% of freshman are
depressed; major intervention is needed!!
? Are you convinced?
? How should the sample have been drawn?
43
44
Independently select each case
Probability Sampling
? Cannot measure the same person twice
• As if the data came from two persons!!!!!!!
? Stratified Sampling
? Always check N
? Proportional Stratified Sampling
• cannot be higher than number of cases!
? Oversampling
? Two recent dissertations
? Systematic sampling
• N = 30, observed 9 times; n = 270
• N = 40, observed 8 times; n = 320
• Use a repeated measures statistic!!!!
? Cluster sampling
? Unit of analysis is child,
not child at time x!
45
If I choose one husband to be a
participant in my study,
46
Statistical tests look at N
? Smaller sample
requires
bigger treatment effect for the
same treatment difference to be significant
? Bigger sample requires
smaller treatment effect for the
same treatment difference to be significant
? All statistical tests assume each case
independent of other cases!
? Is the selection of a wife independent?
? If I choose a child…
? Is the selection of parent independent?
? Therefore, what is the case?
? The individual participant?
? Or the dyad?
47
48
8
Simple Random Sampling
Types of Sampling
?
Each unit of the population has an equal
chance of being selected
? Choice of any unit is independent of choice of
others
? Sampling frame =
list of units from which we draw sample
• Faculty directory
• School directory
• Telephone directory
• List of licensed child care centers in county
? Probability Sampling
? Nonprobability Sampling
49
50
Probability Sampling
Nonprobability sampling
? Requires random sampling of units from
? Selection of measures, treatment, etc.
population
? Allows generalization of findings to
population
? Selects participants or situations for study
? Sometimes used for convenience
? The rarer the population
• the more expensive the sample
• reason for nonprobability sampling
• e.g., clinical samples
51
Sampling unit
?
The family? child? classroom? student?
? Sampling unit is smallest unit receiving
treatment
• If teacher-taught curriculum,
• the classroom is the unit, not the child!!
• If computer-based curriculum
• the child is the unit
• If parent-child interaction
• the dyad is the unit
?
ANOVA allows testing at level of group (first),
then individual level (nested design)
53
52
Stratified Sampling
? Divide members of sampling frame into
strata
? Each strata groups members who share a
characteristic
? This characteristic might affect results
? Examples of strata
• gender
• grade
• ethnic group
? May be studied as a moderating variable
54
9
Proportional Stratified Sampling
Oversampling
?
Members selected from each strata
According to distribution in population
? Multiple strata possible
• By sex within grade
? Sex and grade are additional independent
variables (“Treatments”)
? Test for interactions among independent
variables:
? for Treatment X, those who do better may be
• boys in one grade
• girls in a different grade
?
? Proportional sampling will produce too
small a sample to represent one strata
? Oversample strata with few members
55
56
Cluster sampling
Nonprobability sampling
? Randomly select clusters (typically, defined
? Judgmental sampling
by geographic proximity)
? Purposive sampling
•
•
•
•
? Snowball (chain referral) sampling
neighborhoods,
towns,
schools,
classrooms
? Sequential sampling
? Randomly select members withinclusters
57
58
Purposive sampling
Judgmental sampling
?
Special case of judgmental sampling
Pick out certain interesting cases
• Most representative cases?
• Cases likely to contradict previous results?
? Examples:
• High levels of education, low levels of
income
• Hoffman-Reim: chose cases widely different
from each other
? N.B.: All sampling is purposeful; however,
random sampling is not purposive
60
?
? Often used in qualitative research
? Relies on your experience and knowledge
of theory and previous research findings
59
10
Quota sampling
Snowball sampling
?
Also known as chain referral sampling
? Ask one leader to identify other leaders in
community
? Ask those individuals to name others
? Keep going until you return to first names
? For a difficult -to-locate population (e.g., for study
on unsafe sex practices among homosexual males)
• More common in qualitative research
• May lead to biased sample in quantitative research
? Keep knocking on doors until you fill preset
quotas
•
•
•
•
•
single parent families
DINK’s
two parent families
retired couples
single persons
? Danger!!!
• convenience leads to bias
61
62
Sample random number table
How to randomly select:
? Use the random selection procedure under
Data menu in SPSS (if cases already in
computer)
? Use random number table
? Use the Thomas Andre CRN (Cheap
Random Number) Program—let me know if
you want a copy.
? Write your own Random Number Program
using BASIC code.
35006
20206
64202
76384
19474
33309
33278
00903
12426
08002
85900
42559
14349
17403
23632
57047
43972
20795
87052
26504
09275
78985
83674
53353
27889
74211
10119
94542
14267
41744
32388
05300
66523
44167
47914
63645
89917
92648
20979
41959
63
Sampling strategy
64
Design a sampling strategy for
? The effect of premarital counseling on
? Choose sampling technique
couple communication skills
? Choose unit of analysis
? The effect of welfare reform on family self-
? Choose sampling frame
sufficiency
? The effect of Early Head Start home visits
on parenting skills
? Identify steps
65
66
11