ScWk 240
Week 8
Sampling (continued)
Intro to Causality and Group Designs
“The tendency of the casual mind is to pick out
or stumble upon a sample which supports or
defies its prejudices, and then to make it the
representative of a whole class.”
Walter Lippman (1889-1974), journalist
Review: Name the implied type of reliability or validity testing:
1.
2.
3.
4.
5.
Two clinical observers of parent-child
interaction mostly agree on whether the
parent’s response to the child’s behavior
is flexible or rigid _______________
A researcher wants to make sure that
the instrument she is developing covers
all the various dimensions of the concept
“empowerment.” __________________
A new instrument called the Self
Perceived Empowerment Scale has a
coefficient alpha of .85
__________________
A person scoring higher on a depression
scale (more depressed) will also often
score higher on an anxiety measurement
__________________
A person scoring higher on a depression
scale (more depressed) will also often
have a diagnosis of depression after a
clinical interview _______________
• Inter-rater reliability
• Test-retest reliability
• Internal consistency
reliability
• Criterion-related validity
• Content validity
• Face validity
• Construct validity
2
News flash—much much better
definition of construct validity:
Extent to which an instrument accurately measures
one concept. For example, an instrument
designed to measure alcoholism accurately
measures alcoholism (not depression or other
diagnosis). In addition, construct validity can also
refer to the extent to which a measure of a
construct (like alcoholism) is correlated with
another theoretically-related measure (such as
“addictive personality”)
Source: Faulkner, C. A. & Faulkner, S. S. (2009). Research methods for social workers: A
practice-based approach. Chicago, IL: Lyceum
3
Student report
1. What is the main difference between “openended” and “closed-ended” questions?
Selene
2. Discussion question for class: what are the
implications for reliability and validity for
both types of questions?
4
Two main types of sampling:
• Probability sampling -- everyone in the study
population has an equal chance of selection
into your study.
• Non-probability sampling -- everyone in the
study population does not have an equal
chance of selection
***Discussion: what are the pros and cons of
both?
5
The Logic of Probability Sampling
• Overall purpose of probability sampling: to allow the
researcher to estimate characteristics (parameters) of the
study population from a sample’s characteristics
– e.g. “How much does the sample represent the population
from which it was drawn?”
• Why do we want to “estimate population characteristics”?
– Because in most cases we don’t really know the real
population characteristics (but we’d like to know!)
• So, we estimate a population parameter (e.g. average
depression score of the population), from a sample statistic
(average depression score of the sample)
• Since research means “never having to say you’re certain”, we
won’t be 100% perfect in our estimates, although we can
quantify how confidently we feel our sample matches the
population. How? Stay tuned for ScWk 242.
6
Probability Sampling Example: “How do stress
levels of teen parents compare with stress levels
of other youth?”
Population: All
youth
Select
random
samples of
teen
parents and
non-parent
youth
Probability
sample of
teen
parents
Probability
sample of
non-teen
parent
youth
From the sample
statistic we can
“infer” stress levels
in the larger
population of youth
Statistic –
comparison
of stress
levels
7
How to do probability sampling
• Simple random sample—assigning numbers to
potential participants and selecting numbers (people)
randomly. Similar to picking them out of a hat
• Systematic random sample—choosing every kth
person--selecting persons at a predetermined interval
• Stratified random sampling--Simple or systematic
random sampling with sub-groups
• Cluster sampling—a multi-stage procedure of
randomly choosing levels of analysis units (e.g. cities,
neighborhoods, schools, classrooms)
8
Probability sampling “ripped from the
headlines…”
http://www.pbs.org/newshour/bb/politics/july
-dec12/makingsense_10-12.html
9
Non-probability sampling
•
Quota sampling—sample chosen based on predefined characteristics of study so
that sample will have same proportion of those characteristics as in the study
population
 Think of a journalist standing on a street corner interviewing people who walk by, but who
also wants the opinions of both men and women
***What’s the difference between this and stratified random sampling?
•
Snowball sampling—for difficult-to-locate populations: finding more sample
participants based on recommendations from others in the study
 You’re interviewing a victim of domestic violence and you ask “Can you think of anyone else at
this shelter who might be able to talk with me about this?”
•
Purposive sampling —selecting sample that is thought to yield the most
comprehensive understanding of the study’s topic
 You put up a flyer inviting students to participate whose parents immigrated to the US
•
Convenience sampling (a.k.a. “availability sampling”)—selects participants simply
based on their immediate availability. (Note—participants may also coincidentally
comprise a purposive sample, but not necessarily)
 You decide to interview staff of a county mental health agency about their training in cultural
competence
10
Two families of sampling
Non-probability sampling
Probability sampling
•
•
•
•
Simple random sampling
Systematic sampling
Stratified random sampling
Cluster sampling
•
•
•
•
Quota sampling
Snowball sampling
Purposive sampling
Convenience sampling
11
Student report
• Name some strategies to recruit difficult-toreach populations, and retain them for follow
up -- Annalisse
12
What type of sampling is this? (hint:
first decide if it’s prob or non-prob)
1.
2.
3.
4.
5.
For an experimental study of
cognitive behavioral therapy, every
10th adult with anxiety disorders is
selected randomly from a sampling
frame
An agency manager walks down the
hall and asks staff opinions about
the new caseload policy
The same agency manager makes
sure to ask both women & men
For a study on standardized testing
in schools, researchers randomly
select cities in California, then
randomly select elementary, middle
and high schools in those cities
A social work researcher seeks out
Latino women who were victims of
domestic violence
• Simple random
sampling
• Systematic sampling
• Stratified random
sampling
• Cluster sampling
• Quota sampling
• Snowball sampling
• Purposive sampling
• Convenience sampling
13
Student report
• What is required in order to prove causality?
Jenna
14
Establishing Causality—What are the
Requirements*?
1. Time sequence: the cause (independent
variable, or treatment) precedes the effect
(dependent variable, or outcome)
2. The IV and the DV are associated. (They must
be related or correlated statistically.)
3. The relationship between the IV and DV cannot
be explained by a third variable (or rival
hypothesis—an alternate explanation)
*Original source: John Stuart Mill (1859). A system of logic
15
Causality and an Evaluative Study—
”Does group therapy reduce anxiety?”
1. Time sequence
X
Group therapy
(indep variable)
Y
Anxiety
(dep variable)
2. Correlation (or association)
Those getting group therapy will also have reduced level of
anxiety. (Correlation alone doesn’t ensure causality!)
3. Outcome not caused by 3rd variable (rival
hypothesis)
Medication (X2)
Group tx (X1)
Anxiety
16
Set up experiment to show causality*
1.
2.
3.
4.
5.
Independent variable
should precede DV in time*
Treatment is “manipulated”
as an IV
Establish statistical
association between IV and
DV*
Eliminate rival hypotheses*
(How? See 5.)
Use control group with
Random assignment
Indep var (X)
Dep var (Y)
Indep var (X)
Attributes:
Treatment
No treatment
Treatment is associated with
some change in the DV
X1
Assign
to treatment?
X2
Sample
?
Y
Assign
to control
(no tx)?
17
Random Assignment Example: “The impact of a
parenting class on teen parents’ stress levels”
All teen
parents
Randomly
assign a
sample of
teens to
parenting
class, and
those not in
class
Those in
teen
parenting
class
Those not
in teen
parenting
class
From the sample
statistic we can
“infer” how the
population of teen
parents will benefit
from the class
Statistic –how
much better
teens in
parenting
class did
18
Alternative Ways to Manipulate
Independent Variable (not complete list)
Experimental Group
Control Group
Experimental Treatment
No treatment
Large amount of
experimental treatment
Small amount of
experimental treatment
Experimental Treatment
Other type of tx (or
treatment-as-usual)
Experimental treatment
2 control groups:
1) Other type of tx
2) Experimental treatment
plus other type of tx
19
Random sampling, random assignment
(they’re different & easy to confuse)
Study Population
N =10,000
Sample
n=100
Probability
sampling
(or Non-probability
procedure)
Random*
assignment
Experimental Group
n=50
Control Group
n=50
*Note: random assignment can
also be stratified within the
experimental and control groups
20
Student reports next week:
• What is the most important difference
between experimental group designs and all
others? (Hint—has to do with group
assignment) -- Ashley
• Give an example of a one group, pretest
posttest design -- Kristen
21
Concepts to know
•
•
•
•
•
•
•
•
Three requirements for establishing causality
Random assignment
Rival hypothesis
Experimental group
Control group
“infer” from a sample statistic
“Manipulating the treatment variable”
X is independent variable, Y is dependent variable
22