ScWk 240
Week 7
Measurement Part 2,
& Sampling
October 8, 2012
ScWk 240 Week 7
1
Today’s Agenda
•
•
•
•
Finish Validity & Reliability
Survey development
Intro to Sampling
Literature review – writing tips and your
questions
ScWk 240 Sampling
2
Review: Potential for systematic or
random error?
1. People are more alert in the morning than after
lunch ____________
2. The meaning of “living independently” varies crossculturally __________
3. Male respondents tend to minimize their aggressive
tendencies _________
4. A 16 y.o. taking a test: “I just circle ‘C’ each time
and hope for the best”
5. The person entering data is not careful enough and
mis-codes some of the responses ________
ScWk 240 Week 6
3
Student Report
• How is validity related to cultural issues? -Dawei
ScWk 240 Sampling
4
Reliability
• Concerned with whether or not a particular
technique done repeatedly would yield the
same result each time
– Inter-rater reliability—Two researchers rating the same
behavior: do they rate them the same?
– Test-retest reliability—Is the response from the first test
the same as the response from a re-test?
– Alternate forms reliability—Is there the same response
from slightly different versions of same measure?
– Internal consistency reliability & use of coefficient alpha—
Are similar questions (about the same concept) in survey
are answered in the same way?
ScWk 240 Week 6
5
You‘ll see this in your lit review…
• Methods section—measures used: the author
will report on reliability testing (if done) using
the coefficient alpha
– A number that tells you, overall, how well the
questions in an instrument measuring the same
concepts correlate with each other (internal
consistency reliability)
– A coefficient alpha is considered good if it is .80 or
above
– The author will then report “The instrument
showed good internal consistency reliability”
ScWk 240 Week 6
6
Validity
• Concerned with the accuracy of meaning—are
you measuring what you originally intended?
• Types of validity
– Face validity – experts’ understanding of concept
– Content validity – Does measure cover all relevant
dimensions?
– Criterion-related validity – Does measure of concept agree
with (or relate with) another related concept (external
criterion)?
– Construct validity – Does measure agree with or correlate
with measures of other theoretically related concepts?
ScWk 240 Week 6
7
Validity examples
1. Validity studies show that a set of questions about alcohol
dependence correlates well with the observed daily amount
of alcohol consumption. Criterion-related validity
2. Most experts would agree that children with Attention
Deficit Disorder with Hyperactivity have difficulty staying on
task. Face validity
3. Couples who test as having low levels of marital satisfaction
communicate less effectively, leading to high levels of conflict
and, eventually, divorce. Construct validity
4. The Beck Depression Scale covers those dimensions of
depression that are most supported by evidence. Content
validity
ScWk 240 Week 6
8
Student reports
• What are some strategies for ensuring
reliability when developing survey questions?
-- Amanda
• What are some strategies for ensuring validity
when developing survey questions? -- Sarah
ScWk 240 Sampling
9
Writing survey questions to maximize
validity & reliability
• Question written at appropriate level of
understanding (age, knowledge level, language
spoken)
• Avoid “double-barreled” question
– “Are you satisfied with services and why?”
• Questions are sensitively arranged (emotionallyladen ones later)
• Instructions make sense
• Scaling of attributes makes sense
– Consistent language
– Mutually exclusive choices
ScWk 240 Sampling
10
What’s wrong with this?
How satisfied are you with the services you
received?
1.
2.
3.
4.
5.
Very satisfied
Somewhat satisfied
Don’t know
Somewhat dissatisfied
Terrible
ScWk 240 Sampling
11
Should I use open- or closed-ended
questions?
• Student Report: What is the main difference
between “open-ended” and “closed-ended”
questions? --Selene
• Qualitative vs. quantitative survey purposes
ScWk 240 Sampling
12
Why sampling?
• The problem of quality control on the assembly
line
• Usually can’t observe (take measurements from)
100% of a population
• Making the decision on who or what to observe
involves understanding
– What population do you want to study?
– Whether you want breadth or depth of information
***Discussion: So, how do you decide that?
ScWk 240 Sampling
13
First, a few terms to know…
• Study population--Your study’s focus
– As stated in your problem/research statement, and maybe your
hypothesis
– You want to generalize your study’s results to this larger population.
“Generalize” – to extend the findings of your sample to the larger
population
• Sampling frame--Those eligible to be selected into your study
– Could be your entire study population, or more typically it’s a subset
of the study population meeting criteria for inclusion in your study
– Often, literally, a list (such as a list of agency clients)
– Not every study population comes with a handy sampling frame (e.g.
homeless people)
• Sample--Those ultimately selected to be in your study
ScWk 240 Sampling
14
Study
Population
Sampling
Frame
Sample
ScWk 240 Sampling
15
Example: “The impact of a parenting class
on teen parents’ stress levels”
All teen
parents
Sampling
Frame—list of
all teen
parents in a
school
Sample
ScWk 240 Sampling
From this sample,
we want to
generalize to the
population of all
teen parents
16
Two main types of sampling:
• Probability sampling -- everyone in the study
population has an equal chance of selection
into your study. Keywords: random
assignment
• Non-probability sampling -- everyone in the
study population does not have an equal
chance of selection
***Discussion: what are the pros and cons of
both?
ScWk 240 Sampling
17
The Logic of Probability Sampling
• Overall purpose of probability sampling: to allow the
researcher to estimate characteristics (parameters) of the
study population from a sample’s characteristics
– e.g. “How much does the sample represent the population
from which it was drawn?”
• Why do we want to “estimate population characteristics”?
– Because in most cases we don’t really know the real
population characteristics (but we’d like to know!)
• So, we estimate a population parameter (e.g. average
depression score) of the population, from a sample statistic
(average depression score of the sample)
• Since research means “never having to say you’re certain”, we
won’t be 100% perfect in our estimates, although we can
quantify how confidently we feel our sample matches the
population. How? Stay tuned for ScWk 242.
ScWk 240 Sampling
18
How to do probability sampling
• Simple random sample—assigning numbers to
potential participants and selecting numbers (people)
randomly. Similar to picking them out of a hat
• Systematic random sample—choosing every kth
person--selecting persons at a predetermined interval
• Stratified random sampling--Simple or systematic
random sampling with sub-groups
• Cluster sampling—a multi-stage procedure of
randomly choosing levels of analysis units (e.g. cities,
neighborhoods, schools, classrooms)
ScWk 240 Sampling
19
Non-probability sampling
• Quota sampling—sample chosen based on predefined
characteristics of study so that sample will have same proportion of
those characteristics as in the study population
***What’s the difference between this and stratified random sampling?
• Snowball sampling—for difficult-to-locate populations: finding
more sample participants based on recommendations from others
in the study
• Purposive sampling —selecting sample that is thought to yield the
most comprehensive understanding of the study’s topic
• Convenience sampling (a.k.a. “availability sampling”)—selects
participants simply based on their immediate availability. (Note—
participants may also coincidentally comprise a purposive sample,
but not necessarily)
ScWk 240 Sampling
20
Two families of sampling
Non-probability sampling
Probability sampling
•
•
•
•
Simple random sampling
Systematic sampling
Stratified random sampling
Cluster sampling
•
•
•
•
Quota sampling
Snowball sampling
Purposive sampling
Convenience sampling
ScWk 240 Sampling
21
What type of sampling is this…?
• For an experimental study of cognitive behavioral therapy,
every 10th adult with anxiety disorders is selected randomly
from a sampling frame
• An agency manager walks down the hall and asks staff
opinions about the new caseload policy
• The same agency manager makes sure to ask both women
and men
• For a study on standardized testing in schools, researchers
randomly select cities in California, then randomly select
elementary, middle and high schools in those cities
• A social work researcher seeks out Latino women who were
victims of domestic violence
ScWk 240 Sampling
22
Student reports for next week
1. Name some strategies to recruit difficult-toreach populations, and retain them for follow
up Annalisse
2. What is required in order to prove causality?
Jenna
3. What is the most important difference
between experimental group designs and all
others? (Hint—has to do with group
assignment) Ashley
ScWk 240 Sampling
23
Concepts to know
•
•
•
•
•
•
•
•
•
•
Closed-ended questions
Open-ended questions
Population
Sample
Sampling frame
Population parameter
Statistic
Generalizibilty
Probability sampling (types)
Non-probability sampling (types)
ScWk 240 Sampling
24