First: Review Test Validity
HD FS 503
Survey Methods
Construct Validity (Handbook, p. 7)
• (Face Validity)
• Content Validity
• Convergent
• Discriminant
• Criterion Validity
Susan Hegland & Mary Winter
Applications of Dillman & Krathwohl
– Concurrent
– Predictive
February 18, 2002
1
2
Survey Research
Contents of questions
• a way to collect original data for describing
a population too large to observe directly
• Attitudes
– feelings
– most sensitive to wording
• Beliefs
– whether you think something is true or false
• Behavior
– always reported behavior
• Attributes
– characteristics of the respondent
– includes household
3
Measuring Change over Time
4
Cross-sectional Study Sampling
Advantages and Disadvantages of each?
• Cross-sectional Study Sampling
• Trend Study Sampling
• Cohort Study Sampling
• Panel Study Sampling
• Cohort-Sequential Design
population
sample
Time 1
5
6
1
Trend study sampling
Cohort Study Sampling
Population 1
Population 1
Population 1
Sample 1
Sample 2
Sample 1
Sample 2
Time 1
Cohort
Cohort
Population 2
Time 2
Time 1
Time 2
7
How does attrition affect results
in a longitudinal design?
Panel Study Sampling
(longitudinal study)
Population 1
Sample 1
Time 1
8
• Among which strata is attrition most likely
to occur?
• Where would you look to determine if
attrition (mortality) has differentially
affected your sample?
• How could you statistically correct for such
attrition?
• How will you convince your sponsors that
your results generalize to the whole
population?
Population 1
Sample 1
Time 2
9
10
Cohort Sequential Design
Cohort Sequential Design
• Two Cohorts
• Each measured longitudinally
• Advantages of PanelAND Trend Study
Sample 1
8’s, 12’s
Sample 1
12’s, 16’s
Sample 2
8’s, 12’s
Time 1
11
Time 2
(4 years later)
Sample 2
12’s, 16’s
Time 3
(4 years later)
12
2
Potential sources of error in
surveys:
Cohort Sequential Design
Year:
1980
1985
1995
Cohort 1
18
23
----
Cohort 2
----
18
23
• Coverage error
• Sampling error
• Measurement error
• Nonresponse error
Dillman, ISU speech, Fall 2001
Age
13
Coverage error:
14
Sampling error:
The result that all units in population doNOT
have probability > 0 of being included in
sample:
• Some units may have no chance of
selection
• Some units may have multiple chances of
selection
• Some units may not qualify for the survey
• Reduced by selecting appropriate
sampling frame
• The result of only surveying a portion of the
survey population rather than all of the
members
• Reduced by increasing sample size
• Reduced by insuring random selection
15
16
Measurement error*:
Nonresponse error:
*test reliability and test validity issues
The result of inaccurate answers to
questions that stem from:
• Poor question wording
• Poor interviewing
• Survey mode effects
• Answering behavior of the respondent
• Reduced by careful design and piloting
• The result of NOT getting some people in the
sample to respond to the survey request….
• WHO, had they done so, would have provided a
different distribution of answers than did the
respondents!!
• Reduced by
–
–
–
–
17
Careful design,
Recruitment (e.g., letter of invitation)
Distribution
Follow-up
18
3
Reducing measurement error:
How to increase coverage error:
• A news survey through a call-in 1-800
number
• An on-line survey
• Distributing surveys at a conference of
professionals
• A telephone survey of low-income families
• An e -mail survey of child care providers
19
How to write questions to reduce
measurement error:
• Open ended
Visual elements of the questionnaire:
• Words
• Numbers
• Graphics
Interpretative aspects of questionnaire:
• Cultural Expectations
• Visual Display
– Questions
– Answer Recording
• Cognitive Influences
– Response characteristics
– Task orientation
20
Open-ended questions
• Respondents like these!
• Dillman: always end with one
• Be careful about starting with one
– how do you feel about?
• Close-ended
– ordered choices
– May work with middle-class audiences
• Close ended, no order
• Partially close-ended
– Other (please specify)_____
21
Evaluating questions (Dillman)
Open-ended: Why? Why not?
• Easy to understand?
– Are words simple?
• Advantages
• “Does your child attend to speakers at church, or other
meetings?”
– Own words
– New insights
– Are concepts clear?
– Avoid abbreviations
• Too vague?
• Disadvantages
–
–
–
–
22
– Complex words may have precise meanings
– If respondents know these terms, use them
Hard to code
Costly and time consuming to code
Need elaboration
Can’t analyze unless category more than 20%
• Too precise?
• Biased in wording?
– Which political party developed this survey?
23
24
4
More Dillman concerns:
Is the question...
How not to gather data...
Do you believe that the Social Security program plays a vital and
crucial role in our American Society?
Yes ____ No _____ Undecided _____
• Objectionable? Intrusive?
Establish a context
May require more questions
Do you believe the Second Amendment of the Constitution totally
prohibits any type of control over gun ownership?
Yes ____ No _____ Undecided ____
• Technically accurate?
• Too demanding?
Do you believe that the education of our nation’s children should
be a top priority of government?
Yes ____ No _____ Undecided _____
Do you support federal legislation and strict law enforcement to
protect the patients, doctors, nurses, and employees of women’s
health clinics?
Yes ____ No _____ Undecided _____
What percentage of your income do you spend on food?
• Double -barreled?
Watch out for and’s
Have you stopped beating your wife?
How often you you exchange services? (food, babysitting, etc.)
• Double negative?
25
Does the question...
•
•
•
•
•
•
•
•
Is it not right for a couple to limit family size? Yes__ No__
26
Mutually exclusive choices
Offer mutually exclusive choices?
Offer exhaustive choices?
Assume too much knowledge?
Assume too much about behavior?
Offer an appropriate time referent?
Tap accessible information?
Access socially desirable response?
Intrude?
• Can the respondent put him/herself in only
one place?
– I grew up in a community that was
• urban
• rural
• suburban
27
28
Exhaustive choices
Practice for you
• Can the respondent put him/herself
someplace?
• Indicate your religious preference:
Write two demographic questions:
• Marital status
• Educational level
• Are they mutually exclusive?
• Are they exhaustive?
• Check with a partner
• Share with the group
– Protestant
– Catholic
– Jewish
29
30
5
Don’t assume too much about
behavior...
Don’t assume too much
knowledge...
• How often do you read books to your children?
• Do you believe the Second Amendment of
the Constitution totally prohibits any type
of control over gun ownership?
Yes ____ No _____ Undecided ____
–
–
–
–
Daily____
Several times each week_____
Weekly_____
Monthly_____
• How could you reduce social desirability here?
• What proportion of your income do you spend on
groceries each week?
31
Offer an appropriate time
referent
–
–
–
–
1% or less
More than 1 % but less than 5%
More than 10% but less than 15%
….
32
Likert and Likert-type responses
• Likert
• How to ask about income?
–
–
–
–
–
– weekly income?
– monthly income?
– yearly income?
• Which will yield the most accurate
estimates?
• From which populations?
Strongly agree
Agree
Undecided
Disagree
Strongly disagree
• AVOID three-choice responses:
– Always
– Sometimes
– Never
• Variance is the sine qua non of responses!!
33
Reducing nonresponse error:
Organize the questionnaire
For Young Children (e.g., Harter)
• DO NOT START WITH DEMOGRAPHIC QUESTIONS!
• Entice the respondent into answering
• Why should I tell you my
income/gender/education when I don’t know why
you want this information?
• Start with easy, but important questions
• Pictorial format: prereaders
• Two level choices: primary grades
Mostly Agree
Strongly
Agree
Somewhat
Agree
– That the respondent is likely to care about!
– That the respondent WANTS to tell you about!
Mostly Disagree
Somewhat
Disagree
34
Strongly
Disagree
35
• Start each section with an organizing statement:
“Now we’d like to know about…
your background”
36
6
Reducing coverage error:
Selecting the appropriate
survey type
Advantages of face-to-face
interviews
•
•
•
•
•
•
• Face-to-face
• Self-administered
– Snail-mail
– E-mail
– Web-based
• Telephone
Gets the most detailed information
Highest response rate
Geographic sampling
Know who will respond
Longest possible interview
Fewer no responses
37
Cautions for face-to-face
interviews
38
Self-administered interviews
Advantages
• Least expensive of the three types!
• Don A. Dillman
• Formerly: Tailored Design Method (TDM) (1979)
• Now: Mail and Internet Surveys (2000)
• Higher probability for interviewer fraud
• Know circumstances of interview-– who else was present?
• Respondent can get consultation
• EXTREMELY expensive -- $150/interview!!
– Identify and shape each aspect of survey process
• Likely to affect quality
• Likely to affect quantity of response
– Organize survey efforts to obtain highest response rate
39
Self-administered interviews
Disadvantages
• Need a list, which may be a problem
• Who completed?
• Response selectivity:
who returned? who pitched it?
• More chances for omitted questions
• Least detail, shortest schedule
• Lowest response rate
• Dillman suggests strategies to improve
response rate
40
Telephone interviews
Used to be a decent compromise!
• Sample: lists or Random Digit Dialing
• Callbacks are relatively cheap
• Response rate:
between face-to-face and self-administered
– easy to hang up!
• Half an hour is maximum
– middle in detail between the other two
41
42
7
Internet, e-mail, and interactive
voice surveys
More on telephone interviews
• Low level of “no responses”
• Most sensitive to construction issues:
• Extensive use for surveying geographically
dispersed general populations
• E-mail & Web
– Screw up pretty fast!
• Easiest to avoid distortion and subversion by
interviewers
• Know exactly who the respondent is; only they
can hear you!
• Disadvantages:
– Current coverage is inadequate for most e-mail and
web surveys (October 1998)
• 42% US households had computers
• 26% e-mail connections (up from 4% in 1994!)
– Obtaining large quantities of data CANNOT substitute
for good coverage!
– high percentage of employed Iowans
– Answering machines
– Unlisted phone numbers
• CD-DIAL no longer conducts telephone surveys
• Gallup continues to use
• Okay for populations of university professors & students!
43
– Sampling frame is currently impossible
(directory of e-mail addresses?)
Time and money are issues!
Reducing Nonresponse Error
Do what you can afford!
Test and pretest the instruments
• On anyone: are the questions clear?
• On someone like the potential respondents
• Documenting the actual rate
• Distribute with enticing letter
–
–
–
–
–
44
Dillman: Social Exchange Theory
This is an important issue…
You are one of a few…
Your views are important
Include something of value?
• Follow up with postcard
• Follow up with second mailing
45
Control issues:
46
Sample control
Different for
• Self-administered surveys
• Telephone surveys
• Face-to-face interviews
• Report a response rate
– How many contacted?
– How many refused?
– How many could not be contacted?
• Be specific on how response rate
calculated
47
48
8
Self Administered
Total sample size =
# of completed questionnaires
+ # questionnaires returned by post office
+ # questionnaires not completed because
respondent ineligible
+ # unusable questionnaires
+ # questionnaires not returned
49
Total sample size =
Face-to-Face or Telephone?
# of contacts
+ # not contacted
• Where are you in the sample?
• Has the number of contacts been uniform?
Contacts include:
# completed interviews
+ # not eligible contacts
+ # of refusals
+ # unable to complete (e.g., sickness,
language)
51
52
Caculating response rate
Some gray areas in calculating
response rates
Response rate =
# of completed interviews
total sample size - # not eligible
• Are those who do not speak English not
eligible?
• only if you defined the population as
English speaking!
53
54
9
Additional issues:
Quality control
• Number of attempts should be uniform
across sample
• Progress through the sample:
• Data only as good as accuracy of persons who
collected, coded, and entered it!
• Hire them carefully
• Train them well
– Observe, videotape, calculate percentage
agreement
• Supervise them closely, especially early on
– Verify data--telephone respondents
• Do not be afraid to shift or fire persons
– Keep a sample control sheet
• Remember sampling: draw more than you
want
• if distributed, follow through!
• Have a plan BEFORE you start!
55
Interviewer’s Manual
Secondary survey data sets
• General considerations, including ethics
• Specific techniques
–
–
–
–
• If they have the variables you want from the
population you want-They’re Great!
• Large samples
• High quality interview techniques
• NSHF; PSID; AHEAD; NHES; NSAF
• http://lion.icpsr.umich.edu/IAED/nhes.html
put your hair up
avoid upholstery
avoid slogan clothing or jewelry
Avoid expensive clothing or jewelry
• How to administer the interview
• How to record answers
• Observe interview technique and recording
question by question
• Ensure extensive practice prior to data
collection
57
58
Next week
Practice writing survey
questions
•
•
•
•
•
56
• Ethics in research
– Different from ethics in intervention
Handbook, p. 12
Introduce survey
Demographic background
Open-ended questions
Likert-type questions
• Krathwohl, chapter 10
• Complete ISU web-based training
http://www.research.iastate.edu/training/
– You cannot be involved in any human subjects data
collection until you have received “certification”
– You cannot receive the data set for Assignment 2 until
you have completed the human subjects training!
• Ethics in authorship discussions
• In class exercises:
59
– Research ethics problems, p. 9-10
– Authorship ethics, p. 11-12
60
10
March 4: Exam 1
Breadth:
• 10 terms
• Krathwohl plus moderating/mediating
– briefly explain
– give example from HD FS research
Depth
• Two essay problems:
– Apply understanding of research methods to
real problems
61
62
11