Chapter 6
Tools for Interrogating Frequency Claims:
Surveys and Polls
© 2012 by W. W. Norton & Company
Detailed Learning Objectives:
Question Wording and Observations
1. Give examples of how question wording can change the results of a survey or
poll.
2. Describe the different ways questions can be worded: open-ended, forcedchoice, and using rating scales.
3. Explain how to increase the construct validity of questions by wording them
carefully and by avoiding leading questions, double-barreled questions, and double
negatives.
4. Explain how question order can change the meaning (and validity) of a question.
5. Explain ways to increase the construct validity of questions by preventing
respondent shortcuts (such as nay-saying), biases (such as trying to look good), or
simple inability to report.
6. Describe how observational techniques for measurement are different from
survey techniques.
7. Explain ways to improve the construct validity of observations by reducing
observer bias and target reactivity.
Detailed Learning Objectives:
Sampling
8. Indicate the population that a given sample might represent.
9. Define three sampling problems that lead to biased samples.
10. Understand why a random sample is more likely to be a
representative sample and why representative samples have external
validity to a particular population.
11. Explain five techniques for random sampling: simple random,
multistage, cluster, stratified random sampling, and oversampling.
12. Describe three techniques of nonrandom sampling: purposive,
convenience, and snowball sampling.
13. Explain why representative samples may be especially important
for many frequency claims.
14. Consider times when an unrepresentative sample may be
appropriate for a frequency claim.
15. Explain why it is more important to ask how a sample was
collected rather than how large the sample is.
1.
A response set can be in the form of
a)
b)
c)
d)
2.
A survey question that has you rate a response from agree to disagree is
an example of
a)
b)
c)
3.
A Likert scale
A double barreled question
The semantic differential format
Which of the following is NOT a way to prevent or avoid observer bias
a)
b)
c)
d)
4.
Open ended questions
Leading questions
Yay-saying answers
A Likert scale
Masked study design
Inclusion of filler questions
Measuring traces that the behavior leaves behind
Make unobtrusive observations
Choose the following that are correctly matched (can be more than one)
a)
b)
c)
d)
Random sampling—external validity
Random assignment—internal validity
Random sampling—internal validity
Random assignment—external validity
Question Wording
Construct Validity of Surveys and Polls
• Choosing question formats
• Writing well-worded questions
• Encouraging accurate responses
Choosing Question Formats
Open-ended questions
Forced-choice format
Likert scale
Semantic differential format (example shown)
Writing Well-Worded Questions
How can you
write items
that avoid
these
problems?
• Leading questions
• Double-barreled questions
• Double negatives (vs. negatively
worded items)
• Question order
Encouraging Accurate Responses
What are
some
solutions to
these
problems?
• Response sets (three types)
– Yea-saying/ nay-saying
– Fence sitting
• Faking good or bad
• Saying more than we can know
• Measuring subjectivity vs.
objectivity
Behavioral Observations
Construct Validity of Behavioral
Observations
• Examples of frequency claims based on
observational data
• Are observations better than self-reports?
• Good observations are both reliable and valid
Some Examples of Frequency Claims Based
on Observational Data
Are Observations Better than
Self-Reports?
What are
some solutions
to these
problems?
Observers may see what they expect to see
Observers can affect what they see
Masked research designs
Observer effects
Unobtrusive observations
Wait it out
Measure results of a behavior
Observe ethically
Figure 6.6 Unobtrusive observations
This one-way mirror allows
researchers to unobtrusively record
the behaviors of children in a
preschool classroom.
Good Observations Are Both Reliable and
Valid
Good Observations Are Both Reliable and
Valid
Figure 6.8 Table 1 from Campos and colleagues’ (2009) study of family interactions.
The table depicts the degree of interrater reliability for each of the behaviors coded.
Sampling
Generalizing to Others: Sampling
Participants
•
•
•
•
Populations and samples
How to get a representative sample
When does external validity matter?
When a representative sample is not the top
priority
• Are bigger samples better samples?
www.everydayresearchmethods.com
Key Questions
a. Read the first three paragraphs carefully.
What do you think of the sampling techniques
used in this poll? To what population can we
generalize the results from this poll?
b. Your friend comments, “wow! 375,000 people
responded to this survey! We can definitely
generalize this to the American population.”
What do you say?
Populations and Samples
Population is ?
Sample is ?
Representative Samples
Probability sampling
Simple random sampling
Cluster sampling
Multistage sampling
Stratified random
sampling
Oversampling
Systematic sampling
Unrepresentative Sampling
Self-selected
Purposive
Convenience
Homework
Working with
your own
examples
• What do you think of the
construct validity of the polls your
group brought in?
• What do you think about the
external validity of the polls your
group brought in?
More Practice
Imagine that you are planning to estimate the price of the average
book at the UD college bookstore. The bookstore carries 13,000 titles,
but you plan to sample only 200 books. You will select a sample of 200
books, record the price of each book, and use the average of the 200
books to estimate the average price of the 13,000 books in the
bookstore.
Based on this information, answer the following questions:
a. What is the sample in this study, and what is the population?
b. How might you collect a simple random sample of books?
c. How might you collect a stratified random sample?
d. How might you collect a convenience sample?
e. How might you collect a systematic random sample?
f. How might you collect a cluster sample of 200 books?