Foundations of
Research
13. Survey Research Part 1
This is a PowerPoint Show

Open it as a show by going to “slide show”.

Click through it by pressing any key.

Focus & think about each point; do not just
passively click.
© Dr. David J. McKirnan, 2014
The University of Illinois Chicago
McKirnanUIC@gmail.com
Do not use or reproduce without
permission
Center for Epidemiologic Studies National Institute of Mental Health
1
Foundations of
Research
13. Survey Research Part 1
We will address five topics in the Survey modules:
Part 1 will address the first four.
Topic areas & formats
General issues in Survey research
Sources of bias (or fraud…) in survey research
Examples of surveys
Testing Hypotheses with surveys
2
Foundations of
Research
Survey Research
✓ Topic areas & formats
General issues in Survey research
Sources of bias (or fraud…) in survey research
Examples of surveys
3
Foundations of
Research
What do surveys measure?
4
 Knowledge
 Information re: current events, political or consumer choices
 Awareness: e.g., of Public health resources, government decisions...
 Attitudes and Beliefs
 Preferences or evaluations: e.g., attitudes toward racial or ethnic
groups, consumer preferences...
 Beliefs about political or social events: “which party provides the
strongest security for the U.S….?”
 Feelings or moods: quality of life, depression / anxiety, marital
satisfaction...
 Behavior
 Behavioral intentions; Intent to vote, financial plans, exercise goals.
 Self-reports of previous or on-going behavior; voting in the last
election, alcohol and drug use, exercise patterns.
Foundations of
Research
Survey research;
General uses of surveys
Survey methods have a wide range of applications,
 from single-item consumer satisfaction (“How useful did
you find this web site”)
 to full-fledged, theory-driven behavioral research.
For convenience we will consider 5 categories:
 Descriptive research
 Testing hypotheses;
 Testing the generalizability of experimental results;
 Predicting an event or outcome;
 Pragmatic / applied questions.
5
Foundations of
Research
Uses of surveys; descriptive research
6
 Descriptive research
 Epidemiology is the study of how behaviors, disease states, or
similar issues are distributed across the population.
 Epidemiology uses many methods, such as standard crime or
disease reporting.
 Even Google search data are used to track flu spread.
 Many epidemiological studies use direct survey methods, such as
phone or face-to-face survey interviews.
 Knowledge of, e.g., how to access health care…
 Feelings or moods, such as the rate and distribution of depression…
 Behavioral patterns, such as alcohol or drug use or gun ownership…
Foundations of
Research
The origins of epidemiology
7
 In London of 1854 a cholera outbreak raged through
several poor neighborhoods of London.
 Sewage and other effluvia that ran through gutters
created a dreadful smell (a “miasma”) that was blamed
for the outbreak.
 The concept of infectious disease transmission through
water supplies was not well understood.
 Dr. John Snow, one of the physicians charged with stopping the
epidemic, noted a particularly fetid cesspool in front of 40 Broad St.,
proximal to a water pump used by the neighborhood.
 He decided to empirically map the cholera cases in the area.
 He proposed that water from the pump, not the miasma, was the cause
of the outbreak.
 He was was generally disbelieved, but convinced the town governors by
his evidence.
Image:
http://leabridge.org.uk/waterworks/timeline/1
852-1880.html
Only much later would tracking of disease
outbreak be labeled ‘epidemiology’.
Click for Wikipedia article.
Dr. John Snow’s Cholera map & the closing of the Broad St. pump.
Foundations of
Research
8
The origins of epidemiology
Dr. John Snow’s Cholera map & the closing of the Broad St. pump.
of cases to be near the pump
at 40 Broad St….
Click for Ted talk by Steven Johnson.
 …and to radiate out from
there.
 As he noted in his 1855 book:
"I had an interview with the Board of
Guardians of St. James's parish, on the
evening of Thursday, 7th September, and
represented the above circumstances to them.
In consequence of what I said, the handle of
the pump was removed on the following day.”
 By carefully describing the
distribution of cases and the
circumstances around the pump, Snow was able to empirically
demonstrate a likely cause.
 His hypothesis was supported by the epidemic quickly subsiding once
the pump handle was removed.
S. Johnson, The Ghost Map (2007), Riverhead Books.
 Snow’s map showed the bulk
Foundations of
Research
Uses of surveys; descriptive research
9
 Descriptive research
 Epidemiology
 Political / social description is what we often think of as surveys.
 Opinion polls about society, the government, or current events,
e.g. Gallup Polls, or systemic studies by Pew Memorial Trust.
 The Consumer Confidence Index is a highly standardized poll that
is used for basic economic decision making.
 The Census, of course, is our national information source.
Foundations of
Research
Uses of surveys; descriptive research
10
 Descriptive research
 Epidemiology
 Political / social description.
 Testing hypothesis
 Assessing blocking variables
 We often assess blocking variables to test how a given attitude or
behavior varies across important social groups.
e.g., gender, age group, ethnicity, geographic location…
 Correlational studies
 A key form of analysis is examining the association among different
variables
e.g., what are the correlates of dieting…
Foundations of
Research
Uses of surveys; descriptive research
 Descriptive research
 Epidemiology
 Political / social description.
 Testing hypothesis
 Assessing blocking variables
 Correlational studies
 Examine generalizability of experimental results
 E.g., The Consumer Reports survey on therapy we discussed in quasiexperiments
 Predict event or outcome;
 E.g., election polling
 Pragmatic / applied,
 E.g., marketing, or consumer surveys.
11
Foundations of
Research
Surveys; populations
 Who do we want to generalize to?
 Our sampling frame is based on our hypothesis or empirical
question.
 Sampling: breadth  internal validity tradeoff
 Key dimensions:
 Demographic
 ethnic / age / gender groups,
 “all Americans”…
 Behavioral
 “likely voters”, alcohol users, home buyers...
 Self-identification
 Republicans / Democrats,
 “students”…
See Design and sampling overview See also: diminishing validity of political polling.
12
Question Formats: Closed-ended
Foundations of
Research
items

Specific rating scale or highly structured prompts

Most reliable for concrete behaviors

An attitude can be assessed in several ways:

Direct (face valid) assessment
Research methods is a wonderful course…

Does not 1 2 3 4 5 6 7 strongly
agree at all
agree
Behavioral (content valid) indictors
How many times this semester have you skipped class?
How many hours per day do you spend reading the material?

13
0
______
2.5
______
Researchers typically use the M of several related items to
create a more reliable measure of a variable.
Foundations of
Research
14
“Closed-ended” items, cont.

Example: The CES-D Depression inventory:

We may use the M score of these 9 items as our depression
index…
Moods & Feelings
Below is a list of different feelings. Circle the number that shows how many days you felt each
of these over the PAST WEEK.
I was bothered by things that usually do
not bother me.
I felt I could not shake off the blues even
with help from my friends or family.
I had trouble keeping my mind on what
I was doing.
I felt depressed.
I felt that everything I did was an effort.
My sleep was restless.
I was happy.
I enjoyed life.
I felt sad.
Rarely or
none of
the time
A Little
of the Time
A moderate
amount of
the time
Most or all of
the time
(less than 1 day)
(1 or 2 days)
(3 - 4 days)
(5 - 7 days)
0
1
2
3
0
1
2
3
2
3
2
2
2
2
2
2
3
3
3
3
3
3
0
1
Sum
of item
ratings
/19
0
0
0
0
0
0
1
1
1
1
1
These items are “reversed”
in the final score
Foundations of
Research
15
“Closed-ended” items, cont.

Or we may count the number of symptoms

We may have a cut point: e.g., moderate depression is
defined as 4+ symptoms
Below is a list of different feelings. Circle the number that shows how many days you felt each
of these over the PAST WEEK.
I was bothered by things that usually do
not bother me.
I felt I could not shake off the blues even
with help from my friends or family.
I had trouble keeping my mind on what
I was doing.
I felt depressed.
I felt that everything I did was an effort.
My sleep was restless.
I was happy.
I enjoyed life.
I felt sad.
Rarely or
none of
the time
A Little
of the Time
A moderate
amount of
the time
Most or all of
the time
(less than 1 day)
(1 or 2 days)
(3 - 4 days)
(5 - 7 days)
0
1
2
3
0
1
2
3
0
1
2
3
1
1
1
1
1
2
2
2
2
2
2
3
3
3
3
3
3
# of symptoms:
0 rated 21or 3
items
0
0
0
0
0
Center for Epidemiologic Studies National Institute of Mental Health
Moods & Feelings
Foundations of
Research


Closed-ended items, summary
Chief virtue: clear operationalization

Specific & concrete; we know exactly what the
participant is responding to

Easy to quantify & use statistically

Can be tested for reliability
Chief liability: potential insensitivity

Brief & simply worded; potentially superficial

“Top down”; issues are imposed on the participant


Discrimination studies: no option for “has no attitude”
Attitudes / moods: not sensitive to participants’ personal
perspectives (?).
16
Foundations of
Research
Survey formats;
“Open-ended” items
General textual / qualitative response;

More sensitive to the respondent
How have you enjoyed your methods class so far?
Please list the three things that first come to mind when you
think of The Foundations of Research.

More difficult to interpret

Can be analyzed as qualitative data (see discussion in Descriptive data.)

Can be quantified;
frequency counts of citations or statements
 “linkages” analysis of co-occurring statements


Often presented as textual portrayal plus minor quantitative
analysis.
17
Foundations of
Research
18
Mixed survey formats
Example of a mixed question format
from a survey of women’s sexual
practices.

Closed-ended attitude scale

Open-ended description

Simple behavioral index.
Personal Safer Sex Guidelines
How strict are your personal guidelines or rules for safer sex (e.g., condom use,
“safe relationships,” etc.)?
1
Not at all
Strict
2
3
Somewhat
Strict
4
5
6
Very
Strict
7
Extremely
Strict
What are your rules for safer sex?
Have you ever refused to have sex
with someone to stay safe?
0
1
2
3
Never
once or
twice
a few
times
many
times
Foundations of
Research
SUMMARY




19
Survey topics & item types
Surveys assess:

Knowledge

Attitudes or preferences

Ongoing or intended behavior
Closed-ended formats
Psychology 242 is a wonderful course…

Highly structured, easy to analyze

Potentially insensitive
Open-ended formats
…list the three things that first come to mind…

More sensitive to the participant

Potentially ambiguous or difficult to analyze
Surveys typically…

Use multiple items

Employ several formats.
Foundations of
Research
Survey Research
Topic areas & formats
✓ General issues in Survey research
Sources of bias (or fraud…) in survey research
Examples of surveys
20
Foundations of
Research

Self-report questionnaire


Forms of survey administration
21
“Paper and pencil” or internet-based;

Primarily closed-ended, structured questions

Limited open-ended items
All data collection
increasingly
computer-based

Assume at least moderate reading level

Cheap & easy to administer

Internet: Representativeness very dubious
Face-to-face interview

“Door step”, formal research center, or telephone

Allows in-depth qualitative questions

Many studies combine questionnaire & interview formats

(Telephone version becoming obsolete)
Foundations of
Research

Cost / population access

Different methods are more / less likely to reach certain
populations, e.g.:




General issues in surveys
Disfranchised / poor populations often not reached by internet or telephone
Cell phones & avoidance of telemarketers  less availability for telephone
surveys
Stigmatized populations less available for face-to-face interviews, more
available via internet.
Participant sophistication

Participants may not be able to accurately report certain topics

Attitudes toward stem cell research from readings.

What factors are most important to your choice of political candidate....


Describe the amounts and types of proteins you eat during a typical
week...
“Rationality bias”; many questions (incorrectly?) assume a rational
reason for behavior:

Why do you have unsafe sex...

What is your chief reason for using alcohol each night…
22
Foundations of
Research


Social Desirability Responding
Clear face-valid items addressing embarrassing topics yield
less valid responses

How often are you dishonest with your friends?

Have you ever cheated on an exam....?
High social desirability wording elicits inaccurate responses…

Do you support protecting our Nation’s forests for future generations?
(Does “yes” mean you an “environmentalist”?).

Do you feel there are ways your husband could be closer...?
(Does “yes” make you are unhappy in your marriage”?).

Populations differ in social desirability responding; may be a
confound in studying group differences
 Women report more suicidal thoughts, but may be more willing to
disclose, creating a possible confound…

23
Desirability can be minimized by:

Anonymous surveys

Assurances of confidentiality

Computer administration (no personal interaction)

Careful wording / pilot testing of items
Foundations of
Research
24
Social desirability responding
Click image for NY Times article

Do people lie on surveys?

Men routinely report more sex
partners than do women.

If the sample is unbiased by gender, number of partners should
balance for men & women.

Social desirability hypothesis:

Women underestimate partners

Men overstate partners

Much of the difference due to:
 A high proportion of women
who report 1 partner
 A few men who report many
partners.

Possible sample bias (confound?)
in who responds to such surveys?
Click for article from phys.org
Foundations of
Research

General issues in surveys: Time Frames
Rare(er) events require a long time frame to assess


When was your last doctor’s visit…
These questions asses the last time you left a romantic
relationship…
 Longer term recall can be surprisingly unreliable
 Recall of last doctor visit highly unreliable when checked against
medical records

Shorter time frame yields more reliable responding
 Memory is better for more recent effects
 “Exit interviews” from medical visits far more reliable than even 2week retrospective measures.
 Current, concrete behaviors are more accurately reported than
behavioral trends.
 In general, how often do you miss a dose of your medication
Less reliable than…
 Let’s go over each of the past 7 days and tell me if you took or
missed your medication dose.
25
Foundations of
Research

General issues in surveys: Question Order
Questions trigger participant’s memory or attention,
and can bias questions that follow, e.g.:
Do you think Social Security & Medicare payments have kept up
with inflation..
then
Do you favor or oppose Democratic efforts to expand Medicare
payments...

Bias can be limited by counterbalancing questions.

Using different question orders in different versions of the
survey.
26
Foundations of
Research

Summary

27
Survey administration

Internet increasingly important as self-report method

Face-to-face interviews more common in clinical research

Time frames & question order can influence responses
Population access & sophistication

Some groups are difficult to reach

Creates threat to External validity


General Issues Summary
Assumption that participants understand survey materials
often questionable.
Social desirability responding


Inhibited responding threatens Internal Validity
May represent a confound if groups differ in desirability
set.
Foundations of
Research
Survey Research
Topic areas & formats
General issues in Survey research
✓ Sources of bias (or fraud…)
Examples of surveys
28
Foundations of
Research


Bias / Fraud in survey research
Social research is increasingly important to political &
cultural debates.

Effects of gay marriage

Political “approval” ratings

Scientific consensus on global warming

Research on working mothers …
Pressure for confirmatory results encourages bias or
outright fraud (see this week’s article on Opinion Polls).


In the study structure

Items used

Sample
In the interpretation of results

“Cherry picking”

Simple distortion
29
Foundations of
Research

Opposition to gay marriage



Example
30
Example of fraudulent survey use
Judges have struck down religious doctrine as a basis for
disallowing gay marriage
Opponents have sought evidence of civil
harm to justify discrimination
Harm to children has emerged as key
issue in the debate

APA; multiple studies show no harm.

Mark Regenerus’ Survey: Children in
households with gay/lesbian parent
fare worse.
 Funded & cited widely by gay marriage opponents
 Study sample and interpretation of results wildly biased
 Has been disavowed by Renenerus’ Academic Department and
April DeBoer and Jayne Rowse … after
closing arguments in their challenge to
Michigan’s marriage restriction. Click image
for coverage.
Credit Mandi Wright/Detroit Free Press, via Associated
Press
American Sociological Assn.
 Continues to be central “evidence” in court cases
Foundations of
Research

31
Sources of survey fraud; question wording can elicit a
response desired by the researcher;


Bias in survey research: Leading or biased items
How much do you support the administrations’ actions to protect
you and your children from terrorists…
Wording can “normalize” a response, e.g.,

When do you feel that it is O.K. to cheat on an exam?
..when I really do not know the material
.. when others are doing it
.. when I think the exam is unfair

Vague wording can be interpreted in a biased fashion
Is there anything your husband could do to be more intimate with you?

“Push” polls: a survey can be used to actually create an
attitude.
Foundations of
Research
Push Polls:

An item “traps” the participant into endorsing a specific view.

When the “data” are released the biased wording is ignored.



32
Publicity about the “findings” (e.g., by politically biased news organizations) are
used to further create or change attitudes.
Many political & social organizations use this strategy to…

Ostensibly measure attitudes objectively

Use the “results” to influence popular opinion.
Example of a highly biased survey: The Republican National
Committee ‘Future of American Health Care’ survey.

The survey was distributed in several counties as part of a fund raising
letter.

It is clearly a “push poll” designed to create fear of health care reform.

It got limited distribution, but is a great example of a Push Poll
Foundations of
Research


GOP health survey (2009 – 2010)
Some of these items are simple lies – or manipulations based
on lies – designed to induce anti-health care attitudes…
Other are powerful (and dishonest) emotional manipulations
✓
✓
✓
✓
33
Foundations of
Research


Forms of survey bias:
34
Provide leading or emotionally manipulative information to
induce an attitude rather than simply measure it, to provide
politically useful “data”..
Questions that, if you accept their assumptions, can only
be reasonably answered one way…
Foundations of
Research

Biased surveys: Democratic example
An example from
the Democrats,
that is also used
for fund-raising.
35
Foundations of
Research
Manipulative
presentation
of
questionable
information
Simple
emotional
manipulation
Distorted
description
that may be
changed in
presentation
of findings
“Who could
disagree”
item.
Democratic biased survey (2007)
36
Foundations of
Research
Summary: Manipulating attitudes by surveys
1. Ask manipulative or highly leading questions
Summary
2. Find high levels of agreement (and potentially change
participants’ attitudes).
3. Publicize – and often distort or overstate – the
“findings” via highly biased news sources
4. News reports themselves lead to attitude change
among people who are uncertain or uninformed.
37
Foundations of
Research
Survey Research
Topic areas & formats
General issues in Survey research
Sources of bias (or fraud…)
✓ Examples of surveys
38
Foundations of
Research
39
Examples of surveys & data, 1
Consumer reports survey of mental health care
question
• Satisfaction
with therapy.
• Differences
between types
of therapy.
population
sample
data
findings
Self-Identified
group:
Self-selected
convenience
sample:
Attitudes &
behavior
Descriptive &
hypothesis
tests
U.S. mental
health care
users
Readers who
got therapy &
returned the
survey, n=4000
Self-report
questionnaire,
cross-sectional
High
satisfaction
for most
treatments
Foundations of
Research
40
Examples of surveys & data, 2
“Monitoring the future” youth studies
question
• Social
behavior
• Academics
• Alcohol &
drug use
• Health.
population
sample
data
findings
Demographic
group:
Random
sample:
Knowledge,
attitudes and
behavior
Mostly
descriptive
All U.S. youth,
15 -> 21 years
old.
Sample of High
School health
classes,
n=3000 -> 5000.
Face--to--face
interviews &
questionnaires,
longitudinal
(bi-yearly)
Assess yearly
trends/shifts in
drugs, grades,
emotional well
being
Foundations of
Research
41
Examples of surveys & data, 3
Gallup, Time/CNN, other polls
question
populations
samples
data
findings
Demographic:
National,
random
Knowledge,
Attitudes,
Behavior
Descriptive
Digit dial
telephone,
n=150 to >500
Brief interview,
cross-sectional
Ratings of
politicians,
Consumer
preferences
Approach to
Affordable
Care Act
- Eligible voters,
Political
opinions,
Lifestyle
information
Social
attitudes, e.g.,
managed care
- Target age
groups
Self-identified:
“Democrats”…
Behavioral:
Voters, ACA
users…
General:
- U.S. adult
population
Foundations of
Research
42
Examples of surveys & data, 4
Exit polls
question
Election
outcome,
possibly
stratified by
state / region
population
sample
data
findings
U.S. electorate
Probability
Self-reported
behavior
Descriptive/
predictive
National and/or
local electoral
district
Stratified
random sample
of electoral
districts.
Self-report
interview,
cross-sectional
Increasingly
inaccurate
predictions
See reading on shifts in use of polling data in U.S. politics
Foundations of
Research
43
Examples of surveys & data, 5
”Social Issues Survey” of Chicago gay / lesbian
community
question
- Stress &
coping
- Alcohol &
drug use
- Responses to
HIV / AIDS
population
sample
data
findings
Self-identified
Targeted multiframe
Attitudes &
behavior
Descriptive &
hypotheses
Self-identified
gay, lesbian, &
bisexual adults
in Chicago.
community
newspapers,
organizations,
& mailing lists,
n=3500
Self-report
questionnaire,
cross-sectional
- High
experience of
discrimination
- Less stress &
alcohol-drug
use than
expected
Foundations of
Research
44
Examples of surveys & data, 6
National Institute on Drug Abuse Household survey of
Alcohol and Drug use
question
Alcohol-drug
use and
problems,
treatment use,
health effects.
population
sample
data
findings
National
Random Multistage:
Knowledge,
attitudes &
behavior
Typically
descriptive
All U.S. adults
1. Census tract
2. Household,
3. Any adult
member
Face - to - face
Interview,
successive
cross-sectional
(each 5 years)
Age & regional
differences in
substance use,
trends over
time in use &
problems
n>4000
 Data often used for hypothesis-oriented
secondary analyses (i.e., as archival data).
Foundations of
Research

Surveys typically use multiple items to measure
each hypothetical construct

Summary



Summary: Testing Hypotheses
Correlations among items tell us if they are reliable in
measuring the same construct.
We use Mediating Analyses to

Test hypotheses about correlations between constructs

Build or test theory
Cross-sectional analyses are difficult to interpret

Causal direction?

3rd variable problem
Longitudinal analyses help us determine causal
direction
45