Psychology 242
Introduction
to Research
1
Psychology 242, Dr. McKirnan.
Survey research
 Topic areas & formats
 Testing Hypotheses
 General issues in Survey research
 Sources of bias (or fraud…)
 Examples of surveys
Psychology 242, Dr. McKirnan
Survey research
3/10/14
Psychology 242
Introduction
to Research
2
Psychology 242, Dr. McKirnan.
Survey research
✓ Topic areas & formats

 Testing Hypotheses
 General issues in Survey research
 Sources of bias (or fraud…)
 Examples of surveys
Psychology 242, Dr. McKirnan
Survey research
3/10/14
Psychology 242
Introduction
to Research
What do surveys measure?
 Knowledge
 Information re: current events, political or consumer choices
 Awareness of Public health resources, health practices, etc.
 Attitudes and Beliefs
 Preferences or evaluations: e.g., attitudes toward gays, ethnic
groups, etc., consumer preferences.
 Beliefs about political or social events: “which party provides the
strongest security for the U.S….?”
 Feelings or moods: quality of life, depression / anxiety, marital
satisfaction, etc.
 Behavior
 Behavioral intentions; Intent to vote, financial plans, etc.
 Self-reports of previous or on-going behavior; topics range from
voting to alcohol and drug use.
Psychology 242, Dr. McKirnan
Survey research
3
Psychology 242
Introduction
to Research
Survey research;
General uses of surveys
 Descriptive
 Epidemiology; e.g., drug use surveys, diet surveys
 Political / social description; e.g., opinion polls
 Test hypothesis
 Assess blocking variables: e.g., gender differences in political /
social attitude.
 Correlational studies; e.g., correlates of dieting…
 Examine generalizability of experimental results
 E.g., Consumer reports survey on therapy
 Predict event or outcome;
 E.g., election polling
 Pragmatic / applied,
 E.g., marketing, or consumer surveys.
Psychology 242, Dr. McKirnan
Survey research
4
Psychology 242
Introduction
to Research
Surveys; populations
 Who do we want to generalize to ?
 Our sampling frame is based on our hypothesis or
empirical question.
 Sampling: breadth  internal validity tradeoff
 Key dimensions:
 Demographic
 ethnic / age / gender groups,
 “all Americans”…
 Behavioral
 “likely voters”, alcohol users, home buyers...
 Self-identification
 Republicans / Democrats,
 “students”…
See Design and sampling overview See also: diminishing validity of political polling.
Psychology 242, Dr. McKirnan
Survey research
5
Psychology 242
Introduction
to Research
Question Formats: Closed-ended
items

Specific rating scale or highly structured prompts

Most reliable for concrete behaviors

An attitude can be assessed in several ways:

Direct (face valid) assessment
Psychology 242 is a wonderful course…

Does not 1 2 3 4 5 6 7 strongly
agree at all
agree
Behavioral (content valid) indictors
How many times this semester have you skipped 242 class?
How many hours per day do you spend reading 242 material?

6
0
______
2.5
______
Researchers typically use the M of several related
items to create a more reliable measure of a variable.
Psychology 242, Dr. McKirnan
Survey research
Psychology 242
Introduction
to Research


7
“Closed-ended” items, cont.
Example: CES-D Depression inventory:
We may use the M score of these 9 items as our
depression index…
Moods & Feelings
Below is a list of different feelings. Circle the number that shows how many days you felt each
of these over the PAST WEEK.
I was bothered by things that usually do
not bother me.
I felt I could not shake off the blues even
with help from my friends or family.
I had trouble keeping my mind on what
I was doing.
I felt depressed.
I felt that everything I did was an effort.
My sleep was restless.
I was happy.
I enjoyed life.
I felt sad.
Rarely or
none of
the time
A Little
of the Time
A moderate
amount of
the time
Most or all of
the time
(less than 1 day)
(1 or 2 days)
(3 - 4 days)
(5 - 7 days)
0
1
2
3
0
1
2
3
2
3
2
2
2
2
2
2
3
3
3
3
3
3
0
1
Sum
of item
ratings
/19
0
0
0
0
0
0
1
1
1
1
1
These items are “reversed”
in the final score
Psychology 242
Introduction
to Research


8
“Closed-ended” items, cont.
Or we may count the number of symptoms
We may have a cut point: e.g., moderate
depression is defined as 4+ symptoms
Moods & Feelings
Below is a list of different feelings. Circle the number that shows how many days you felt each
of these over the PAST WEEK.
I was bothered by things that usually do
not bother me.
I felt I could not shake off the blues even
with help from my friends or family.
I had trouble keeping my mind on what
I was doing.
I felt depressed.
I felt that everything I did was an effort.
My sleep was restless.
I was happy.
I enjoyed life.
I felt sad.
Rarely or
none of
the time
A Little
of the Time
A moderate
amount of
the time
Most or all of
the time
(less than 1 day)
(1 or 2 days)
(3 - 4 days)
(5 - 7 days)
0
1
2
3
0
1
2
3
0
1
2
3
1
1
1
1
1
2
2
2
2
2
2
3
3
3
3
3
3
# of symptoms:
items0 rated 2 1or 3
0
0
0
0
0
Psychology 242
Introduction
to Research


Closed-ended items, summary
Chief virtue: clear operationalization

Specific & concrete; we know exactly what the
participant is responding to

Easy to quantify & use statistically

Can be tested for reliability
Chief liability: potential insensitivity

Brief & simply worded; potentially superficial

“Top down”; issues are imposed on the participant


Discrimination studies: no option for “has no attitude”
Attitudes / moods: not sensitive to participants’ personal
perspectives (?).
Psychology 242, Dr. McKirnan
Survey research
9
Psychology 242
Introduction
to Research
Survey formats;
“Open-ended” items
General textual / qualitative response;

More sensitive to the respondent
How have you enjoyed your methods class so far?
Please list the three things that first come to mind when you
think of Psychology 242.

More difficult to interpret

Can be analyzed as qualitative data (see discussion in Descriptive data.)

Can be quantified;
frequency counts of citations or statements
 “linkages” analysis of co-occurring statements


Often presented as textual portrayal plus minor
quantitative analysis.
Psychology 242, Dr. McKirnan
Survey research
10
Psychology 242
Introduction
to Research
11
Example of mixed survey formats
Example of mixed question
format from survey of
women’s sexual practices.

Closed-ended attitude scale

Open-ended description

Simple behavioral index.
Personal Safer Sex Guidelines
How strict are your personal guidelines or rules for safer sex (e.g., condom use,
“safe relationships,” etc.)?
1
Not at all
Strict
2
3
Somewhat
Strict
4
5
6
Very
Strict
7
Extremely
Strict
What are your rules for safer sex?
Have you ever refused to have sex
with someone to stay safe?
Psychology 242, Dr. McKirnan
Survey research
0
1
2
3
Never
once or
twice
a few
times
many
times
Psychology 242
Introduction
to Research
SUMMARY




12
Survey topics & item types
Surveys assess:

Knowledge

Attitudes or preferences

Ongoing or intended behavior
Closed-ended formats
Psychology 242 is a wonderful course…

Highly structured, easy to analyze

Potentially insensitive
Open-ended formats
…list the three things that first come to mind…

More sensitive to the participant

Potentially ambiguous or difficult to analyze
Surveys typically…

Use multiple items

Employ several formats.
Psychology 242, Dr. McKirnan
Survey research
Psychology 242
Introduction
to Research
13
Psychology 242, Dr. McKirnan.
Survey research
 Topic areas & formats
✓
 Testing Hypotheses
 General issues in Survey research
 Sources of bias (or fraud…)
 Examples of surveys
Psychology 242, Dr. McKirnan
Survey research
3/10/14
Psychology 242
Introduction
to Research
Alcohol: Knowledge 1
What is the chemical in
alcohol?
Let’s do our own practice survey!
a) ethanol
b) zymase
c) dehydrogenase
d) nicotine
Psychology 242, Dr. McKirnan
Survey research
14
Psychology 242
Introduction
to Research
Alcohol: Knowledge 2
Which of these drinks
contains the least
alcohol?
A = 12 ounces of beer
B = 5 ounces of wine
C = 1 1/2 ounces of vodka
D = 8 ounces of malt liquor
E = All are equal
Psychology 242, Dr. McKirnan
Survey research
15
Psychology 242
Introduction
to Research
Alcohol: Knowledge 3
How long does it take for
alcohol to affect the
brain?
a) 10 seconds
b) 60 seconds
c) 10 minutes
d) 90 minutes
Psychology 242, Dr. McKirnan
Survey research
16
Psychology 242
Introduction
to Research
Alcohol attitudes, 1
Drinking helps me feel
whatever way I want to
feel.
A = agree strongly
B = agree
C = somewhat agree
D = slightly agree
E = do not agree
Psychology 242, Dr. McKirnan
Survey research
17
Psychology 242
Introduction
to Research
Alcohol attitudes, 2
Alcohol makes me feel
happy.
A = agree strongly
B = agree
C = somewhat agree
D = slightly agree
E = do not agree
Psychology 242, Dr. McKirnan
Survey research
18
Psychology 242
Introduction
to Research
Alcohol attitudes, 3
Drinking adds a certain
warmth to social
occasions.
A = agree strongly
B = agree
C = somewhat agree
D = slightly agree
E = do not agree
Psychology 242, Dr. McKirnan
Survey research
19
Psychology 242
Introduction
to Research
Alcohol behavior, 1
How often do you have a
drink containing
alcohol?
A = never
B = monthly or less
C = 2 to 4 times a month
D = 2 to 3 times a week
E = 4 or more times a week
Psychology 242, Dr. McKirnan
Survey research
20
Psychology 242
Introduction
to Research
Alcohol behavior, 2
How many drinks
containing alcohol do
you have on a typical
day when you are
drinking?
A=1
B=2
C=3
D = 4 or 5
E = 6 or more
Psychology 242, Dr. McKirnan
Survey research
21
Psychology 242
Introduction
to Research
Alcohol behavior, 3
How often during the last
year have you found that
you were not able to stop
drinking once you had
started?
A = never
B = less than monthly
C = monthly
D = weekly
E = daily or almost every day
Psychology 242, Dr. McKirnan
Survey research
22
Psychology 242
Introduction
to Research
Alcohol behavior, 4
How often during the last
year have you failed to do
what was normally
expected from you
because of drinking?
A = never
B = less than monthly
C = monthly
D = weekly
E = daily or almost every day
Survey research
Psychology 242, Dr. McKirnan
23
Psychology 242
Introduction
to Research
24
Knowledge / attitude / behavior composites
Here
is our
survey.
Our data
set:
Knowledge
 Several items assess each
Hypothetical Construct.
 Which of these drinks contains the least alcohol?
 Typically we also collect
 How long does it take for alcohol to affect
the brain? (age, gender,
demographics
ethnicity…) and other Ψ
Attitudes and beliefs
variables.
 Drinking helps me feel whatever wayI want
to feel. items based on our
We choose
 What is the chemical in alcohol
 Alcohol makes me feel happy.
Theory about alcohol use and
problems.
 Drinking adds a certain warmth to social occasions.
 We can simply Describe our
Behavior
data in each category.
 How often do you have a drink containing alcohol?
 We might Block the data by
 How
many
drinks
containing
alcohol doimportant
you have categories
on a typicalofday
How
many
students
use
alcohol at
participants
all?when you are drinking?
Do
more the
positive
 men
How have
often aduring
last year have you
found
thatTest
you Hypotheses
were not
 Or
we can
attitude
alcoholonce
than you had started?
able totoward
stop drinking
about correlations among
women?
variables
 How often during the last year have you failed to do what was
normally expected from you because of drinking?
Survey research
Psychology 242
Introduction
to Research
Knowledge / attitude / behavior composites
Knowledge
25
Research questions:
 How well do the items within each category
 What is the chemical correlate?
in alcohol
 Which of these drinks
least alcohol?
• contains
Are they the
internally
Reliable?
 How long does it take
alcohol
to affectathe
brain?
• for
Can
we construct
Composite
Variable from
Attitudes and beliefs
each item set?
Blocking
: Gender?
 Drinking helps me 
feel
whateverVariables
way I want
to feel. Ethnicity? Age?...
What
are the Correlations among composite
 Alcohol makes mefeel
happy.
variables?
 Drinking adds a certain warmth to social occasions.
• How much are knowledge or attitudes & beliefs
associated with alcohol use?
Behavior
• Are
alcohol
use andalcohol?
problems correlated?
 How often do you have
a drink
containing
• Are these
effects
Mediated
byaother
variables?
 How many drinks containing
alcohol
do you
have on
typical
day
when you are drinking? Do depression, isolation, or stress make people
vulnerable
to alcohol
problems?
 How often during the last year
have you
founduse
thatoryou
were not
able to stop drinking once you had started?
 How often during the last year have you failed to do what was
normally expected from you because of drinking?
Survey research
Psychology 242, Dr. McKirnan
Psychology 242
Introduction
to Research
Class data, n = 97
Multiple items assessed the Hypothetical Constructs of
Alcohol- related information, attitudes, use, and problems.
 Information: # of correct answers for items 1  3.
 Attitudes: items 4, 5 & 6.
 Use: drinking frequency * # drinks / occasion (items 7 & 8).
Example
 Problems: Items 9 & 10.
Information:
Mode = 2 items correct.
Psychology 242, Dr. McKirnan
Survey research
26
Psychology 242
Introduction
to Research
Class survey data, Attitudes
 To test weather the 3 attitude items comprised a single
Hypothetical Construct, we examine their
intercorrelations
Example
 All the correlations and high and statistically significant,
so we can combine them into a single composite variable.
Psychology 242, Dr. McKirnan
Survey research
27
Psychology 242
Introduction
to Research
Class survey data, Attitudes
 The frequency distribution for the composite attitude
variable shows an approximate normal distribution.
Example
 The Mean and Median are
equal, near the center of the
distribution
 The mode reflects a larger
block of participants with
moderately strong attitudes.
28
Psychology 242
Introduction
to Research
Class survey data, Alcohol problems
 We perform the same operations for items 9 & 10, the alcohol
problems measures.
 The correlation between these items is very high.
Example
 We can validly combine them into one variable.
29
Psychology 242
Introduction
to Research
Class survey data, Alcohol problems
 Most people report no problems, as shown in the Mean and
Median
 The rest of the distribution shows a strong positive skew.
Example
 This skew pulls the Mean higher than the Mode & Median.
30
Psychology 242
Introduction
to Research
Class survey data, Correlation results
 The amount of information people have about alcohol is
unrelated to their alcohol use or problems:
 The correlations are low;
 Neither is statistically significant.
 Positive attitudes toward alcohol are strongly associated with
alcohol use.
Example
 … and moderately (statistically significantly) with alcohol problems.
 Thus, attitudes are more important to alcohol abuse than is
knowledge.
31
Psychology 242
Introduction
to Research
32
Correlations and mediators
With more data we could use mediating analyses to
develop to test theories about alcohol use / problems

Say we find that people who are lonely or depressed (negative
affect) have more alcohol use & problems.

Why might this be?

People with negative affect may come to expect (hope?) that
alcohol will make them feel better.

The attitude may be what actually leads to alcohol use.

Here attitudes / beliefs would mediate the effect of negative
affect on alcohol use & problems.
Positive
attitudes
toward
alcohol
Depression
loneliness
stress
Psychology 242, Dr. McKirnan
Survey research
Alcohol
use &
Problems
Psychology 242
Introduction
to Research
33
Correlations and mediators
Using mediating analyses to develop to test theories
about alcohol use / problems


Say we find that people who are lonely or depressed (negative
 use
Using
a mediating analysis to ask
affect) have more alcohol
& problems.
“why” is the core step in Theory
Why might this be?
Development.
 People with negative affect may come to expect (hope?) that
alcohol will make them feel better.

The attitude may be what actually leads to alcohol use.

Here attitudes / beliefs would mediate the effect of negative
affect on alcohol use & problems.
Positive
attitudes
toward
alcohol
Depression
loneliness
stress
Psychology 242, Dr. McKirnan
Survey research
Alcohol
use &
Problems
Psychology 242
Introduction
to Research
34
Correlations and mediators
Survey data are important for developing or testing
new theories of behavior

Can have high external validity

Can model different relations among variables.
Problems?

The data are correlational and cross-sectional.

The causal arrows may go in different directions.
Depression
loneliness
stress
Psychology 242, Dr. McKirnan
Positive
expectation
of alcohol
Survey research
Alcohol
use &
Problems
Psychology 242
Introduction
to Research
35
Correlations and mediators
Survey data are important for developing or testing
new theories of behavior

Can have high external validity

Can model different relations among variables.

The causal arrows may go in different directions.

There may be a third variable that we did not measure.
Genetics:
Positive
alcohol effects
Problems?
& negative

The data are correlational
and cross-sectional.
affectivity
Depression
loneliness
stress
Psychology 242, Dr. McKirnan
Positive
expectation
of alcohol
Survey research
Alcohol
use &
Problems
Psychology 242
Introduction
to Research
36
cures
Cures for correlation problems?

Longitudinal studies


Examine changes over time to
better determine cause and
effect
Experimental studies

Simulate and manipulate the
predictors, e.g.;
 Induce temporary stress /
negative affect
 Assess attitudes
 Assess alcohol use
Psychology 242, Dr. McKirnan
Survey research
Psychology 242
Introduction
to Research
37
We Use Longitudinal data
 To examine change
 To approximate “cause and effect”

Research has shown that
depression, loneliness &
stress are correlated.

So, we combine them into a
single composite variable.

This is our Predictor
Depression
Loneliness
Negative
Affect
Stress
(Measurement version of IV)


The same for alcohol use
and problems.
This is our outcome
Alcohol
problems
(Measurement version of DV)
Psychology 242, Dr. McKirnan
Alcohol
use
Survey research
Alcohol
abuse
Psychology 242
Introduction
to Research


38
Longitudinal data: Path analysis
We test the hypothesis that Negative Affect
correlate with alcohol abuse & problems…

…at baseline

…and at follow-up
Data at each time frame
are cross-sectional:
 We cannot determine if
emotions “cause” the
alcohol outcome, or
visa versa.
Psychology 242, Dr. McKirnan
Survey research
Time 1
Time 2
(baseline)
(follow-up)
Negative
Affect
Negative
Affect
Alcohol
abuse
Alcohol
abuse
Psychology 242
Introduction
to Research

39
Longitudinal data: Path analysis
Looking at patterns across time lets us make more sensitive
statements.
Time 1
(baseline)
If the measures are
reliable Negative Affect at
Time 1 will correlate with
NA at Time 2.
Time 2
(follow-up)
Negative
Affect
Negative
Affect
And the same for the
alcohol variable
Alcohol
abuse
Alcohol
abuse
Psychology 242, Dr. McKirnan
Survey research
Psychology 242
Introduction
to Research

40
Longitudinal data: Path analysis
To test the hypothesis we look at the “cross” correlations…
Time 1
(baseline)
Negative
Affect
Alcohol
abuse
Psychology 242, Dr. McKirnan
How strongly does
Affect at baseline
correlate with Alcohol
at follow-up?
Time 2
(follow-up)
Negative
Affect
A strong correlation
would tell us that Affect
does lead to more
Alcohol abuse over time.
Survey research
Alcohol
abuse
Psychology 242
Introduction
to Research

41
Longitudinal data: Path analysis
To test the hypothesis we look at the “cross” correlations…
Time 1
(baseline)
Negative
Affect
Alcohol
abuse
Psychology 242, Dr. McKirnan
Similarly, how
strongly does Alcohol
at T1 correlate with
changes in Affect at
T2?
This correlation tells us
how much alcohol use
leads to negative affect
over time.
Survey research
Time 2
(follow-up)
Negative
Affect
Alcohol
abuse
Psychology 242
Introduction
to Research

42
Longitudinal data: Path analysis
Testing the hypothesis: does Negative Affect ‘cause’ Alcohol
use & problems
Time 1
(baseline)
Negative
Affect
If the correlation
between Affect at Time
1 and Alcohol at
follow-up is very
strong…
Alcohol
abuse
Psychology 242, Dr. McKirnan
Time 2
(follow-up)
Negative
Affect
Alcohol
abuse
Survey research
Psychology 242
Introduction
to Research

43
Longitudinal data: Path analysis
Testing the hypothesis: does Negative Affect ‘cause’ Alcohol
use & problems
Time 1
(baseline)
Negative
Affect
…and the correlation
between Alcohol at T1
and Affect at T2 is
relatively weak…
Alcohol
abuse
Psychology 242, Dr. McKirnan
Time 2
(follow-up)
Negative
Affect
Alcohol
abuse
Survey research
Psychology 242
Introduction
to Research

44
Longitudinal data: Path analysis
Testing the hypothesis

Time 1
(baseline)
Negative
Affect
…we can conclude
that Affect may be a
more important
“cause” of alcohol
use / problems than
the other way around.
Alcohol
abuse
Psychology 242, Dr. McKirnan
Time 2
(follow-up)
Negative
Affect
Alcohol
abuse
Survey research
Psychology 242
Introduction
to Research

45
Longitudinal data: Path analysis
Testing the hypothesis
Time 1
(baseline)
Of course we may find
that Alcohol “causes”
Affect, not the other
way around…
Negative
Affect
Alcohol
abuse
Psychology 242, Dr. McKirnan
Time 2
(follow-up)
Negative
Affect
Either way, we have a
better sense of cause
and effect than with
cross-sectional data.
Survey research
Alcohol
abuse
Psychology 242
Introduction
to Research

Summary: Testing Hypotheses
Surveys typically use multiple items to measure
each hypothetical construct

Summary



Correlations among items tell us if they are reliable in
measuring the same construct.
We use Mediating Analyses to

Test hypotheses about correlations between constructs

Build or test theory
Cross-sectional analyses are difficult to interpret

Causal direction?

3rd variable problem
Longitudinal analyses help us determine causal
direction
Psychology 242, Dr. McKirnan
Survey research
46
Psychology 242
Introduction
to Research
47
Psychology 242, Dr. McKirnan.
Survey research
 Topic areas & formats
 Testing Hypotheses
 General issues in Survey research
 Sources of bias (or fraud…)
 Examples of surveys
Psychology 242, Dr. McKirnan
Survey research
3/10/14
Forms of survey administration
Self-report questionnaire
Psychology 242
Introduction
to Research



Primarily closed-ended, structured questions

Limited open-ended items

Assume at least moderate reading level

Cheap & easy to administer


“Paper and pencil” or internet-based;
All data collection
increasingly
Internet: Representativeness very dubious
computer-based
Face-to-face interview

“Door step”, formal research center, or telephone

Allows in-depth qualitative questions

Many studies combine questionnaire & interview formats

(Telephone version becoming obsolete)
Psychology 242, Dr. McKirnan
Survey research
48
Psychology 242
Introduction
to Research

Cost / population access

Different methods are more / less likely to reach certain
populations, e.g.:




General issues in surveys
Disfranchised / poor populations often not reached by internet or telephone
Cell phones & avoidance of telemarketers  less availability for telephone
surveys
Stigmatized populations less available for face-to-face interviews, more
available via internet.
Participant sophistication

Participants may not be able to accurately report certain topics

Attitudes toward stem cell research from readings.

What factors are most important to your choice of political candidate....


Describe the amounts and types of proteins you eat during a typical
week...
“Rationality bias”; many questions (incorrectly?) assume a rational
reason for behavior:

Why do you have unsafe sex...

What is your chief reason for using alcohol each night…
Psychology 242, Dr. McKirnan
Survey research
49
Psychology 242
Introduction
to Research


Social Desirability Responding
Clear face-valid items addressing embarrassing topics yield
less valid responses

How often are you dishonest with your friends?

Have you ever cheated on an exam....?
High social desirability wording elicits inaccurate responses…

Do you support protecting our Nation’s forests for future generations?
(Does “yes” mean you an “environmentalist”?).

Do you feel there are ways your husband could be closer...?
(Does “yes” make you are unhappy in your marriage”?).

Populations differ in social desirability responding; may be a
confound in studying group differences
 Women report more suicidal thoughts, but may be more willing to
disclose, creating a possible confound…

50
Desirability can be minimized by:

Anonymous surveys

Assurances of confidentiality

Computer administration (no personal interaction)

Careful wording / pilot testing of items
Psychology 242
Introduction
to Research

Surveys: Social
desirability responding
Do people lie on surveys?

Men routinely report more sex partners
than do women.

If the sample is unbiased by gender,
partners should balance for men &
women.

Social desirability hypothesis:

Women underestimate partners

Men overstate partners

Much of the difference due to:
 A high proportion of women who report 1 partner
 A few men who report many partners.

Possible sample bias (confound?) in who responds to such
surveys?
Click for the article
51
Psychology 242
Introduction
to Research

General issues in surveys: Time Frames
Rare(er) events require a long time frame to assess


When was your last doctor’s visit…
These questions asses the last time you left a romantic
relationship…
 Longer term recall can be surprisingly unreliable
 Recall of last doctor visit highly unreliable when checked against
medical records

Shorter time frame yields more reliable responding
 Memory is better for more recent effects
 “Exit interviews” from medical visits far more reliable than even 2week retrospective measures.
 Current, concrete behaviors are more accurately reported than
behavioral trends.
 In general, how often do you miss a dose of your medication
Less reliable than…
 Let’s go over each of the past 7 days and tell me if you took or
missed your medication dose…
Psychology 242, Dr. McKirnan
Survey research
52
Psychology 242
Introduction
to Research

General issues in surveys: Question Order
Questions trigger participant’s memory or attention,
and can bias questions that follow, e.g.:
Do you think Social Security & Medicare payments have kept up
with inflation..
then
Do you favor or oppose Democratic efforts to expand Medicare
payments...

Bias can be limited by counterbalancing questions.

Using different question orders in different versions of the
survey.
Psychology 242, Dr. McKirnan
Survey research
53
Psychology 242
Introduction
to Research

Summary

General Issues Summary
Survey administration

Internet increasingly important as self-report method

Face-to-face interviews more common in clinical research

Time frames & question order can influence responses
Population access & sophistication

Some groups are difficult to reach

Creates threat to External validity


54
Assumption that participants understand survey materials
often questionable.
Social desirability responding


Inhibited responding threatens Internal Validity
May represent a confound if groups differ in desirability
set.
Psychology 242
Introduction
to Research
55
Psychology 242, Dr. McKirnan.
Survey research
 Topic areas & formats
 Testing Hypotheses
 General issues in Survey research
 Sources of bias (or fraud…)
 Examples of surveys
Psychology 242, Dr. McKirnan
Survey research
3/10/14
Psychology 242
Introduction
to Research


Bias / Fraud in survey research
Social research is increasingly important to political &
cultural debates.

Effects of gay marriage

Political “approval” ratings

Scientific consensus on global warming

Research on working mothers …
Pressure for confirmatory results encourages bias or
outright fraud (see this week’s article on Opinion Polls).


In the study structure

Items used

Sample
In the interpretation of results

“Cherry picking”

Simple distortion
56
Psychology 242
Introduction
to Research

Opposition to gay marriage


Example



57
Example of fraudulent survey use
Judges have struck down religious doctrine as a basis for
disallowing gay marriage
Opponents have sought evidence of civil
harm to justify discrimination
Harm to children has emerged as key
issue in the debate
APA; multiple studies show no harm.
April DeBoer, left, and Jayne Rowse on
Friday after closing arguments in their
challenge to Michigan’s marriage
restriction. Credit Mandi Wright/Detroit Free Press,
Mark Regenerus’ Survey: Children in
via Associated Press
households with gay/lesbian parent
fare worse.
 Funded & cited widely by gay marriage opponents
 Study sample and interpretation of results wildly biased
 Has been disavowed by Renenerus’ Academic Department and
American Sociological Assn.
 Continues to be central “evidence” in court cases
Psychology 242
Introduction
to Research

58
Source of survey fraud; question wording can elicit a
response desired by the researcher;


Bias in survey research: Leading or biased items
How much do you support the administrations’ actions to protect
you and your children from terrorists…
Wording can “normalize” a response, e.g.,

When do you feel that it is O.K. to cheat on an exam?
..when I really do not know the material
.. when others are doing it
.. when I think the exam is unfair

Vague wording can be interpreted in a biased fashion
Is there anything your husband could do to be more intimate with you?

“Push” polls: a survey can be used to actually create an
attitude.
Psychology 242, Dr. McKirnan
Survey research
Psychology 242
Introduction
to Research
Push Polls:
59

An item “traps” the participant into endorsing a specific view.

When the “data” are released the biased wording is ignored.



Publicity about the “findings” (e.g., by politically biased news
organizations) are used to further create or change attitudes.
Many political & social organizations use this strategy to…

Ostensibly measure attitudes objectively

Use the “results” to influence popular opinion.
Example of a highly biased survey: The Republican National
Comittee health care survey.

This “survey” attempted to create fear of health care reform.

It got limited distribution, but is a great example of a Push Poll
Psychology 242, Dr. McKirnan
Survey research
Psychology 242
Introduction
to Research


GOP health survey (2009 – 2010)
Some of these items are simple lies – or manipulations based
on lies – designed to induce anti-health care attitudes…
Other are powerful (and dishonest) emotional manipulations
✓
✓
✓
✓
Psychology 242, Dr. McKirnan
Survey research
60
Psychology 242
Introduction
to Research


Forms of survey bias:
Provide leading or emotionally manipulative
information to induce an attitude rather than simply
measure it, to provide politically useful “data”..
Questions that, if you accept their assumptions, can
only be reasonably answered one way…
Psychology 242, Dr. McKirnan
Survey research
61
Psychology 242
Introduction
to Research

Biased surveys: Democratic example
An example from
the Democrats,
that is also used
for fund-raising.
Psychology 242, Dr. McKirnan
Survey research
62
Psychology 242
Introduction
to Research
Democratic biased survey (2007)
Manipulative
presentation
of
questionable
information
Simple
emotional
manipulation
Distorted
description
that may be
changed in
presentation
of findings
“Who could
disagree”
item.
Psychology 242, Dr. McKirnan
Survey research
63
Psychology 242
Introduction
to Research
Summary: Manipulating attitudes by surveys
1. Ask manipulative or highly leading questions
Summary
2. Find high levels of agreement (and potentially
change participants’ attitudes).
3. Publicize – and often distort or overstate –
the “findings” via highly biased news
sources
4. News reports themselves lead to attitude
change among people who are uncertain or
uninformed.
Psychology 242, Dr. McKirnan
Survey research
64
Psychology 242
Introduction
to Research
65
Psychology 242, Dr. McKirnan.
Survey research
 Topic areas & formats
 Testing Hypotheses
 General issues in Survey research
 Sources of bias (or fraud…)
 Examples of surveys
 By key elements of survey design
Survey research
3/10/14
Psychology 242
Introduction
to Research
66
Examples of surveys & data, 1
Consumer reports survey of mental health care
question
• Satisfaction
with therapy.
• Differences
between types
of therapy.
Psychology 242, Dr. McKirnan
population
sample
data
findings
Self-Identified
group:
Self-selected
convenience
sample:
Attitudes &
behavior
Descriptive &
hypothesis
tests
U.S. mental
health care
users
Readers who
got therapy &
returned the
survey, n=4000
Self-report
questionnaire,
cross-sectional
High
satisfaction
for most
treatments
Survey research
Psychology 242
Introduction
to Research
67
Examples of surveys & data, 2
“Monitoring the future” youth studies
question
• Social
behavior
• Academics
• Alcohol &
drug use
• Health.
Psychology 242, Dr. McKirnan
population
sample
data
findings
Demographic
group:
Random
sample:
Knowledge,
attitudes and
behavior
Mostly
descriptive
All U.S. youth,
15 -> 21 years
old.
Sample of High
School health
classes,
n=3000 -> 5000.
Face--to--face
interviews &
questionnaires,
longitudinal
(bi-yearly)
Assess yearly
trends/shifts in
drugs, grades,
emotional well
being
Survey research
Psychology 242
Introduction
to Research
68
Examples of surveys & data, 3
Gallup, Time/CNN, other polls
question
populations
samples
data
findings
Demographic:
National,
random
Knowledge,
Attitudes,
Behavior
Descriptive
Digit dial
telephone,
n=150 to >500
Brief interview,
cross-sectional
Ratings of
politicians,
Consumer
preferences
Approach to
Affordable
Care Act
- Eligible voters,
Political
opinions,
Lifestyle
information
Social
attitudes, e.g.,
managed care
- Target age
groups
Self-identified:
“Democrats”…
Behavioral:
Voters, ACA
users…
General:
- U.S. adult
population
Psychology 242, Dr. McKirnan
Survey research
Psychology 242
Introduction
to Research
69
Examples of surveys & data, 4
Exit polls
population
sample
data
findings
U.S. electorate
Probability
Self-reported
behavior
Descriptive/
predictive
National and/or
local electoral
district
Stratified
random sample
of electoral
districts.
Self-report
interview,
cross-sectional
Increasingly
inaccurate
predictions
question
Election
outcome,
possibly
stratified by
state / region
See reading on shifts in use of polling data in U.S. politics
Psychology 242, Dr. McKirnan
Survey research
Psychology 242
Introduction
to Research
70
Examples of surveys & data, 5
”Social Issues Survey” of Chicago gay / lesbian
community
question
- Stress &
coping
- Alcohol &
drug use
- Responses to
HIV / AIDS
Psychology 242, Dr. McKirnan
population
sample
data
findings
Self-identified
Targeted multiframe
Attitudes &
behavior
Descriptive &
hypotheses
Self-identified
gay, lesbian, &
bisexual adults
in Chicago.
community
newspapers,
organizations,
& mailing lists,
n=3500
Self-report
questionnaire,
cross-sectional
- High
experience of
discrimination
- Less stress &
alcohol-drug
use than
expected
Survey research
Psychology 242
Introduction
to Research
71
Examples of surveys & data, 6
National Institute on Drug Abuse Household survey of
Alcohol and Drug use
question
Alcohol-drug
use and
problems,
treatment use,
health effects.
population
sample
data
findings
National
Random Multistage:
Knowledge,
attitudes &
behavior
Typically
descriptive
All U.S. adults
1. Census tract
2. Household,
3. Any adult
member
Face - to - face
Interview,
successive
cross-sectional
(each 5 years)
Age & regional
differences in
substance use,
trends over
time in use &
problems
n>4000
 Data often used for hypothesis-oriented 2nd
analyses (i.e., as archival data).
Psychology 242, Dr. McKirnan
Survey research