Diana C. Mutz
University of Pennsylvania
Random sampling + Random assignment =
Population-Based Experiments
Goal to increase methodological innovation
and to expand the diversity of methods used
across the social sciences
Particularly studies that benefit from
combining the internal validity of experiments
with the external validity of representative
population samples.
NSF-supported infrastructure project that
provides social scientists with
opportunities for innovative data
collection
 TESS uses two large-scale data collection
instruments

 Internet surveys
 Telephone Interviews

Our goal is to increase methodological
innovation and to expand the diversity of
methods used across the social sciences.
Experiments as the Gold Standard for
establishing causality:
“There are no complex statistical
techniques that provide the power and
elegance of a simple experimental
design."

“College sophomores may not be people.” (C.
Hovland).

“A greater effort must be made to conduct research
on persons from life stages other than late
adolescence.”

“… I suspect that even middle-aged people,
separated from family and friends and confronted
with test-like materials on novel and artificial topics in
an academic laboratory, would often behave like
college students do.” (D. Sears).




Allowed extensive pretesting.
Investigators can listen in on pretest
interviews, and revise accordingly.
Respondents are sent pre-survey
incentives and are paid for their
interviews.
Timing can be coordinated with
Internet data collection.
Address-based sampling (to include cellphone-only households) for recruitment.
 Free notebook computer and internet
access if needed so both internet and
non-internet households are included
 Payment for study participation
 Special subpopulations known in
advance
 Multimedia: Pictures/Video/Audio
 Rapid Turnaround






Decreasing investigator demand for
this mode of data collection.
Limitations on experimental stimuli
More difficult to recontact the same
respondents (greater attrition)
Evidence on reliability of measures
generally better via internet
Ability to randomly sample
specialized populations

Any faculty member or graduate student in
any social science department anywhere in
the world.

Proposals may come from any substantive
area in the social sciences as long as they
utilize experimental or quasi-experimental
designs.

Repeated use is encouraged.
Researchers to build on previous findings in
subsequent proposals.
Any experiment or quasi-experimental study
designed to explore substantive or
methodological hypotheses within the social
sciences.
 Limits of request for sample and number of
questions
 Must make a clear contribution to
knowledge.
 Particularly studies that would benefit from
combining the internal validity of experiments
with the external validity of representative
population samples.


Proposals are limited to 5 pages in
length.
 Minimal investment of time in proposal
preparation
 Encourages fast turnaround from reviewers

Proposals are submitted through an easy
online process.
 With just a few clicks of the mouse, your
proposal enters the review process.
Diverse Associate PIs oversee the review process…
Diversity of topics reduces potential contamination
of respondents.
Investigators need not share a survey interview.
Capitalizes on economies of scale to
provide more data to more investigators
than would otherwise be possible for the
same cost.
 Encourages social scientists to use
methodologies they would not otherwise
use.
 Encourages quicker pace of research
progress
 Promotes more policy-relevant research

1.
Experimental designs on policy-relevant
populations
Experimentally
investigated what
preventative care
benefits were most
important to consumers
Respondents:
 500 with a minor
present in their home
 500 without a minor
present in their home
Respondents choose among three health plans, with the
possibility of opting into special vaccination coverage.
Respondents randomly assigned to receive threatening
or non-threatening feedback on a general knowledge
test or a “masculine knowledge test:”

When it is sold in the US, what country does the beer Heineken come
from?

What are the true measures of a “two by four”?

Who had back-to-back MVP seasons in the NFL besides Joe Montana?

Which filter on a car typically requires changing every 3000 miles?

What is considered a good maximum benching weight?

Which tool clamps onto an object and will stay clamped without being
held in place?
The conventional wisdom about infidelity
based on student samples:

Men value sexual fidelity, whereas women value
emotional fidelity.
The finding when research is based on a
representative sample of adults:

Both men and women show more anger and
blame over sexual infidelity, and both men and
women show greater hurt feelings over emotional
infidelity.
Do OCD screening batteries work less well for
African-Americans?
If so, why?
2 by 2 experiment with representative samples
of European-Americans (50%) and AfricanAmericans (50%), asked about ethnic and
racial identity either before the OCD
battery (to increase salience of race), or
after the scales were asked.
1) Same racial difference in average scores in
representative samples as in college student sample.
2) Increasing the salience of race increases AfricanAmericans’ OCD screening scores.
1.
2.
Experimental designs on policy-relevant
populations
Using advance knowledge of respondent
characteristics
1.
2.
Experimental designs on policy-relevant
populations
Using advance knowledge of respondent
characteristics


Representative sample of participants drawn
from the population of individuals whose
family income is less than $30,000/year
Representative sample of people residing in
coastal areas
1.
2.
3.
Experimental designs on policy-relevant
populations
Using advance knowledge of
respondent characteristics
More complex, subtle and engaging
interactions with respondents
A person is “laid off” because his or her company had
to reduce its staff.
“Think for a moment about the person and then tell
me how much government help, if any, that
person should receive while looking for a new job.”
One vignette, 96 variations:
The person is a [white/black] [male/female] in
[his/her] [early 20’s/mid 30’s/early 40’s]. [He/She] is
[single/a single parent/married/married and has
children] and [is/is not] a dependable worker.
Random samples of Black and White men
and women evaluate Black and White,
male and female models.
Asked to indicate when model becomes
“overweight,” and when “obese”
Results:
Female image perceived as overweight
and obese at significantly smaller body
sizes than the male image.
Race of image did not matter, but race of
perceiver did; White men judged bodies
as obese and overweight at smaller
body sizes than Black men
Does seeing the aftermath of natural
disasters affect willingness to insure
against them?
Tornadoes, Hurricanes, Fires
Disentangling direction of causality:
Perceived quality of neighborhood
Level of social organization, interaction
Do the deaths of private security personnel
(i.e., “contract soldiers”) affect
Americans less than deaths of military
soldiers?
Do they elicit less of a response?
Do they help insulate support for the war
from the death toll?
1.
2.
3.
4.
Experimental designs on policy-relevant
populations
Using advance knowledge of respondent
characteristics
More complex, subtle and engaging
interactions with respondents
Repeated contact over time
Did Sotomayor’s confirmation change
Hispanic attitudes toward government?
Toward the Supreme Court?
Three groups interviewed after nomination,
but before confirmation, then again
after confirmation
Spanish-speaking Hispanics
English-speaking Hispanics
Non-Hispanic Whites
.500
Mean Government Fairness
.400
.300
.200
.100
.000
-.100
-.200
Nominated, but not confirmed
Post-confirmation
Spanish-speaking Hispanics
English-speaking Hispanics
Non-Hispanic Whites
Mean Favorability Toward Supreme Court
.800
.700
.600
.500
.400
.300
.200
.100
.000
-.100
-.200
Nominated, but not confirmed
Post-confirmation
1.
2.
3.
4.
5.
Experimental designs on policy-relevant
populations
Using advance knowledge of respondent
characteristics
More complex, subtle and engaging
interactions with respondents
Repeated contact over time
“Firehouse” studies in response to
unfolding events

Respondents:
 Georgia Residents
 Half of Respondents,
African-American
 Half of Respondents,
White
1.
2.
3.
4.
5.
6.
Experimental designs on policy-relevant
populations
Using advance knowledge of respondent
characteristics
More complex, subtle and engaging
interactions with respondents
Repeated contact over time
Firehouse studies
Improving measurement, especially sensitive
issues and behavioral outcomes



Why such large discrepancies between
the self-reports of heterosexual men and
women?
Social desirability?
Memory strategies?
 Men - top of head multiplication
 Women – enumeration of individuals
Men
Women
18
16
14
12
10
8
6
4
2
0
Top of head
Enumerate individuals
Control group
Ball tossing game to induce feelings of rejection
and social ostracism.
Source of ideas
and examples for
population-based
experimental
designs
Six thousand respondents answer two
questions each.
 Here is a list of things some people
have done, please tell us how many
you have done:

 Written a letter to an editor of a newspaper
 Called into a talk radio program
 Visited a country outside of the United
States

Bodenhousen, “The role of gender in mental
illness stigma”

Corrigan, “Shame, Blame and
Contamination: Mental Illness Stigma and
the Family”

Use of between subjects design, large
samples, and anonymous internet surveys
help acquire sensitive data
2 (masculine vs. general knowledge test) x
2(threatening vs. non-threatening feedback)
experimental design.
Rs complete a 10-question test. They are randomly
assigned to the masculine-knowledge test or the
general-knowledge test.
After the test, Rs receive feedback on how they
did. Rs are randomly assigned to receive
threatening feedback or non-threatening
feedback.
Dependent measures involve a series of questions
Masculine Knowledge Test
When it is sold in the US, what country does the
beer Heineken come from?
What are the true measures of a “two by four”?
Who had back-to-back MVP seasons in the NFL
besides Joe Montana?
Which filter on a car typically requires changing
every 3000 miles?
What is considered a good maximum benching
weight?
Which tool clamps onto an object and will stay
clamped without being held in place?

Proposers may designate the field within
which they would like their proposal
reviewed.

After reading the proposal, a TESS Associate
PI selects two reviewers appropriate to the
topic and methods.

The reviewers are sent the anonymous
proposal via email and respond within two
weeks.

Continuous panel recruitment

Rotating panel with tenure of approximately 3
years
Maximum of 1 survey per week; usually 3
surveys per month involving a broad spectrum
of topics and customers
 Average 10-15 survey minutes in length per
week, with some surveys of 25-35 minutes with
incentives
 Sample selection controls for prior survey taking
to minimize panel conditioning
