Online panels for health
surveys
Charles DiSogra
Academy Health, June 14, 2011
© 2011 Knowledge Networks, Inc.
Outline
Panels and types of online panels
Frames and accuracy
Innovations in address-based sample (ABS) recruitment
Calibrating probability and non-probability samples
Example of a health tracking study
1
Panels
Panels are pre-recruited persons that have agreed to
participate in multiple surveys over time
Examples: The U.S. Current Population Survey
Longitudinal surveys with repeated measures
A panel can be a “true” sample of
the population … or not.
Depends on: how constructed
who is recruited
2
Online panels
The similarity (mode):
Web administered questionnaires
The difference (frame):
How panels recruit members
is not the same
Key Questions:
 What population does this panel represent?
 Are survey findings generalizable?
3
Types of online panels
Non-probability opt-in panels
Probability-based panels
4
Non-probability opt-In Web panels
 Large, volunteer membership – in the millions
 People on the Web join through ads, aggregator Websites, pop-up
invitations, e-mail marketing
 Used extensively by market researchers
Low cost
Rapid turnaround
Can target defined audiences with member profile data
 Recruitment, sampling, weighting methods are not transparent
 Survey completion rates are low (2-9%)
 Not generalizable for prevalence estimates – but people do this all
the time!
 Industry organizations, e.g., Advertising Research Foundation,
set voluntary standards for membership management
(i.e., minimize professional respondents, overlap among panels)
5
Probability-based Web Panels
 Recruited with probability samples (no non-sampled volunteers)
Random-digit dial (formerly)
RDD dual frame samples add cell phone augment
Address-based sampling (now more common)
 Members have known selection probability (adj. in base weights)
 Used by government, academic and non-profit researchers and
private companies where more rigor is desired
 Completion rates are high (65-70%)
 Results are generalizable, can calculate prevalence estimates with
applicable confidence intervals
 American Association of Public Opinion Research recognizes
probability based samples, ergo panels recruited as such, as a
valid and reliable survey method
6
50,000 members
representing
America
Probability-based recruitment, representative of U.S. adult
population
Includes:
 Households with no Internet access at time of recruitment
– 33% of US adults have no Internet access – KN provides laptop computer,
free monthly ISP
 Cell phone only households (28% of US) through ABS mail recruitment
 Spanish-language households
 Extensive profile data maintained on member demographics,
attitudes, opinions, behaviors, health conditions, media usage, etc.
•
Samples from the panel are assigned to studies using
e-mail invitations and a link to the online survey questionnaire
77
Population Coverage by Mode/Frame
KN-ABS Frame
probability based and covers
non-internet HHs
U.S. Adults
KnowledgePanel Frame
Landline Telephone
Frame
Unknowable clustering
of non-probability
opt-in Web panel
samples
Internet Survey
Frame
74%2
72%1
97%
1
8
2
National Center for Health Statistics.
Internet access from any location. October 2009 CPS.
100%
non-probability
online panels
Accuracy of probability-based KnowledgePanel
and RDD samples
“Both were significantly more accurate than any of the non-probability opt-in
sample Internet surveys.”
Average absolute errors for probability and non-probability sample surveys across 13 secondary
demographics and non-demographics, with post stratification.
Non-Probability Samples
Probability
Samples
Source: Yeager, Krosnick, et. al., Comparing the Accuracy of RDD Telephone Surveys and
Internet Surveys Conducted with Probability and Non-Probability Samples. August, 2009.
99
The American Association of Public Opinion Research
(AAPOR) Task Force On Online Panels
Avoid non-probability online panels when objective is to accurately estimate
population values
Claims of “representativeness” should be avoided when using these sample
sources
Non-probability online panels are generally less accurate when compared to
benchmark data
“AAPOR Report on Online Panels,” prepared by the AAPOR Online Task Force Report, March 2010.
Available at www.aapor.org.
10
10
Using an ABS Sample Frame to Recruit
a Probability-Based Online Panel
The Knowledge Networks Experience
11
50,000 members
representing
America
Probability-based recruitment, representative of U.S. adult
population
Includes:
 Households with no Internet access at time of recruitment
– 33% of US adults have no Internet access – KN provides laptop computer,
free monthly ISP
 Cell phone only households (28% of US) through ABS mail recruitment
 Spanish-language households
 Extensive profile data maintained on member demographics,
attitudes, opinions, behaviors, media usage, etc.
•
Samples from the panel are assigned to studies using
e-mail invitations and a link to the online survey questionnaire
12
12
Mail Recruit Address-based Sample (ABS)
U.S. Postal Service Computerized Delivery Sequence File
(CDSF)
 ~97% coverage of physical addresses
 Frequently updated including status of addresses, such as,
seasonal homes, vacant houses, etc.
 Can be matched to available telephone numbers
 Can be geo-coded
 Can attach demographic data (actual and modeled) from a variety
of sources for purposes of
Non-response analyses
Targeted demographic mailings
13
13
Mail Materials and Schedule
Day 7 Reminder PC
Day 28 NR Letter
Current Resident / Residente Actual
123 Your Street
The City, State 99999
Initial Mailing
14
Three Response Modes for Joining
Respond by:
1. Mail
2. Online
3. Telephone
Toll-free
number
Non-Responders:
Outbound Telephone Recruitment
15
15
ABS Recruitment Breaking new ground in 2011
Expanding KnowledgePanel with innovative recruitment samples
Objective: Recruit more Hispanics and young adults (18-24)
Predictive ancillary information is used to define targeted strata
16
Calibrating probability and non-probability
samples
When the finite size of the probability-based panel is unable
to deliver enough sample for a given study
Supplement with quota-controlled “opt-in” sample
Include ancillary information among weighting variables to
calibrate the opt-in cases to KN panel cases
“Early adopter” behavior consistently differentiates to two
sample sources
Goal: minimize bias introduced by opt-in cases
17
Example of a calibrated sample
Figure 1.
Blended using standard variables
18
Figure 2.
Blended using standard
variables plus a calibration
variable (early adopter Q)
Health tracking study example
California Tobacco Control Program
19
California Tracking Survey – Design
Specific media evaluation survey
General population, approx. 2,000 adults (ages 18-55)
Smokers and non-smokers
 Supplemented with California smokers from opt-in sample
Fielded immediately after a flight of media
 Every December and June/July
Version 1.0 RDD telephone data collection
July 2001 to June 2004 (5 waves)
Version 2.0 KnowledgePanel online data collection
December 2005 to ongoing (11 waves to date of these results)
Longitudinal + fresh cross-sectional samples (combined for x-sec)
California Tracking Survey – Measures
Aided ad awareness
 Executional and message awareness
 Talk with others
 New information
Attitudes and knowledge
 Second hand smoke (SHS), anti-tobacco industry, regulation
 Health effects
Self-reported smoking and cessation behavior
Use of tracking data
Cross-sectional analysis





General ad and campaign awareness
Ad message awareness and/or impart new information
Take action step
Tracking of attitudes over time
Examine the relationship between ad recall and ad placement
Longitudinal analysis
 Is ad awareness related to attitudinal changes?
 Is ad awareness related to behavioral changes?
Percent of Californians that agree that multi-unit housing or apartment
buildings should require that at least half of their rental units be smoke-free,
2006-2010 (KnowledgePanel data)
90
80
70
Percent
60
50
Ads placed that specifically address secondhand smoke in multi‐unit housing
40
30
20
10
Dec‐10
Jun‐10
Dec‐09
Jun‐09
Dec‐08
Jun‐08
Dec‐07
Jun‐07
Dec‐06
Jun‐06
0
Percent of Californians that have seen or heard any anti-tobacco
messages and media spending, 2000-2009
RDD
KnowledgePanel
Effectiveness
of DRTV
DRTV = Direct Response TV advertisement; TRP = Target Rating Points
Thank you!
cdisogra@knowledgenetworks.com
© 2011 Knowledge Networks, Inc.