Joint Statistical Meetings - Section on Health Policy Statistics (HPSS)
USE OF CONSUMER PANEL SURVEY DATA FOR PUBLIC HEALTH COMMUNICATION
PLANNING: AN EVALUATION OF SURVEY RESULTS
William E. Pollard
Office of Communication
Centers for Disease Control and Prevention
1600 Clifton Rd., N.E., Mailstop D-42
Atlanta, Georgia 30333
than a million individuals across the range of census
demographics for use in market research.
KEY WORDS: Survey Methods, Mail Panel
Surveys, Health Communication Data
As discussed in a research report by one such
firm (Market Facts, 1994), panels have several advantages for conducting surveys:
OVERVIEW
The Office of Communication at the Centers
for Disease Control and Prevention licenses syndicated
market research data for understanding audiences in
health communication planning. Market research data
bases are widely used in the commercial sector to
analyze audiences and develop messages to promote
products and services to potential customers. They
contain proprietary and public information on sociodemographic characteristics, consumer behavior,
lifestyle activities, and media habits of potential customers, and are available, with individual identifying
information removed, through licensing and contractual
agreements. These data can be used to identify population segments to target with a message, and to identify
audience segments that may differ in interests, lifestyle,
and media habits in order to design messages with
appropriate design, content, and media channels.
(1) response rates are high;
(2) attrition rates in longitudinal research are
low;
(3) customized samples can be easily constructed from data on hand;
(4) samples can be made demographically
representative on multiple variables from data
on hand;
(5) much respondent and household information is already on file which saves time and
space on surveys;
(6) they facilitate otherwise very difficult or
expensive research, such as surveys of children and brand loyalty studies.
A widely-used method for obtaining such data
in the field of market research is through sample
surveys where the samples are drawn from pools of
individuals and households that are pre-recruited for
use in various surveys, and these pools are known as
omnibus or multi-purpose panels (Burns and Bush,
2000, Ch. 5). Because this method differs from national probability sampling methods typically used in
health research, in this paper we examine the reliability
and validity of results from panel surveys for use in
health communication planning.
However, the self-selection involved in
agreeing to participate in a panel raises questions about
the representativeness of the results. Are participants in
panels different from the general population in their
survey responses? In this paper, we address this
question by comparing results from national mail panel
surveys with those obtained from national probability
sample surveys.
PREVIOUS RESEARCH
Multi-purpose Consumer Panels in Market Research
In the Market Facts (1994) report cited above,
the researchers examined the comparability of results
obtained from panel and non-panel samples by conducting parallel telephone surveys of a nationally representative random sample of members from their panel of
over 600,000 U.S. households, and a national non-panel
Panels are made up of groups of individuals
and households who have agreed to participate in
periodic surveys concerning products and lifestyle.
Commercial marketing firms develop and maintain
nationwide panels of several hundred thousand to more
2720
Joint Statistical Meetings - Section on Health Policy Statistics (HPSS)
on age, race, sex, income, and family size (Korn and
Graubard, 1999; Lohr, 2000). For a full description of
the survey see Maibach, et al. (1996).
sample obtained through conventional random digit dial
procedures. They found that the two samples were very
similar on consumer behavior, lifestyle, and health
status items. In addition, they found no differences in
positive outlook, altruism, and amount of free time.
A number of the items on the HealthStyles
survey are the same as, or similar to, items on the
Behavioral Risk Factor Surveillance System (BRFSS)
survey. This is a national probability sample telephone
interview survey. It is a collaborative project of the
CDC and U.S. states. The sample size for core items
administered in all interviews is 150,000 to 180,000
respondents annually. It is an accepted and widelyused public health surveillance and planning survey.
Annual statistics for selected items from the survey are
summarized on the CDC web site (www.cdc.gov) and
are used here in the comparisons with HealthStyles
items. Most of the national prevalence percentages
reported in the tables on the web site are medians of
state values.
In an extensive, widely-discussed study of
civic involvement in the U.S., Putnam (2000) used
results from the General Social Survey and the DDB
Needham Lifestyles survey. The General Social Survey
is a personal interview survey conducted by the National Opinion Research Center (NORC) using a
national probability sample. The Lifestyles survey is a
national mail panel survey conducted annually with
independent samples by Market Facts for DDB
Needham, a large advertising agency. The Lifestyles
samples are drawn from the panel through stratified
random sampling. The two surveys contained a number
of similar opinion and lifestyle items. Putnam compared results of the surveys over a period of 25 years
and found very close agreement in (1) the level of
response, (2) trends over time, and (3) the pattern of
demographic correlates. He writes that “for purposes of
explaining this wide range of attitudes and behavior, the
two surveys are virtually indistinguishable” (p. 422).
There were nine items from the BRFSS on the
web site that were comparable to items on the
HealthStyles survey from 1995 to 2001. These include:
Health Conditions: arthritis, asthma, diabetes,
high blood pressure, and overweight or obese,
where this weight category is defined in terms
of Body Mass Index (BMI) of 25.00 or
greater;
CURRENT STUDY
Objectives
Attitudes: health status self-rating;
The objective of this study is to examine the
reliability and validity of a health communication data
base in which the data are obtained from a national mail
panel survey. Reliability is assessed in terms of stability of item results over time: results should be stable
from year to year in the absence of known trends or
events that would be expected to cause changes.
Validity is assessed in terms of agreement with other
measures of the items that are themselves considered to
be valid.
Behaviors: flu shot, pneumonia vaccination,
smoke cigarettes.
Note that not every item is measured every year on
either survey.
The HealthStyles survey is not a surveillance
survey, but is designed to provide data for understanding audiences for communication and health promotion
planning. Many of the items do not have counterparts
on other surveys that can be used to examine validity.
However to the extent that results for the above items
can be shown to be valid in comparison with results
from an accepted national probability sample survey, it
provides indirect support for the validity of the results
of other items on the survey.
Methods
In this paper we examine data from the
HealthStyles mail panel survey conducted for Porter
Novelli, a social marketing and public relations firm.
This survey has been conducted annually with independent samples since 1995 and contains items on
health attitudes, behaviors, conditions, and knowledge.
From 1995 to 2001, the survey has been administered
by Market Facts to respondents to the DDB Needham
Lifestyles panel survey, with oversampling for low
income and minority groups. The sample size is around
2,500 to 3,000 respondents per year. Results are
poststratified and weighted to US census benchmarks
Results
The basic findings are shown in the bar charts
in Figure 1. The 95% confidence interval for the
HealthStyles values is approximately plus or minus two
percentage points around the reported value. In terms
2721
Joint Statistical Meetings - Section on Health Policy Statistics (HPSS)
of reliability, the HealthStyles results are generally
stable from year to year. The systematic changes over
time that do appear are consistent with known trends
mentioned below. In terms of validity, the levels are
generally within a few percentage points of those from
the BRFSS. Both surveys show trends for increasing
prevalence of diabetes and overweight or obese
(Mokdad, et al., 2000; Flegal, et al., 2002). In addition,
further analyses showed similar demographic patterns
in both data sets. For diabetes, both data sets showed
increasing prevalence with increasing age of respondents, and higher prevalence in black respondents. For
overweight or obese, both showed increasing prevalence with increasing age or respondents until age 65+,
where prevalence declines, and higher prevalence in
black respondents. Overall, there was agreement on
levels, trends over time, and demographic breakdowns.
REFERENCES
Burns, A.G. and Bush, R.F. (2000). Marketing Research, (3rd. ed.). Upper Saddle River, NJ:
Prentice Hall.
Flegal, K.M., Carroll, M.D., Ogden, C.L., and Johnson
C. (2002). Prevalence and trends in obesity
among US adults, 1999-2000. Journal of the
American Medical Association, 288(14), 1723
- 1727.
Korn, E.L. and Graubard, B.I. (1999). Analysis of
Health Surveys. NY: John Wiley & Sons, Inc.
Lohr, S.L. (1999). Sampling: Design and Analysis.
Pacific Grove, CA: Duxbury Press.
In quantitatively summarizing the validity, for
the nine items over the seven years, there were 34
same-year data pairs where the two surveys could be
directly compared. For these 34 pairs of percentages,
the average difference was 2.35 percentage points and
the average of the absolute value of the differences was
2.88 percentage points.
A scatterplot of the
HealthStyles and BRFSS percentages is shown in
Figure 2. For these 34 paired percentages, the correlation between the HealthStyles and BRFSS values was
r = .99, reflecting the close agreement.
Maibach, E., Maxfield, A., Ladin, K., and Slater, M.
(1996). Translating health psychology into
effective health communication: The American HealthStyles Audience Segmentation
Project. Journal of Health Psychology, 1,
261-277.
CONCLUSION
Mokdad, A.H., Ford, E.S., Bowman, B.A., Nelson, D.
E., Engelgau, M.M., Vinicor, F., and Marks,
J.S. (2000). Diabetes trends in the U.S.: 19901998. Diabetes Care, 23(9), 1287-1283.
Market Facts (1994). Mail Panels vs. General Samples: How Similar and How Different? Research on Research, Report No. 59. Arlington
Heights, IL, Market Facts.
The findings provide support for the reliability
and validity of the HealthStyles mail panel data for the
items examined, and provide indirect support for the
reliability and validity of other communication-relevant
items on the survey. Data from the HealthStyles mail
panel survey complement existing surveillance data and
provide useful information for understanding audiences
in health communication planning.
Putnam, R.D. (2000). Bowling Alone: The Collapse
and Revival of American Community. Appendix I: Measuring Social Change. New York:
Simon and Schuster.
__________________________________________
These findings, along with those from the
previous research, also have some more general practical and theoretical implications for survey research. As
mentioned earlier, panels have a number of significant
advantages for facilitating the conduct of surveys and
reducing costs. It would be helpful to have some
general guidelines concerning the conditions under
which panel and non-panel surveys will yield comparable results. Additional empirical and theoretical work
on this topic would be very useful to survey researchers.
The author would like to express his thanks to Dr.
Edward Maibach and Dr. Deanne Weber from Porter
Novelli for their encouragement of this project, and to
Dr. Weber for help in understanding the HealthStyles
methodology and the features of the data files.
2722
Joint Statistical Meetings - Section on Health Policy Statistics (HPSS)
Figure 1. Bar charts showing HealthStyles and BRFSS results from 1995 to 2001.
2723
Joint Statistical Meetings - Section on Health Policy Statistics (HPSS)
Figure 2. Scatterplot of HealthStyles and BRFSS values (r = .99).
2724