Total Survey Error: Past,
Present, and Future
Robert M. Groves
U.S. Census Bureau
Lars Lyberg
Statistics Sweden and Stockholm University
1
Outline
1.
2.
3.
4.
Evolution of “total survey error” (TSE)
Weaknesses of the TSE framework
Strengths of the TSE framework
Future of surveys and research agenda
ideas
2
Deming (1944)
“On Errors in Surveys”
• American Sociological Review!
• First listing of sources of problems,
beyond sampling, facing surveys
• The 13 factors
3
Deming’s 13 factors
-The 13 factors that affect the usefulness of a
survey
-To point out the need for directing effort toward all
of them in the planning process with a view to
usefulness and funds available
-To point out the futility of concentrating on only
one or two of them
-To point out the need for theories of bias and
variability that correlate accumulated experience
4
5
Comments on Deming (1944)
• Does include nonresponse, sampling,
interviewer effects, mode effects, various
other measurement errors, and processing
errors
• Omits coverage errors
• Includes nonstatistical notions (auspices)
• Includes estimation step errors (wrong
weighting)
• “Total survey error” not used as a term
6
Sampling Text Treatment of Total
Survey Error
• Kish, Survey Sampling, 1965
– Graphic on biases
– 65 of 643 pages on various errors, with
specified relationship among errors
7
Frame biases
Sampling Biases
“Consistent” Sampling Bias
Constant Statistical Bias
Noncoverage
Nonobservation
Nonresponse
Field: data collection
Nonsampling
Observation
Office: processing
Biases
8
Sampling Text Treatment of Total
Survey Error
• Särndal, Swensson, Wretman, Model
Assisted Survey Sampling, 1992
– Part IV, 124 pp. of 694, coverage,
nonresponse, measurement error; omits
processing error
• Lohr, Sampling Design and Analysis, 1999
– 34 of 500 pages, nonresponse, randomized
response
9
Other textbooks
• Cochran (1953). Sampling Techniques.
40 pages in concluding chapter on “sources of
error in surveys”
• Deming (1950). Some Theory of Sampling.
Starts with the 1944 factors but then continues
with pure sampling
• Hansen, Hurwitz and Madow (1953). Sample
Survey Methods and Theory, Vol 1. Nine pages
on survey errors.
• Zarkovich 1966. Quality of Statistical Data.
10
Total Survey Error (1979)
Anderson, Kasper, Frankel, and Associates
• Empirical studies on nonresponse,
measurement, and processing errors for
health survey data
• Initial total survey error framework in more
elaborated nested structure
11
Sampling
Variable
Error
Field
Nonsampling
Processing
Frame
Total Error
Sampling
Consistent
Noncoverage
Bias
Nonobservation
Nonsampling
Nonresponse
Field
Observation
12
Processing
Survey Errors and Survey Costs
(1989), Groves
• Attempts conceptual linkages between total
survey error framework and
– psychometric true score theories
– econometric measurement error and selection bias
notions
• Ignores processing error
• Highest conceptual break on variance vs. bias
• Second conceptual break on errors of
nonobservation vs. errors of observation
13
Mean Square Error
construct validity
theoretical validity
empirical validity
reliability
Variance
Errors of
Nonobservation
Coverage
Nonresponse
Observational
Errors
Sampling
Interviewer
Respondent
Instrument
Mode
criterion validity
- predictive validity
- concurrent validity
Bias
Observational
Errors
Errors of
Nonobservation
Coverage
Nonresponse
Sampling
Interviewer
Respondent
Instrument14 Mode
Nonsampling Error in Surveys
(1992), Lessler and Kalsbeek
• Evokes “total survey design” more than
total survey error
• Omits processing error
15
Components of Error
Frame errors
Topics
Missing elements
Nonpopulation elements
Unrecognized multiplicities
Improper use of clustered frames
Sampling errors
Nonresponse errors
Deterministic vs. stochastic view of nonresponse
Unit nonresponse
Item nonresponse
Measurement errors
Error models of numeric and categorical data
Studies with and without special data collections
16
Introduction to Survey Quality,
(2003), Biemer and Lyberg
• Major division of sampling and
nonsampling error
• Adds “specification error” (a la “construct
validity”) or relevance error
• Formally discusses process quality
• Discusses “fitness for use” as quality
definition
17
Sources of Error
Types of Error
Specification error
Concepts
Objectives
Data element
Frame error
Omissions
Erroneous inclusions
Duplications
Nonresponse error
Whole unit
Within unit
Item
Incomplete Information
Measurement error
Information system
Setting
Mode of data collection
Respondent
Interview
Instrument
Processing error
Editing
Data entry
Coding
Weighting
Tabulation
18
Survey Methodology, (2004)
Groves, Fowler, Couper, Lepkowski, Singer,
Tourangeau
• Notes twin inferential processes in surveys
– from a datum reported to the given construct
of a sampled unit
– from estimate based on respondents to the
target population parameter
• Links inferential steps to error sources
19
Representation
Measurement
Inferential Population
Construct
Validity
Target Population
Measurement
Sampling Frame
Measurement
Error
Response
Sample
Processing
Error
Coverage
Error
Sampling
Error
Nonresponse
Error
Edited Data
Respondents
Survey Statistic
20
Key Statistical Developments in
Total Survey Error 1
•
•
•
•
•
Errors of observers can be correlated (1902),
Karl Pearson
Interpenetrating samples (1946),
Mahalanobis
Criteria for true values (1951), Hansen,
Hurwitz, Marks and Mauldin
Essential survey conditions, correlated
response variance (1959), H-H-Bershad
BC survey model “mixed-error model”(1961),
H-H-B
21
Key Statistical Developments in
Total Survey Error 2
• Interviewer effects using ANOVA (Kish 1962)
• Simple response variance via reinterviews (1964), H-HPritzker
• Relaxed assumptions of zero covariance of true values
and response deviations (1964, 1974), Fellegi
• Errors of Measurement (1968), Cochran
• Estimating model components via basic study schemes
using replication, interpenetreation and combinations of
the two (1969), Bailar and Dalenius
• Estimating nonsampling variance using mixed linerar
models (1978), Hartley and Rao
• “Error Profile” of Current Population Survey (1978),
Brooks and Bailar
• Multi-method multi-trait models on survey measures
22
(1984), Wothke…Browne
Key Statistical Developments in
Total Survey Error 3
• Measurement of imputation error variance through
multiple imputation (1987), Rubin
• Total error model for PES (1991), Mulry and Spencer
• Measurement Errors in Surveys (1991)
Section (Biemer and Stokes..Colm..Fuller and others
– attempts to juxtapose psychometric notions with
survey statistical notions of measurement error
• Latent class model applications to survey errors (late
1990’s), Biemer, Tucker and others
• Measurement error effects on analysis (1997) Biemer
and Trewin
23
Key Statistical Developments in
Total Survey Error 4
•
Certain components understudied
–
–
–
–
•
•
coverage error variance
nonresponse error variance
all processing errors
biases in general
The movement from measurement of MSE to
CBM, SOP thereby getting a situation where
Var becomes an approximation of MSE (errorfree processes)
Simultaneous treatment of more than one error
source (ITSEW)
24
Summary of the Evolution of
“Total Survey Error”
• Roots in cautioning against sole attention to
sampling error
• Framework contains statistical and nonstatistical
notions
• Most statistical attention on variance
components, most on measurement error
variance
• Late 1970’s attention to “total survey design”
(methods and theory for resource allocation to
the control of various sources of error
• 1980’s-1990’s attempt to import psychometric
notions, CASM
25
• Key omissions in research
Weaknesses of the Common
Usage of “Total Survey Error”
–
–
–
–
–
–
Notably a user perspective is missing
Key quality dimensions are missing in the TSE
paradigm
User often cannot or prefers not to question
accuracy
The complexity does not invite outside scrutiny of
accuracy
Users not really informed about real levels of error
or uncertainty
We don’t really know how users perceive
information on errors
26
Other Weaknesses of the Total
Survey Error Paradigm 1
1. Lack of routine measurements
No agency does this
Error/quality profiles are useful but rare
2. Ineffective influence on professional standards
Little expansion beyond sampling error in practice
Press releases on Federal statistics rarely contain
even sampling errors
Survey error research compartmentalized rather
than integrated
Methodologists tend to specialize
Root causes of error often still missing
How about OMB’s requirement of NR bias studies if
NR expects to exceed 20%?
27
Other Weaknesses of the Total
Survey Error Paradigm 2
3.Large burden on design of some estimators
Interpenetration, reinterviews for variance
estimation complicated and costly
Intractable expressions for some
components
4. Some assumptions unrealistic
28
Strengths of the Total Survey Error
Framework
1.
Taxonomic decomposition of errors
• nomenclature for different components
2. Separation of phenomena affecting statistics in
different ways
• variance vs. bias; observation vs. nonobservation;
respondent/interviewer/measurement task;
processing
3. Conceptual foundation of the field of survey
methodology
• subfields defined by errors
4. Tool to identifying gaps in the research literature
• e.g., where are the error evaluation papers on
processing?
29
Needed Steps in a Research
Agenda for Total Survey Error 1
1. Integrating causal models of survey errors
•
cognitive psychological mechanisms (anchoring,
recall decay)
2. Research on interplay of two or more error
sources jointly
•
e.g., nonresponse and measurement error
3. Research on the interplay of biases and
variances
•
e.g., does simple response variance increase
accompany some response bias reductions (selfadministration effects)?
30
Needed Steps in a Research
Agenda for Total Survey Error 2
Guidance on tradeoffs between quality measurement and
quality maximization and between measures and developing
error-free processes
- how much should we spend on quality enhancement vs.
measurement of quality (Spencer, 1985)?
5. Integrating other notions of quality into the total survey error
paradigm
- if “fitness for use” predominates as a conceptual base, how
can we launch research that incorporates error variation
associated with different uses? Australian Bureau of Statistics
4.
31
Needed Steps in a Research
Agenda for Total Survey Error 3
6. Exploiting a multiple-mode, multiple
frame, multiple phase survey world
7. Need for methodological studies to assist
the user
8. Costs and risks
9. Develop theories for optimal design of
specific operations, design principles
10. More standards?
32