Total Survey Error: Past,
Present, and Future
Robert M. Groves
University of Michigan and
Joint Program in Survey Methodology
1
Outline
1.
2.
3.
4.
Evolution of “total survey error” (TSE)
Weaknesses of the TSE framework
Strengths of the TSE framework
Future of surveys and research agenda
ideas
2
Deming (1944)
“On Errors in Surveys”
• American Sociological Review!
• First listing of sources of problems,
beyond sampling, facing surveys
3
4
Comments on Deming (1944)
• Does include nonresponse, sampling,
interviewer effects, mode effects, various
other measurement errors, and processing
errors
• Omits coverage errors
• Includes nonstatistical notions (auspices)
• Includes estimation step errors (wrong
weighting)
• “total survey error” not used as a term
5
Sampling Text Treatments of Total
Survey Error
• Deming, Some Theory of Sampling, 1950
– repeats set of comments in 1944 article
• Hansen, Hurwitz, Madow, Sample Survey
Methods and Theory, 1953
– 9 of 638 pages on response and other
nonsampling errors
– “errors due to faulty planning,” “coverage
errors,” “classification errors,” “publication
errors”
6
Sampling Text Treatment of Total
Survey Error
• Kish, Survey Sampling, 1965
– 65 of 643 pages on various errors, with
specified relationship among errors
– Graphic on biases
7
Frame biases
Sampling Biases
“Consistent” Sampling Bias
Constant Statistical Bias
Noncoverage
Nonobservation
Nonresponse
Field: data collection
Nonsampling
Observation
Office: processing
Biases
8
Sampling Text Treatment of Total
Survey Error
• Särndal, Swensson, Wretman, Model
Assisted Survey Sampling, 1992
– Part IV, 124 pp. of 694, coverage,
nonresponse, measurement error; omits
processing error
• Lohr, Sampling Design and Analysis, 1999
– 34 of 500 pages, nonresponse, randomized
response
9
Health Survey Methods
Conferences
• Mid 1970’s, “data error archive” idea of
Horvitz
• 1977 key paper, “Total Survey Design:
Effect of Nonresponse Bias and
Procedures for Controlling Measurement
Errors,” Kalsbeek and Lessler
10
Total Survey Error (1979)
Anderson, Kasper, Frankel, and Associates
• Empirical studies on nonresponse,
measurement, and processing errors for
health survey data
• Initial total survey error framework in more
elaborated nested structure
11
Sampling
Variable
Error
Field
Nonsampling
Processing
Frame
Total Error
Sampling
Consistent
Noncoverage
Bias
Nonobservation
Nonsampling
Nonresponse
Field
Observation
12
Processing
Survey Errors and Survey Costs
(1989), Groves
• Attempts conceptual linkages between total
survey error framework and
– psychometric true score theories
– econometric measurement error and selection bias
notions
• Ignores processing error
• Highest conceptual break on variance vs. bias
• Second conceptual break on errors of
nonobservation vs. errors of observation
13
Mean Square Error
construct validity
theoretical validity
empirical validity
reliability
Variance
Errors of
Nonobservation
Coverage
Nonresponse
Observational
Errors
Sampling
Interviewer
Respondent
Instrument
Mode
criterion validity
- predictive validity
- concurrent validity
Bias
Observational
Errors
Errors of
Nonobservation
Coverage
Nonresponse
Sampling
Interviewer
Respondent
Instrument14 Mode
Nonsampling Error in Surveys
(1992), Lessler and Kalsbeek
• Evokes “total survey design” more than
total survey error
• Omits processing error
15
Components of Error
Frame errors
Topics
Missing elements
Nonpopulation elements
Unrecognized multiplicities
Improper use of clustered frames
Sampling errors
Nonresponse errors
Deterministic vs. stochastic view of nonresponse
Unit nonresponse
Item nonresponse
Measurement errors
Error models of numeric and categorical data
Studies with and without special data collections
16
Introduction to Survey Quality,
(2003), Biemer and Lyberg
• Major division of sampling and
nonsampling error
• Adds “specification error” (a la “construct
validity”)
• Formally discusses process quality
• Discusses “fitness for use” as quality
definition
17
Sources of Error
Types of Error
Specification error
Concepts
Objectives
Data element
Frame error
Omissions
Erroneous inclusions
Duplications
Nonresponse error
Whole unit
Within unit
Item
Incomplete Information
Measurement error
Information system
Setting
Mode of data collection
Respondent
Interview
Instrument
Processing error
Editing
Data entry
Coding
Weighting
Tabulation
18
Survey Methodology, (2004)
Groves, Fowler, Couper, Lepkowski, Singer,
Tourangeau
• Notes twin inferential processes in surveys
– from a datum reported to the given construct
of a sampled unit
– from estimate based on respondents to the
target population parameter
• Links inferential steps to error sources
19
Representation
Measurement
Inferential Population
Construct
Validity
Target Population
Measurement
Sampling Frame
Measurement
Error
Response
Sample
Processing
Error
Coverage
Error
Sampling
Error
Nonresponse
Error
Edited Data
Respondents
Survey Statistic
20
Key Statistical Developments in
Total Survey Error 1
•
Criteria for true values (1951)
1. uniquely defined
2. defined in a manner that purposes of survey are
met
3. when possible, defined in terms of operations that
can be carried through
•
Essential survey conditions, correlated
response variance (1959)
–
–
–
conditioning factors for variance estimate
factors that affect value of variance
often ignore by users of  r [1  (m  1) ]
2
m
21
Key Statistical Developments in
Total Survey Error 2
• Correlated response variance (1959)
–
–
–
–
assume no covariance between true values and response deviations
assume no correlation of response deviations across interviewers
correlated response variance independent of workload
use of interpenetration
• Simple response variance via reinterviews (1964)
– assumes no covariance between trials
• Relaxed assumptions of zero covariance of true values and
response deviations (1964)
– combine use of interpenetration and reinterview
• “Error Profile” of Current Population Survey (1978)
• Multi-method multi-trait models on survey measures (1984)
– within-construct, among item covariance as tool for “simple response
variance
– imported from psychometrics
22
Key Statistical Developments in
Total Survey Error 3
• Measurement of imputation error variance
through multiple imputation (1987)
• Total error model for PES (1991)
• Measurement Errors in Surveys (1992)
– attempt to juxtapose psychometric notions
with survey statistical notions of measurement
error
• Latent class model applications to survey
errors (late 1990’s)
23
Key Statistical Developments in
Total Survey Error 4
• Understudied components
– coverage error variance
– nonresponse error variance
– all processing errors
– biases
24
Bias
Variance
25
Summary of the Evolution of
“Total Survey Error”
• Roots in cautioning against sole attention to
sampling error
• Framework contains statistical and nonstatistical
notions
• Most statistical attention on variance
components, most on measurement error
variance
• Late 1970’s attention to “total survey design”
• 1980’s-1990’s attempt to import psychometric
notions
• Key omissions in research
26
Weaknesses of the Common
Usage of “Total Survey Error” 1
1. Exclusions of key quality concepts
– notably a user perspective is missing
27
Key National Indicator Initiative
Quality Workshop
28
Credibility
29
Relevance
30
Estimator Quality
31
Data Quality
32
Weaknesses of the Total Survey
Error Paradigm 2
2. Lack of routine measurements
- error/quality profiles are useful but rare
3. Ineffective influence on professional standards
- little expansion beyond sampling error in practice
- press releases on Federal statistics rarely contain even sampling
errors
4. Large burden on design of some estimators
- interpenetration, reinterviews for variance estimation not routine
5. Patently wrong assumptions
- correlation of true values with errors (drug use reports)
- intraclass correlation independent of workload (interviewer
experience effects)
33
Strengths of the Total Survey Error
Framework
1.
Taxonomic decomposition of errors
•
2.
nomenclature for different components
Separation of phenomena affecting statistics in
different ways
•
3.
variance vs. bias; observation vs. nonobservation;
respondent/interviewer/measurement task; processing
Conceptual foundation of the field of survey
methodology
•
4.
subfields defined by errors
Tool to identifying lacunae in the research literature
•
e.g., where are the error evaluation papers on processing?
34
A Vision of the Future
Why “total survey errorists” have permanent
employment opportunities:
1. new areas of study using surveys
2. populations being studied change their
behavior
3. external world offers new measurement
tools
35
Needed Steps in a Research
Agenda for Total Survey Error 1
1.
Integrating causal models of survey errors
•
2.
cognitive psychological mechanisms (anchoring, recall decay)
Research on interplay of two or more error sources
jointly
•
3.
e.g., nonresponse and measurement error
Research on the interplay of biases and variances
•
4.
e.g., does simple response variance increase accompany
some response bias reductions (self-administration effects)
Development of diagnostic tools to study model-error
•
progress is slowed with just-identified models without
sensitivity analysis
36
Needed Steps in a Research
Agenda for Total Survey Error 2
5. Guidance on tradeoffs between quality measurement
and quality maximization
- how much should we spend on quality enhancement vs.
measurement of quality (Spencer, 1985)?
6. Integrating other notions of quality into the total survey
error paradigm
- if “fitness for use” predominates as a conceptual base, how can we
launch research that incorporates error variation associated with
different uses?
7. Exploiting a multiple-mode, multiple frame, multiple
phase survey world
- how can we build models that exploit non-randomized design
variations to give insight into various survey errors?
37
A Summary Pitch for the Union of
Design and Estimation
“To err is human, to forgive divine – but
to include errors in your design is statistical.
Kish, 1977
38