Showing Uncertainty of Official Statistics
Keywords: uncertainty, visualisation
1.
INTRODUCTION
Statistics is the science of quantitative estimation of social, economic or natural
phenomena. Estimation entails that the resulting figures are not precise: they have
measure of uncertainty. In the sciences measurements without a confidence interval
indicating their uncertainty are usually not accepted. Strangely enough most Official
Statistics are communicated without a measure of uncertainty, even when it is available.
For example economic growth and inflation are often presented as being exact, though
they form the basis for important decisions. [1] argues that omitting uncertainty in the
presentation of official statistics leads their users to think that the data is error-free or lets
them conjecture wrong error sizes. Besides the fact that uncertainty may be very difficult
to estimate due to systematic or measurement errors, one reason for leaving out
uncertainty in the communication is the assumption that users do not appreciate error
margins or cannot interpret them. We conducted two user experiments to test these
assumptions: an interactive software prototype in which users could select and view open
data from a NSI and a carefully designed user survey with synthetic data in which the
effect of showing uncertainty on user statements was tested.
1.1. Context
Both user experiments were conducted in the StatMine project. This research project
tests several improvements on the Open Data dissemination database of a NSI by
implementing an enhancement in a software prototype and testing it on statistical
analysts, policy makers and journalists. One proposed improvement is displaying
uncertainty intervals with published statistical figures. Furthermore, a more general
enhancement was to publish and present interval estimates. Interval estimates expand the
range of statistical methods that can be used for producing official statistics. For example
in statistical disclosure control interval estimates allow more cells to be published than
point estimate methods.
We use a simple but general notion of uncertainty: each statistic y has at least an interval
estimate, meaning that has a value that lies with high probability (e.g. 95%) in the
interval [𝑦̂𝑏 , 𝑦̂𝑢 ]. In addition a statistic y may have a point estimate 𝑦̂ indicating it’s most
likely value. No further assumptions are made on the shape of the probability density
function or the source of uncertainty. It can be a traditional 95% confidence interval
using a normality assumption, it may be Bayesian credible interval or it might include
systematic and measurement error estimates.
2.
METHODS
2.1. Experiment 1: Interactive software prototype
The interactive prototype of the StatMine project was adapted to include and display
uncertainty intervals on several data sets. A cognitive psychologist conducted a user test
with 10 users in which a user answered several questions with the prototype. Each test
was preceded and concluded with a detailed interview.
1
2.2. Effect of displaying uncertainty on user interpretation
A survey (N=108) was conducted which tested for the effect of displaying uncertainty in
line and bar charts [2] on user interpretation. A line chart is mostly used for time series
and helps a user to detect if the series is decreasing, constant or increasing. A bar chart is
mostly used to compare absolute values of different subpopulations: Is the average wage
of men higher than the wage of women? Several types of line and bar charts were tested
on different datasets.
Each dataset was tested with a different chart type on a different group respondents. Each
respondent saw all datasets with all charts types, but each group saw a different
combination. For each line chart a respondent had
to answer several questions, including a question
if the line was increasing (5 certainty levels). For
each bar chart a respondent answered multiple
questions including if bar A was higher than bar B
(5 certainty levels). The last task was, depending
on the dataset and the display of error bars,
sometimes ambiguous.
The survey included questions on user preference,
which were confronted with their performance.
Figure 1 Different line chart types: line, line +
ribbon, line + error bar, ribbon, error bar
3.
RESULTS
3.1.
Interactive prototype
The tested users do not react confused, but find the uncertainty margins useful. For
example policy makers frame the interval [𝑦̂𝑏 , 𝑦̂𝑢 ] as the range between a worst and a
best case scenario, with 𝑦̂ being the most likely scenario. When uncertainty margins are
displayed many users ask what the exact meaning of [𝑦̂𝑏 , 𝑦̂𝑢 ] is, which indicates genuine
user interest in those figures. When offered the choice to omit uncertainty margins, users
prefer to keep them.
3.2.
Survey on effect
When uncertainty is displayed in line charts, users are more often correct in detecting a
statistically significant trend in the data. Users are overconfident in detecting a trend in a
line chart without uncertainty. The Ribbon +line scores best and is the preferred option
for displaying time series. However when no point estimates is available, error bar
should be chosen.
Table 1. Correctness of trend assessment in line charts
Type
Ribbon + line
Error bar
Answers
90
83
Trend correct
88.9%
85.5%
2
Error bar + line
Ribbon
Line
82
86
91
81.7%
80.2%
74.%
In bar charts many respondents prefer error bars, although they perform slightly worse
than other options (see also [3]): this may be an effect of education.
4.
CONCLUSIONS
Against common opinion, the statistical analysts, policy makers and journalists
appreciate error margins when they are displayed. Displaying uncertainty improves the
validity of user assessment of trends in line charts. This opens up the publication of
uncertainty measures and interval estimates.
REFERENCES
[1] Manski, C. F. (2014). Communicating uncertainty in official economic statistics.
Technical report, National Bureau of Economic Research.
[2] Playfair, W. (1786). The Commercial and Political Atlas: Representing, by Means
of Stained Copper-Plate Charts, the Progress of the Commerce, Revenues,
Expenditure and Debts of England during the Whole of the Eighteenth Century.
[3] Sanyal, J., Zhang, S., Bhattacharya, G., Amburn, P., and Moorhead, R. J. (2009).
A user study to compare four uncertainty visualization methods for 1D and 2D
datasets. IEEE transactions on visualization and computer graphics, 15(6):1209–
18.
3