Benchmarking
Training Workshop on the Compilation of Quarterly National Accounts
for Economic Cooperation Organization Member Countries
8-11 March 2015
Tehran, Islamic Republic of Iran
United Nations Statistics Division
Outline






What is benchmarking?
Purpose of benchmarking
Benchmarking methods
Benchmarking software for QNA
Conclusions
Questions
2
What is benchmarking?

Benchmarking is a statistical technique to
correct inconsistencies between estimates of
the same variable obtained from data
collected at different frequencies to produce
a consistent time series
•

Example: quarterly and annual value-added
estimates which are compiled using different data
sources such as quarterly and annual surveys
It is usually done retrospectively as annual
benchmark data are available sometime
after the quarterly data
3
What is benchmarking?
Reasons for differences between quarterly
and annual data sources

Difference in coverage and sample
•
•

Difference in definitions and variables
•

Output replaces value added for growth measures
Accounting methods
•

Annual survey has broader coverage and a more
representative sample
Differences in frame and sample size may exist
Different quarterly and annual accounting methods
Estimation method, non-response treatment,
respondent error, sampling error...
4
What is benchmarking?

Aspects of benchmarking
•
•
Quarterization of annual data to construct time
series of historical QNA estimates (back series)
and revise preliminary QNA estimates to align
them with new annual estimates when they
become available
Extrapolation to update the series from
movements in the indicator for the most current
period (forward series)
5
Purpose of benchmarking


Combines the relative strengths of low-frequency
data (say, annual data) and high-frequency data
(say, quarterly data) while preserving as much as
possible the short-term movements
Creates a coherent high-frequency data series by
correcting the difference between benchmark and
indicator values (indicator bias)
•
Ensures benchmark to indicator (BI) ratio becomes 1

Example: Adjust indicator I
q, y
so that
BI 
A
I
y
4
q 1
1
q, y
6
Purpose of benchmarking

Ensures, for forward series, that the sum of the
four quarters of the current year is as close as
possible to the unknown future annual data

Improves quality of national accounts data in
terms of comparability and coherence
7
Benchmarking methods

Numerical methods
•
•

Pro rata distribution method
Proportional Denton method
Statistical modelling methods
•
•
ARIMA-model-based methods
General least-squares regression models


Cholette Dagum
Chow-Lin
8
Benchmarking methods
Pro rata distribution method

For any benchmark year, distribute annual value in
direct proportion to the quarterly values
X q, y
X q, y
I q, y
Ay
Ay
I
q

 I q, y
 Ay  
  I q, y
 q



 Ay
  I q, y   I

  q, y

 q





is the level of the QNA estimate for quarter q of year y
is the level of the indicator for quarter q of year y
is the level of the annual benchmark value for year y
is the BI ratio
q, y
9
Benchmarking methods
Pro rata distribution method
Advantages
Disadvantages
Easy to compute and interpret
Smoothens quarterly estimates
only within a year
No special software needed
Quarterly estimates can be
derived each year
independently
Estimates are well aligned to
benchmark value and are fairly
reliable when BI ratios are
stable
Concentrates bias in one
quarter and cause abrupt
change (“step problem”)
Not recommended for longer
time series
10
Benchmarking methods
Example of pro rata distribution method
Yellow cell shows step problem caused by change in BI ratio
Source: IMF QNA manual (2001)
11
Benchmarking methods
Pro rata distribution method – step problem
Source: IMF QNA manual (2001)
12
Benchmarking methods
Pro rata distribution method
Benchmark-to-indicator ratio
Source: IMF QNA manual (2001)
13
Benchmarking methods
Proportional Denton method
 Goal: Find new estimates with minimal
deviation from original indicator series
2
 X t X t 1 
min   
t 1,...4 ,.., T 

( X1 ,.. X 4  ., X T )
I t 1 
t 2  It
T
T
under restriction (for flow series):
t
Xt
It
Ay

T
is time
X
t 2
t
 Ay , y  1,...,  
is the derived QNA estimate for quarter t
is the level of the indicator for quarter t
is the annual data for year y
is the last year for which an annual benchmark is available
is the last quarter for which quarterly source data are available
14
Benchmarking methods
Proportional Denton method

Avoids the step problem seen in the pro rata
distribution method by implicitly constructing
from the annual observed BI ratios a time
series of quarterly benchmarked QNA
estimates-to-indicator (quarterly BI) ratios that
•
•
•

Is as smooth as possible
Average to the annual BI ratios for each year for the
back series
Are kept constant and equal to the ratio for the last
quarter of the last benchmark year
Ensures quarterly growth rates are adjusted by
relatively similar amounts
15
Benchmarking methods
Example of proportional Denton method
No big
jump in
periodto-period
change
Source: IMF QNA manual (2001)
16
Benchmarking methods
Proportional Denton method
Solution to step problem
Source: IMF QNA manual (2001)
17
Benchmarking methods
Proportional Denton method
Benchmark-to-indicator ratio
Source: IMF QNA manual (2001)
18
Benchmarking methods
Proportional Denton method


Method requires that the indicator contains positive
values only
For series with zeros but no negative values
•

Replace zeroes with values infinitesimally close to zero
For series with both negative and positive values
and are derived as the difference between two nonnegative series (for example, changes in
inventories)
•
•
Apply method to opening and closing inventory levels, or
Turn the indicator into positive series by



Adding a large constant to all periods
Doing benchmarking
Removing the large constant from the results
19
Benchmarking methods
Proportional Denton method

Impact on back series
•
•
•
•

QNA quarter-to-quarter growth rates differ from those in the
indicator
May introduce new turning points, or
May change timing of turning points
These changes are desirable result of incorporating
information in the annual data
Impact on forward series
•
•
Quarter-on-quarter growth rates are identical to those in the
indicator
But, annual growth rate for the first year of the forward series
differs from that of the source data
20
Benchmarking methods
Proportional Denton method


Forward series estimates are of great interest to
users
Forward series estimates can be improved to reduce
size of future revisions by using enhanced version
of proportional Denton method which forecast
annual BI ratios by incorporating information on
past systematic movements in the annual BI ratios
•
Example


A study of movements in annual BI ratios show that the
indicator on average understates annual rate of growth by
2.0%
Adjust annual BI ratio for forward series by 2.0%
21
Benchmarking methods
Example of enhanced proportional Denton method
Source: IMF QNA manual (2001)
22
Benchmarking methods
Statistical modelling methods



Take into account any known information
about the stochastic properties of the series
being benchmarked
Were mostly actually proposed to improve
survey estimates where the survey design may
provide information about the stochastic
(sampling) properties of the series
In QNA, information about the stochastic
properties of the series is usually non-existent
and non-sampling errors may be more
important than sampling errors
23
Benchmarking methods
Statistical modelling methods


Unlike the Denton method, these methods may
over-adjust the series by interpreting as errors,
and thus removing, true irregular movements
that do not fit the regular patterns of the
statistical model
Models may mis-specify the error terms,
resulting in inaccurate estimates
24
Benchmarking software for QNA



BENCH
ECOTRIM
XLPBM
25
Benchmarking software for QNA
BENCH




Was developed by Statistics Canada
Runs under Unix/DOS operating systems
Allows users to choose among many
alternative options offered by the CholetteDagum regression-based model
Requires input data to be prepared in preformatted text files
26
Benchmarking software for QNA
ECOTRIM






Was developed by Eurostat
Runs in Windows
Provides interactive and batch support
Allows input data to be prepared in text files or
spreadsheets
Allows validation of results using graphical
tools
Has option for Denton method
27
Benchmarking software for QNA
XLPBM



Was recently developed by IMF Statistics
Department
Comprises a Microsoft Excel add-in function
with options for the enhanced proportional
Denton and Cholette-Dagum methods
Is particularly useful for QNA processing
systems based on spreadsheets.
28
Conclusions

Benchmarking is needed to derive QNA series
that are
•
•
•




Temporally consistent with the ANA benchmarks
Preserve as much as possible the movements in the
quarterly indicators
Provide accurate extrapolations for the current year
There are a number of methods to perform
benchmarking
The pro rata distribution method is not
recommended due to the step problem
The proportional Denton method is robust,
simple and well suited for large-scale
applications
A number of software are available to do
benchmarking
29
Questions


What method is your country using to do
benchmarking in the compilation of quarterly
GDP estimates? Why?
Is the benchmarking method described in the
national accounts methodological notes?
30
Thank you
31