Factors influencing the reliability of
costings of policy proposals
The PBO’s approach to reliability ratings
Technical note no. 01/2015
Date issued: 30 June 2015
PBO technical notes
PBO technical notes are published to help explain the underlying data, concepts and
methodologies that the PBO utilises in preparing costings of policy proposals and analyses of
the budget and fiscal policy settings. The focus of PBO technical notes is different from that
of PBO research reports which are aimed at informing public understanding of budget and
fiscal policy issues more broadly.
© Commonwealth of Australia 2015
ISBN 978-0-9925131-8-4 (Online)
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs
3.0 Australia License.
The details of this licence are available on the Creative Commons website:
http://creativecommons.org/licenses/by-nc-nd/3.0/au/
Use of the Coat of Arms
The terms under which the Coat of Arms can be used are detailed on the following
website: www.itsanhonour.gov.au/coat-arms
Produced by: Parliamentary Budget Office
Designed by: Studio Tweed
First Assistant Parliamentary Budget Officer
Budget Analysis Division
Parliamentary Budget Office
Parliament House
PO Box 6010
CANBERRA ACT 2600
Phone: (02) 6277 9500
Email: pbo@pbo.gov.au
Contents
Overview ________________________________________________________________ 1
1
Why include reliability ratings? __________________________________________ 1
2
Factors affecting the reliability of costings __________________________________ 2
3
2.1
Data ___________________________________________________________ 2
2.2
Assumptions _____________________________________________________ 2
2.3
Volatility of the costing base ________________________________________ 3
How these factors are taken into account __________________________________ 4
iii
Overview
Costings are estimates of the financial impact of policy proposals, generally over some future
time period. Despite being the best possible estimates of the impact of a policy, all costing
estimates are subject to uncertainty, the level of which will vary from costing to costing
depending upon factors such as data quality, assumptions and the volatility of the costing
base. Because the level of uncertainty varies, the Parliamentary Budget Office (PBO)
includes reliability ratings in its costings as an indication of the level of confidence a user of
the costing can have that the actual outcome of a costing would correspond to the costing
estimate. The reliability rating is not a reflection on the policy proposal being analysed or on
the quality of the costing analysis.
1
Why include reliability ratings?
The PBO undertakes costings of policy proposals for parliamentarians. The PBO’s estimates
are prepared subject to the same rules and conventions as government budget estimates and
are the best possible estimates of the financial impact of a policy, given the information, time
and resources available.
Notwithstanding this, there are a number of elements that introduce uncertainty into the
costing process which mean that the point estimate values in some costings will more
accurately represent the actual outcomes (were the policy implemented) than is the case for
other policy costings.
The PBO includes reliability ratings in its costings of policy proposals to show that costing
estimates are uncertain and to indicate the level of confidence that a user of the costing can
have that the actual outcome of the policy costed (if implemented) would correspond to the
costing estimate.
When assessing the level of uncertainty that attaches to a costing estimate, the PBO
considers issues such as the quality of the data underpinning the costing, the number and
veracity of any assumptions made in the costing and the inherent volatility in the costing base
being estimated. These factors are discussed in Section 2.
The reliability rating is not a reflection on the policy proposal, nor does it reflect on the
quality of the costing analysis undertaken. Uncertainty is something that affects all costings,
regardless of who produces them. For instance, the United Kingdom’s Office of Budget
Responsibility recently adopted a system of uncertainty ratings for each certified policy
costing included in the United Kingdom Budget based on the approach of the Australian PBO1
outlined in this note.
1
See: “Economic and fiscal outlook”, Office for Budget Responsibility, United Kingdom, March 2015. Page
201
Factors influencing the reliability of costings of policy proposals
1
The issue of uncertainty in projections and costings has also been recognised by the United
States Congressional Budget Office which also seeks to highlight the uncertainty in its
estimates2.
2
Factors affecting the reliability of costings
The three most important factors affecting the reliability of costings are the quality of the
data available to undertake the costing, the number and soundness of any assumptions made
in the costing analysis and the volatility of the costing base itself.
2.1
Data
Data is the factual base from which the costing analysis starts. Data is used as the basis for
describing the costing base and/or eligible population for a costing analysis. The data used in
policy costings can come from a range of sources which can differ significantly in quality,
where quality is measured in terms of the certainty with which the data represents the target
population for a costing analysis.
Data from sources such as unit record administrative data can be regarded as being of very
high reliability as it represents the actual outcome of programmes and in many cases can
provide a high level of detail regarding the target population for a costing analysis.
The Australian Bureau of Statistics (ABS) produces a wide range of high quality data that is the
product of rigorous statistical techniques. Many ABS surveys come with quantitative
measures of the quality of the data including measures of the error margin of estimates and
indicators against the more uncertain data values.
Less reliable data will be that which is out of date, does not directly relate to the costing base
being estimated, comes from small sample surveys, or which originates from less rigorous
statistical sources or those with an uncertain reputation.
In many cases, high quality data may not be available, with the result that a costing will have
to be based on lower quality data sources, introducing a greater level of uncertainty into the
costing estimates.
2.2
Assumptions
Assumptions are utilised in costings to fill in gaps where data is not available and to take
account of matters pertinent to the costing, such as the behavioural responses of those
affected by a proposal.
Assumptions can be used to fill data gaps where the data available for a costing does not
provide the level of detail needed to cost the policy specified.
2
2
See: “Communicating the Uncertainty of CBO's Estimates”, post by Doug Elmendorf, December 15, 2014 at
http://www.cbo.gov/publication/49860.
Factors influencing the reliability of costings of policy proposals
In such cases assumptions may be used to fill in the gaps in the data in order to estimate
values for the target population for the costing. Ideally, this would be done by using other
data sources (such as distributional data for a similar population) to estimate the values for
the target population (proxy data). In this case, the model would be assuming that the proxy
data provides a good means of estimating the values for the target population from the data
source being used. The reliability of such assumptions will depend upon both the quality of
the proxy data source and upon how closely the distribution of the proxy data fits that of the
target population for the costing. The least reliable approach would be to use an assumption
based on judgement relating to what would be considered a ‘reasonable’ value for the
missing data.
Assumptions may also be used to take account of the impact of the behavioural responses of
the target population to the proposal being costed. The quality of behavioural response
assumptions may range from those based on relevant and well-researched studies, to those
based on the judgement of the analyst.
In the context of behavioural responses, an assumption that frequently implicitly forms the
basis of costing analyses is that of ‘no behavioural change’. This assumption is that the policy
being costed has no impact on the base volume (or value) of transactions being undertaken.
For policies that impact on the ‘price’ of transactions or which are intended to change
behaviour, this is an extreme behavioural assumption that implies that the transactions
concerned are completely price inelastic. This assumption, if not soundly based, can detract
from the reliability of a costing as much as any other costing assumption.
The impact of an assumption on a costing can be tested by sensitivity analysis that varies the
assumption over the plausible range of values and notes the range of potential outcomes.
The wider the range of potential costing outcomes from this analysis, the less certainty there
will be in the costing results.
The number of assumptions needed to complete a costing will also have a bearing on the
costing’s reliability, with costings having a higher level of certainty the fewer assumptions
that have to be made.
2.3
Volatility of the costing base
Volatility of the costing base affects the certainty with which the costing base can be forecast
over the projection period for a costing. Costing bases that show predictable growth over the
projection period with little variance will be more amenable to reliable costings.
On the other hand, some costing bases show highly erratic and uncertain growth, varying
significantly from year to year. These year-to-year variations may make forecasting these
bases extremely uncertain and introduce a large degree of uncertainty into costings.
For instance, costings of gross income tax withholding (ITW) revenue (ie personal income tax
withheld from wages and salaries) for the whole population are likely to be more reliable
than costings of capital gains tax (CGT) revenue. This is because ITW revenue tends to be very
stable from year to year, affected only at the margin by variations in nominal wages growth
and employment growth.
Factors influencing the reliability of costings of policy proposals
3
On the other hand, CGT revenue can vary substantially as asset prices rise or fall and as
taxpayers adjust the realisation rate of assets in response to asset price changes.
The certainty with which the costing base can be forecast will generally decline the longer the
forecasting horizon becomes because forecasting errors tend to compound over time.
3
How these factors are taken into account
The PBO uses a qualitative assessment process to determine the reliability rating for its
costings. Table 1 provides a summary guide to determining the reliability rating for a costing
based on each of the factors described above. Costings may have different reliability ratings
over different timeframes, and the reliability of a costing may vary for different options that
are costed.
The reliability ratings for costings range from “High” to “Very low”. In the qualitative rating
scale a “High” rating indicates that the costing result is more certain than a rating of
“Medium-High” and so on. A “High” rating indicates that the costing result could be expected
to closely match an actual costing outcome, as would be the case with a costing of a grants
programme where the expenditure is fixed, little or no data is required to make the estimate
and there are no or few assumptions made. A “Medium” rating indicates that the costing
result is expected to be a fair estimate of an actual costing outcome, but that the costing
could be based on lower quality data, rely on some reasonable assumptions or there could be
some uncertainty around the growth rate of the costing base. A “Very low” rating is one
where the costing result is considered to indicate a broad order of magnitude only, due to
little or no quality data being available, reliance on unverifiable assumptions or a high level of
uncertainty around the costing base itself.
4
Factors influencing the reliability of costings of policy proposals
Table 1: Characteristics of costings by reliability rating and the factors taken into account
Ratings
High
Factor
Data
Underlying assumptions
Stability of costing base
High quality data that would be
Modelling with few or no
Well established, stable and predictable
expected to have a very low
assumptions as it utilises
growth rates, indexation and behaviour.
variance, which can be
complete high quality data.
Examples include use of consistent
considered close to zero error
recipients for a program across time
on the basis that it is actually a
periods.
census of the available
information.
Examples: ATO tax return data,
unit-record data (ie Medicare).
ABS data with no caveats.
Medium-High
High quality data with some
Modelling with few
Established and reasonably predictable
chance of error.
assumptions that are less
growth rates, indexation and behaviour.
certain and do not have a
Examples may involve a new or changed
significant influence.
program or tax for which behaviour is
Examples include ABS survey
based statistics marked ^ or
*3.
fairly predictable.
Medium
Basic data only, of reasonable
Modelling with assumptions
Either growth rates, indexation or
quality that may be from less
that have an influence but
behaviour are not established and less
reliable sources (than above).
are reliable.
predictable.
Examples include non-
Examples may include significant new
government studies and
program or tax or existing program/tax
surveys and ABS survey based
for which behaviour is hard to predict.
statistics marked
Medium-Low
**4.
Incomplete data, of reasonable
Modelling with assumptions
A combination of growth rates,
quality.
that have a significant
indexation and behaviour are not
Examples include data as above
influence and are less
established and less predictable.
however, missing a parameter
reliable.
Examples include projecting demand-
or time periods.
driven programs based on highly
volatile historic trends in growth.
Low
Little data, much of it low
Modelling includes
Growth rates and behaviour (including
quality.
unverifiable assumptions
any other factors) are volatile.
Examples include use of proxy
that would have a significant
Examples include reliance on factors
data with an indirect link to the
influence.
independent of Government policy.
Very little or no available data
Costing cannot be completed
Almost no information on potential
and of poor quality.
without relying on
growth rates and behaviour.
unverifiable assumptions.
For example, Capital Gains Tax.
required parameters.
Very-Low
Note all three factors include the development of forecasts/projections.
3
For example, “^ estimate has a relative standard error of 10 per cent to less than 25 per cent and should be
used with caution” or “* estimate has a relative standard error of 25 per cent to 50 per cent and should be
used with caution” (ABS Cat. No. 8155.0 - Australian Industry, 2012–13).
4
For example, “**estimate has a relative standard error of greater than 50 per cent and is considered too
unreliable for general use” (ABS Cat. No. 8155.0 - Australian Industry, 2012–13).
Factors influencing the reliability of costings of policy proposals
5
www.pbo.gov.au
Factors influencing the reliability of costings of policy proposals
1