DEVELOPING KEY NATIONAL INDICATORS FOR
THE UNITED STATES
KATHERINE WALLMAN, KENNETH PREWITT, AND SUSAN
SCHECHTER
Comments by John Barber
This is a project with ambitious objectives which deserves our support
and which other countries will hope to emulate. However it faces a
number of problems and pitfalls many of which are referred to in the
paper. I propose to comment on these from the perspective of a potential
user. In my time I have worked as a macro-economic forecaster, as an
energy analyst and latterly as an innovation policy maker. Recently I have
been engaged in a study for the OECD which among others covers the
role of Science, Technology and Innovation indicators in the formulation
of technology and innovation policy.
2. The authors define an indicator as a statistical measure that tracks
change over an extended period of time. They set out a number of strict
criteria which any series to be included in the “Key National Indicators
Initiative” (KNII) must meet. These include ‘accuracy’, lack of variation
unrelated to real changes in the underlying phenomenon being measured
and completeness which reflects the degree of match between what the
indicator measures and the phenomenon of interest.
3. The use of indicators to understand what is happening, by policy and
business analysts and by researchers, requires that the phenomenon which
the indicator represents must be embedded in some kind of structural
explanation, theory or model describing the system in which that
phenomenon is embedded and which determines the value of the
indicator and the influence that the phenomenon concerned exerts on
related entities. The significance of the indicator can only be properly
understood in relationship to the system concerned and in the movements
in the other indicators/variables which help to describe the functioning of
the system. The authors provide an excellent of this when they refer to
climbing US divorce rates.
4. The divorce example throws up another important issue. Because the
world in which we live is complex and our understanding of most
economic and social phenomena is somewhat limited, there are often
competing explanations or hypotheses attempting to explain the same set
of related phenomenon and indicators. In the absence of perfect
knowledge there is scope for politicians, analysts and researchers to hold
a range of different prior beliefs. This tends to further increase the variety
of explanations on offer. The same indicator may have a different
significance for different groups or individuals depending on which
explanation they subscribe to. Moreover individual explanations evolve
over time, new ones arise and others go out of fashion as circumstances
change. For example I heard stockbrokers say in at different times in the
1970s that rapidly rising supply of money caused both rising and falling
share prices in London. Thus, even if an indicator continues to closely
represent an unchanging phenomenon, its significance for analysts,
policy-makers and researchers will nevertheless change over time. [Since
the 1960s the principle focus of UK macro-economic policy has varied
among the current balance of payments, the money supply, the exchange
rate, the growth of money GDP and the rate of inflation.]
5. Competing explanations require different data sets and focus on
different indicators and thus make different demands on official
statisticians. Some important and relevant phenomena can be measured in
line with the criteria set out by the authors for inclusion in the KNII but
this is not possible for others. For example it is possible to obtain
reasonable data for R&D undertaken by firms but not for other forms of
technology generation such engineering development. As the relative
importance between R&D and other forms of technology generation vary
systematically across sectors our inability to measure the latter can
significantly distort the making of innovation policy. We need to be
careful that the part of economic and social development which we can
measure does not cause us to ignore equally important phenomena for
which measurement is either more difficult or not possible at all.
6. The boundary between Business R&D and other forms of technology
development is described in the OECD Frascati Manual which underpins
the definition of R&D for tax purposes in a number of countries. Despite
the OECD’s best efforts there are suspicions both in the industrial and the
policymaking communities that the definition is not applied consistently
across advanced industrial countries and that caution needs to exercised
when comparing R&D performance across countries. International
comparisons of the results of innovation surveys, answers to which are
often subjective, involves significant difficulties. These sorts of problem
can without doubt be found in other areas of policymaking.
7. National Indicators are by their nature aggregates across sectors,
regions or different groups of the population. Even if the phenomenon
being measured remains unchanged within each subcategory the relative
importance of the various categories can change significantly over a
longish period of time. If behaviour differs across subcategories then the
aggregate behaviour of the indicator will change even though behaviour
remains unchanged within the individual categories themselves. Both
providers and users of data need to be constantly on their guard against
such ‘aggregation bias’. In the 1960s Professor Terence Gorman set out
the qualities required of aggregate economic indicators for use in estimate
structural behaviour. While his approach may have represented a counsel
of perfection it served to demonstrate the kinds of restrictive assumptions
which are implicit in estimating behavioural relationships between
aggregate indicators.
8. Macro-economic analysts and forecasters often focus on the correlation
between various indicators (so-called reduced form relationships). This is
often used to calibrate less well founded data sources against more
established indicators. Such data sources can provide useful ‘lead
indicators’ of key official statistical sources. Similarly a consistent pattern
of reduced form relationships between related data series can generate
confidence in the picture they present even if the quality of each of the
individual series is indifferent.
9. Some indicators such as infant mortality or GDP represent social or
economic outputs which are desirable in the own right. Others are mainly
instrumental, for example we target the money supply in order to control
inflation. Both kinds of indicators may be used as policy targets or
measures of policy success encountering a number of possible problems:
 Some indicators may be subject to ‘Goodhart’s Law’ in that using
them as a target may change their behaviour and significance e.g.
money supply, number of articles published in academic journals;
 Focussing on one indicator to the exclusion of others may mean
that the measured success of a policy may be bought at the expense
of other desirable outcomes e.g. hospital weighting lists,
proficiency of school children at certain core subjects;
 Some indicators may not be as complete a measure of the
corresponding phenomenon in which e.g. the long-running debate
on the deficiencies of GDP as a measure of economic welfare.
10. The authors see an important role for indicators in allocating
resources to different public uses. One might perhaps distinguish between
the justifying a given allocation of resources to a wider public and the
process of deciding what that allocation of resources should be. The latter
involves the weighting of different desirable outcomes which is difficult
and contentious process as illustrated by Professor Kenneth Arrow’s
work on the impossibility of constructing a social utility function. It also
involves comparison of incommensurable outcomes e.g. educational
attainment of young children against the health of old people.
11. I would like to finish by urging those responsible the KNII to closely
involve expert users at all stages. Their work requires them to take a close
interest in the construction and significance of the data and they are often
the first to spot any unforeseen errors, any changes in the behaviour and
significance of an indicator and to identify any special factors which may
distorting its movement. They often have access to other data sources
which can validate the indicator or warn of emerging problems. They are
also likely to be an important source of demand for new indicators.
12. Finally I want to wish the authors and their colleagues success in
constructing the KNII. None of the above is intended to deter them in
their task but instead to help ensure that the indicators are used as
effectively as possible.
John Barber
11th November 2004