2014 TAFTIE Policy Forum
“Measuring innovation”
Can we measure innovation?
Lessons from innovation scoreboards
Hugo Hollanders
Brussels, November 25, 2014
www.taftie.org
www.parp.gov.pl
What is an innovation scoreboard?
• “Innovation scoreboards consist of a collection of
national or regional indicators of relevance to
innovation”
Arundel, A. and H. Hollanders, "Innovation Scoreboards:
Indicators and Policy Use", in C. Nauwelaers and R. Wintjes (eds.),
Innovation Policy in Europe, Edward Elgar: Cheltenham, 2008
• “Innovation scoreboards are an attempt to
summarize innovation indicators and to compare
innovation performance of countries, regions or
sectors”
Hollanders, H. and N. Janz, "Scoreboard and indicator reports", in
F. Gault (ed.), Handbook of Innovation Indicators and
Measurement, Edward Elgar : Cheltenham, 2013
www.taftie.org
www.parp.gov.pl
Purpose of innovation scoreboards
• The main purpose of an innovation scoreboard is to
assist policy by summarizing a range of innovation
indicators at the national, regional or sectoral level
by permitting a comparison of the relative success
or failure of the innovation system, or through the
identification of specific aspects of the innovation
system which perform well or poorly
www.taftie.org
www.parp.gov.pl
Policy relevance
• Innovation scoreboards serve three policy needs
• They act as an ‘early-warning’ system for potential
problems
• If used over time, they can track changes in innovation
strengths and weaknesses
• They can attract the interest of policy-makers and as such
can help build a consensus to introduce actions and
policies to improve the innovation capabilities of firms and
institutions
• Policy use
• Innovation scoreboards will be most useful for policy when
combined with an integrated system for evaluating
innovation policies
www.taftie.org
www.parp.gov.pl
Principles of scoreboard design
• The indicators should
• Be of similar importance as measures of the drivers of
innovation activity
• Be based on reliable statistics
• Hold their value over time
• Be of relevance to medium- and long-run policy issues
www.taftie.org
www.parp.gov.pl
Examples of innovation scoreboards
• European Commission
• Innovation Union Scoreboard (IUS)
• Innovation Union Competitiveness Report
• OECD
• Science, Technology and Industry Scoreboard
• OECD Science, Technology and Industry Outlook
• INSEAD/WIPO
• Global Innovation Index
• World Economic Forum
• Global Competitiveness Report
• World Bank
• Knowledge Assessment Methodology
• Deutsche Telekom Stiftung, BDI
• Innovationsindikator
www.taftie.org
www.parp.gov.pl
Examples of innovation scoreboards
Name
First Last Frequency Standard Target
Number thereof
Ed. Ed.
structure countries of
based on
indicators innovation
surveys
European
MERIT
2001 2014 Annual
Yes
EU
25
6
Commission
Member
States
European
European
2011 2013 Biennial
unknown EU
51
None
Commission Commission
Member
States
OECD
OECD
1991 2013 Biennial
No
OECD
~180
34
countries
Selection
criteria
OECD
OECD
1998 2012 Biennial
No
Not
available
INSEAD,
WIPO
Telekom
Stiftung,
BDI
INSEAD
2007 2014 Annual
Yes
Fraunhofer 2005 2014 Annual
ISI,
ZEW, MERIT
Yes
Global
Competitiveness
Report
World
Economic
Forum
Knowledge
Assessment
World
Bank
Centre for
1979 2013- Annual
Global
2014
Competitiveness and
Performance
World Bank 2001 2012 Regularly
updated
Innovation Union
Scoreboard
Innovation Union
Competitiveness
Report
OECD Science,
Technology and
Industry Scoreboard
OECD Science,
Technology and
Industry Outlook
Global Innovation
Index
Innovationsindikator
www.taftie.org
www.parp.gov.pl
Editing
Institution
Preparing
Institution
Composite
Indicator
Reasoning, Yes
Correlation
analysis
No
Reasoning No
OECD
22
countries (country
fiches)
World
84
None
None
Yes
None
Model,
Yes
Regression
analysis
Yes
Germany 38
and
selected
countries
World
116
None
Yes
Yes
World
None
Yes
148
No
Innovation Union Scoreboard
• IUS monitors the implementation of the
Europe 2020 Innovation Union flagship by
providing a comparative assessment of the
innovation performance of the EU28
Member States and the relative strengths
and weaknesses of their innovation systems
• IUS uses 25 indicators which are grouped into 3
main categories:
• Enablers capture the main drivers of innovation
performance external to the firm
• Firm activities capture the innovation efforts at firm level
• Outputs capture the effects of firms’ innovation activities
• Average performance is measured by a composite
indicator, the Summary Innovation Index
www.taftie.org
www.parp.gov.pl
8
IUS Measurement framework
Summary
Innovation
Index (SII)
ENABLERS
Human
resources
Open, excellent, attractive research
systems
OUTPUTS
FIRM
ACTIVITIES
Finance and
support
Firm
investments
Linkages &
entrepreneurship
Intellectual
assets
Innovators
Economic
effects
New doctorate
graduates
International
scientific copublications
R&D
expenditure in
the public
sector
R&D
expenditure in
the business
sector
SMEs
innovating inhouse
PCT patent
applications
SMEs with
product or
process
innovations
Employment
in knowledgeintensive
activities
Population
aged 30-34
with tertiary
education
Top 10% most
cited scientific
publications
Venture
capital
investments
Non-R&D
innovation
expenditure
Innovative
SMEs
collaborating
with others
PCT patent
applications in
societal
challenges
SMEs with
marketing or
organisational
innovations
Medium-high
& high-tech
product
exports
Youth with at
least upper
secondary
education
Non-EU
doctorate
students
Public-private
copublications
Community
trademarks
Employment
fast-growing
firms innovative sectors
Knowledgeintensive
services
exports
Community
designs
Sales of new
to market and
new to firm
innovations
as % of
turnover
License and
patent
revenues from
abroad
www.taftie.org
www.parp.gov.pl
9
IUS Performance groups
www.taftie.org
www.parp.gov.pl
10
Differences in rankings
www.taftie.org
www.parp.gov.pl
IUS International comparison
• Europe’s performance gap with the US and Japan is
closing but the gap with South Korea is increasing
• Europe is
www.taftie.org
www.parp.gov.pl
Lessons from innovation scoreboards
• What are the strengths and weaknesses of the
innovation measurement tools and indicators we
use today?
• Strengths
• Country coverage, measuring variety of innovation
activities, good coverage of inputs and throughputs, use of
innovation survey data
• Weaknesses
• Partial coverage of outputs, use of innovation survey data
www.taftie.org
www.parp.gov.pl
Lessons from innovation scoreboards
• What’s the state of play on the development of new
indicators?
• Community Innovation Survey 2014
• Better coverage of public sector as a driver of innovation
• Public procurement
• Ad-hoc modules: 2012 on growth, 2014 on innovations with
environmental benefits
• EC’s Innovation Output indicator
• Combines 5 of the IUS indicators into one indicator measuring
outputs of which one is a new indicator:
• Employment in fast-growing firms of innovative sectors
www.taftie.org
www.parp.gov.pl
Fast-growing firms innovative sectors
• 2010: High-Level Panel asked to recommend new
headline innovation indicator
• 2011-2013: work by OECD/Eurostat to identify the
innovativeness of sectors
• 2013: indicator launched by EC
• Employment in fast-growing firms of innovative sectors
provides an indication of the dynamism of fast-growing
firms in innovative sectors as compared to all fast-growing
business activities. It captures the capacity of a country to
transform rapidly its economy to respond to new needs
and to take advantage of emerging demand
www.taftie.org
www.parp.gov.pl
Lessons from innovation scoreboards
• What has not been measured yet?
• Outputs, current indicators focus on
• Employment in high-tech and knowledge-intensive activities
• Exports in high-tech and knowledge-intensive activities
Both of which do not necessarily have to be directly linked to
innovation
• Sales of innovative products
Data are not available for all countries and there are issues
about the comparability of results across countries
• Commercialization activities
• Innobarometer 2014
www.taftie.org
www.parp.gov.pl
Commercialization of innovations
www.taftie.org
www.parp.gov.pl
Commercialization of innovations
www.taftie.org
www.parp.gov.pl
Thank You
www.taftie.org
www.parp.gov.pl