History and design of the
European Community
Innovation Survey (CIS)
questionnaire
Prof. Anthony Arundel
Australian Innovation Research Centre, University of Tasmania, Australia
and UNU-MERIT, Maastricht, the Netherlands
1. Introduction
Responses to the
CIS from over
200,000
enterprises
Conducted every
two years (7
surveys so far)
CIS 2014 under
development
Uses of the CIS-1
• Policy relevant indicators used in the 2013
European Innovation Union Scoreboard:
1.
2.
3.
4.
Non-R&D innovation expenditures
Percent of SMEs that innovate in-house
Percent of SMEs that collaborate on innovation
Percent of SMEs with product or process
innovations
5. Percent of SMEs with organisational/marketing
innovations
6. The share of total sales from innovative products
National performance on the Innovation
Union Scoreboard, 2013
Benchmarking
Policy uses of
innovation
surveys:
Benchmarking
motivates, but indepth analysis is
required to know
what to do.
In-depth
analyses
Number of academic papers using Community Innovation
Survey data (1994-2012)
500
100
451
450
428
90
Cumulative Number
Number of studies
400
367
Annual Count
350
80
70
316
300
60
261
250
50
211
200
165
182
140
150
40
30
100
100
81
50
4
9
14
20
27
34
20
50
10
0
0
1994 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
Publication year
Research themes over time
40
35
Who innovates
Innovation intensity
Number of papers
30
Innovation strategies
25
Effects on performance
20
15
10
5
0
1994
1996
1997
1998
1999
2000
2001
2002
2003
2004
Publication year
2005
2006
2007
2008
2009
2010
2011
2012
Effects on performance
1. CIS output indicators
- Change in annual turnover or employment over
three years
- Innovation sales share
- Crude estimate of productivity
2. Data linkage to administrative data for
profits, productivity, production etc.
3. Panel data: can examine effect over time on
innovation sales share
2. R&D versus innovation
The origins – R&D surveys
• First attempts to measure R&D:
– 1917: First R&D survey, with further
experimentation up until the late 1930s.
– 1953: First large American R&D survey.
– 1963: First international R&D survey.
– 1981: OECD considers R&D data quality
and international comparability acceptable.
R&D focus of early
innovation research
• Are innovation
activities a
supplement to
R&D? (perspective
of Frascati Manual)
or
• Is innovation is a
unique activity that
may or may not
involve R&D?
13
June 2008
14
2003 hourly labour productivity by 1998 R&D intensity
Labour productivity (100 = EU average)
150
NO
Lux
125
BE
R2 = -0.04
FR
IRL
US
NL
DE
100
UK
ES
DK
FIN
AT
IT
IS
J
75
50
0
0.5
1
1.5
2
2.5
R&D intensity (GERD/GDP)
3
3.5
4
What is going on?
• There is a lot more than R&D – other factors
contribute to economic development and
productivity improvements.
– Technology purchases (adoption as in the
fishing sector)
– Organisational innovation
– Innovative activities that do not require R&D
16
Ways of innovating
without R&D
• Technology adoption
• Minor modifications or incremental
changes, including use of engineering
knowledge
• Imitation including reverse engineering
• Combining existing knowledge in new
ways
Percent of firms
innovating without R&D
45.4% of innovative firms in this sample did not perform R&D
Of R&D performing firms, 63% reported innovations that did
not require R&D
Method of innovation (from least to most advanced
in-house capabilities) for product/process
innovation and for organisational innovation
R&D versus innovation
• R&D is largely an activity of manufacturing
firms – much rarer in services
– Innovation data are a better measure of
innovation activities in services
• Definition of R&D (requirement for
scientific or technological novelty) is
confusing for firms in services
– What about development without research?
Definition of Innovation
• An innovation is the implementation of a new
or significantly improved product (good or
service), process, new marketing method, or
new organisational method.
Source: 3rd revision, Oslo Manual, OECD
– Innovation is NOT invention.
– It does NOT require creative effort – adoption of
new manufacturing technology is an innovation.
– It does NOT need to be commercially successful.
22
Percent of innovative firms in
Europe that do not perform R&D
• Innobarometer 2007: 52.5% of
innovative firms did not perform R&D.
• CIS-3 (2002- 2004): 50% did not
perform R&D
Percentage of Innovative and R&D Performing
Firms by Size: CIS-1 Estimates for 8 Countries Combined
100
90
80
70
60
50
Innovators
R&D performers
40
30
20
10
0
< 20
20-99
100-249
250-499
500-999
1000-1999
Number of Employees
Countries: Germany, Italy, Belgium, Netherlands, Luxembourg, Ireland, Denmark, Norway
> 2000
Innovative firms that did not perform R&D in 2000: Breakdown by Country (CIS-3)
100%
90%
80%
70%
60%
50%
40%
79
30%
20%
41
36
34
46
48
46
50
55
53
52
65
60
59
55
69
10%
Non R&D Innovators
In between R&D Innovators
R&D Innovators
BG
IS
RO
ES
LV
CZ
SK
EE
PT
Av
er
ag
e
LT
EU
DE
HU
G
R
BE
NO
0%
Innovative status of an enterprise
in the CIS
• An innovative firm has introduced one or more
types of innovations within a defined period of
time (3 years).
• Innovative status can depend on the maximum
degree of novelty of at least one of the firm’s
innovations:
– ‘World first’ – often based on in-house creative
effort.
– New to the firm’s market: could be based on
creative effort (reverse engineering, engineering
improvements) or bought from another firm.
– Only new to the firm: purchased off the shelf.
Innovation as a creative versus
diffusion activity
Maximum Creative
effort
New product/process
resulting from large
investment in in-house R&D
Purchased technology is
adapted to firm’s needs
Minimum
creative effort
New technology bought
‘off the shelf’
R&D status by change in turnover 2004 to 2006
No R&D
R&D
in-house
Decreased > 25%
1.7%
1.7%
Decreased 5% - 25%
7.8%
7.8%
Little change
54.6%
52.9%
Increased 10% to 50%
32.7%
34.6%
Increased > 50%
3.2%
3.0%
100.0%
100.0%
Change in income
Source: Arundel, Bordoy & Kanerva, 2008
Total
Results confirmed in an econometric model that controls for size,
sector, country, total innovation expenditures, innovative capabilities.
3. History of the Community
Innovation Survey (CIS)
First innovation surveys
• Object based: collect data on specific innovations
identified in trade journal advertisements or other
sources.
– Townsend, 1981, Kleinknect (various studies)
• De Bresson & Murray (1984): stratified random survey of
all Canadian firms (not just R&D performers), asked for
descriptions of three most important innovations.
– Defined innovation as “any new or improved product
which has withstood the trial of the market and
generated a return on investment, or a new or
improved process for commercial production. By new
we mean new to Canada”
.
Subject-based innovation surveys
• The focus is on the firm or enterprise, with
questions about its innovation activities.
• First surveys in the 1970s in Canada, limited to
R&D performing firms.
• Advantages over an object approach:
– Can collect information on all types of innovations
– Can cover process and other types of innovations.
Experimental innovation
surveys of the 1980s
• MIT (United States) & Fraunhofer (Germany) surveys
only sent to R&D performing firms.
• Scandinavian survey: conflicting definitions, product
innovations defined as deriving from ‘R&D projects that
resulted in marketable new products”.
• Ifo (Germany) referred to new or improved products and
processes.
• Dutch, French and Italian surveys asked if the firm had
introduced an innovation that was new to the firm or its
local region.
First Oslo Manual (1992)
• “The core task is to integrate an understanding of the
R&D contribution with an account of the non-R&D inputs
to the innovation process”
– Innovation is a supplement to R&D
• What differentiates a change from an innovation are
“elements of novelty and significance”
– (similar wording to the Frascati Manual)
• But, definition of an incremental innovation is ambiguous
about the need for R&D: “an incremental product
innovation is an existing product whose performance has
been significantly enhanced or upgraded.
CIS-1 (implemented in 1993)
• CIS-1 definition of a product innovation
• “A significant innovation is a newly-marketed
product whose intended use, performance,
characteristics, technical construction, design, or
use of materials and components is new or
substantially changed.”
• “An incremental innovation is an existing
product whose technical characteristics have
been enhanced or upgraded.”
– Whether or not R&D is required is left undefined.
CIS-2 to CIS-4: removing
ambiguity over R&D
• CIS-2 (implemented in 1997):
– Asked respondents ‘who developed’ their innovations and
included the option ‘mainly other enterprises or organisations’
• CIS-3 (implemented in 2001):
– Changed definition of product and process innovations to state
that they only needed to be ‘new to your enterprise’.
• CIS-4 (implemented in 2005):
– Requirement for R&D for collaboration removed, with definition
of collaboration changed to ‘joint R&D and other collaboration
projects’
Survey
CIS-1
CIS-2
Observation Main changes or additions compared to the previous survey
period
1990-1992
Added questions on who developed product and process innovations.
1994-1996
Cooperation question extended to cover more than R&D.
Deleted questions on sources of new technology, technology transfer outside the
enterprise, appropriation methods, and product life cycles.
CIS-3
1998-2000
Changed question on the innovation sales share to the share of sales from
unchanged, new-to-firm and new-to-market products.
CIS-4
2002-2004
Added questions on three types of organizational innovation and two types of
marketing innovation, plus questions on the effects of organizational innovation.
Under product innovation included separate questions for goods and services.
Asked about three types of process innovations.
CIS-2006 2004-2006
Frequency increased to every two years.
Implemented cognitive testing for all question changes and additions.
Increased coverage of organizational and marketing innovation.
One-page module for questions of high policy interest, with the first module on
environmental innovation.
Separate question on expenditures for design.
Module on creativity and skills.
Module on strategies and obstacles to growth.
Reintroduced a modified version of the CIS-1 appropriation question.
New questions on public procurement and innovation
CIS-2008 2006-2008
CIS-2010 2008-2010
CIS-2012 2010-2012
But, the forces for
R&D might be back,
with a proposal to
combine the CIS
and R&D surveys!
Countries combining the CIS 2010 with the R&D survey
No
76%
Yes, but only
for some
respondents
3%
Yes, for all
respondents
14%
Not relevant
7%
0%
10%
20%
30%
40%
50%
60%
70%
80%
4. The CIS today
• The European CIS is a ‘general’
innovation survey that covers a large
number of topics, none of them in
particular depth.
– Exception: knowledge sourcing
Limitations of the CIS
• Cross-sectional survey –only a few
countries have panel data:
– Norway, Germany, Tasmania (not a CIS
survey but has some similar questions)
• As a general rule, does not include
questions on rare activities
– These questions collect little information of
value but add to the response burden
Who answers the CIS?
Ideally, the CEO or R&D manager
Who might use the CIS?
Policy analysts
Businesses
Academics
Three different users
• Business managers
– Need timely data for benchmarking against other
firms A general failure of large surveys, but
feasible for smaller surveys.
• Academics
– Need access to micro data for testing innovation
theory (many data access limitations)
• Policy community (main target)
– Need results that are directly relevant to policy
issues (rarely provided by academics); often rely
on indicators and descriptive analyses but
econometrics are useful if properly explained.
Definition of innovative
enterprises
• At least one product, process, marketing
or organisational innovation in the
preceding three years.
• Only needs to be ‘new to the firm’ –
includes simple technology adoption.
– Criticized for being too broad
Indicator for ‘how’ enterprises innovate: Spain
Strategic
25
20
EU average
15
10
5
Adopters
0
Intermittent
Modifiers
Finland versus Portugal
Strategic
Strategic
Adopters
25
25
15
15
5
5
-5
Modifiers Adopters
Intermittent
Finland
-5
Intermittent
Portugal
Modifiers
Main players in the CIS
• Eurostat (statistical agency of the European Union) coordinates the
design of the CIS questionnaire and makes recommendations for the
survey methodology.
• CIS Working Group: All European countries that implement the CIS
are members. OECD and the European Commission also attend.
They assess the revised questionnaire from the Task Force and can
request changes. Decision to accept based on consensus.
• National Statistical Offices: Responsible for implementing the CIS
in their country. By EU law, must provide specific indicators to
Eurostat; can voluntarily provide full data for the Eurostat Safe
Centre and anonymized data.
CIS Task Force
Topics covered in the CIS
• Firm’s main markets
• Product and process innovation
–
–
–
–
–
–
–
•
•
•
•
Who developed
Innovation sales share (output variable)
Novelty (world-first, Europe first, country first)
Innovation activities & expenditures
Public support
Sources of information
Types of collaboration partners
Organisational and marketing innovations
Barriers to innovation (drivers in CIS-2014)
Innovation objectives (not asked in every CIS)
Module of questions of policy interest
Percent of UK firms reporting cost
factors as of high importance as barriers
to innovation (CIS-4)
Source: D’Este et al, 2012
Proposed new question on barriers
for non-innovative enterprises
Importance of innovation drivers
Modules
• CIS 2008:
– Environmental innovation
• CIS 2010:
– Creativity and skills (including how the firm encourages creativity)
• CIS 2012:
– Strategies and obstacles for growth (meeting enterprise goals)
– Appropriation methods (also used in CIS-1)
• CIS 2014:
– Improved questions on environmental innovation, including
importance of different drivers
Required & optional CIS
questions
• Many questions required by EU law (EC
regulation 1450/2004)
• Optional questions include:
– Novelty (world first) of product and process
innovations
– Innovation module
– Questions on public procurement, appropriation
• National statistical offices can and do add
other questions of high relevance to their
own policy issues.
Keeping the CIS relevant
CIS questions used in
benchmarking need to stay
the same, but new questions
and topics are needed every
year to attract academic
researchers and evidence for
topical policy questions.
In response, since CIS 2008
the questionnaire includes a
module of one-off questions
that can change in every CIS.
System for continuous
improvement of the CIS
1. Cognitive testing of all new questions & major changes
2. Quality reports for each CIS survey
-non-response rates for specific questions
- Experiences of National Statistical Offices
3. Research by NSOs on methodology
- Effect of combining R&D and innovation surveys
- How respondents understand key concepts
4. Bi-annual surveys of NSOs
- Survey method (online, mailed, face-to-face)
- Post survey methods for improving data quality
5. Surveys of policy users
- What innovation survey data do they use
6. Database of academic papers that use the CIS
- Which questions do they use and do not use
- New questions that academics would like for their research
Cognitive testing
• Face-to-face interviews with firm
managers
– Goal to ensure that all questions are
understood as intended by all respondents,
and
– Respondents can provide accurate responses
• Introduced on a consistent basis in 2004
(recommendation of report on CIS-4).
Logical error
1. What was your firm’s domestic and world-wide sales in 2010?
Don’t know
______________’000 $

______________’000 $

Logical error due to
question placement
1. Does your business unit cooperate with other business
units, firms, or institutions to develop new technologies?
Yes
No




Cooperation is defined as active participation with other firms, other
technology users or suppliers, organizations or individuals (other than
those in your business unit). Contracting out, where there is no active
participation is not considered cooperation.
2. Does your business unit share the new technologies that it
has developed with other firms or institutions?
Sharing is defined as intentionally allowing other firms or institutions to
replicate or use the new technologies developed by your business unit.
Accuracy of responses
(not all problems can be solved)
In-house R&D (Include current expenditures including labour costs and
expenditures on buildings and equipment specifically for R&D)
capital
External R&D
Acquisition of machinery, equipment, software & buildings
expenditures on these items that are for R&D)
(Exclude
Acquisition of existing knowledge from other enterprises or organisations
All other innovation activities including design, training, marketing, and other
relevant activities
Total expenditures on innovation activities (Sum of expenditures for all
innovation activities)
types of
Two questions in one
During 2010 to 2012, how important were the following factors in driving the introduction
of innovations with environmental benefits in your enterprise?
High
Medium
Low
Not
relevant
Existing environmental regulations and taxes




Need to improve reputation to ensure consumer loyalty




Government grants, subsidies or other financial incentives for environmental innovation




Item non-response rates:
use of quality reports to track progress
• Improvements to question design:
– Decreased item non-response rates for the
innovation sales share question from
approximately 25% to less than 3% by CIS-4
– Decreased item non response rates for the
question on innovation expenditures from
36% to approximately 15% by CIS-4.
Research by National
Statistical Offices
• Norway tested the effect of combining the
R&D and CIS surveys
– Found small declines in number of innovative
firms.
• Netherlands compared responses to online
and mailed surveys.
– Found much higher share of innovators to
online surveys (70%) compared to mailed
surveys (50%)
NSO survey
• For every CIS, survey NSOs on a range of
relevant questions
• Topics include:
– issues with new questions in the previous
survey
– survey methods, particularly the use of online
surveys
– New questions that may have tried and which
might be of interest to all NSOs
NSO survey results
Main survey method for CIS 2010
60%
55%
50%
40%
38%
30%
20%
10%
7%
0%
0%
Mail
Online
Telephone interview Face to face
0%
Other
Combinations of survey methods for CIS 2010
Mailed with online as
secondary method
14%
Online with mail as
secondary method
45%
Mailed with telephone
interview as secondary …
17%
Online with telephone
interview as secondary …
17%
Other combinations
7%
0%
10%
20%
30%
40%
50%
Software for online surveys
25
20
20
15
10
5
3
1
0
0
Created online Survey monkey
survey internally
Lime survey
24 countries provided an online option (80%)
Other commercial
software
Possible questions for removal
from CIS 2014 (% of countries)
First choice for
removal
Second choice
for removal
Last choice for
removal
Sources of
information (Q6.1)
31.0%
34.5%
34.5%
Cooperation and
type of cooperation
(Q6.2 – Q6.4)
13.8%
24.1%
62.1%
Public sector
procurement and
innovation (Q10)
48.3%
31.0%
20.7%
Number of NSOs that added a non-standard
question to their national 2012 CIS survey
18
16
16
14
14
12
10
8
8
6
4
2
0
Yes
No
Useful for future CIS
surveys
Surveys of policy users
• Infrequent because expensive
– Last large scale survey in 2005
– New survey planned for 2014
• Interested in the types of indicators that
policy makers find useful
• If CIS results have influenced policy
2005 Policy survey results
R&D
indicators
CIS indicators
Academic users
• Small-scale surveys of10 to 20 leading
academics, based on number of
publications using CIS data.
• Ask which types of new questions they
would like, which questions perform
poorly in analyses, etc.
• Maintain database of publications using
the CIS.
Academic use of CIS data
Percent of studies by year that 1) examine non-R&D innovation
and 2) discuss the relevance of the results for policy
70.0
61.8
60.0
55.2
Non-R&D
Policy
54.0
50.0
44.3
44.0
40.9
40.0
30.0
18.0
20.0
9.9
12.1
13.8
10.0
10.0
11.5
7.3
9.1
0.0
Up to 2006
2007
2008
Results for 2012 incomplete
2009
2010
2011
2012
Research themes over time (percentage of annual count)
70%
Effects on performance
60%
Innovation strategies
Percentage of annual count
50%
40%
30%
20%
10%
0%
2001
2002
2003
2004
Results for 2012 incomplete
2005
2006
2007
Publication year
2008
2009
2010
2011
2012
5. Conclusions:
Key lessons for a useful
innovation survey
10 recommendations: not all of which
have been fully implemented in Europe
1. Collect data on question quality, including
item non-response rates, ‘not relevant’
response rates, number of respondents who
contact you for better explanations, etc.
2. Cognitively test all new questions and major
changes to existing questions (50 interviews)
3. The working group to design innovation
questionnaires should include
representatives from both statistical offices
and user communities.
4. The addition or deletion of CIS questions
should be backed by empirical data on
data quality and on the relevance of the
question to users.
5. Allow room for experimentation with new
questions – perhaps by province?
6. System for continuous improvement.
7. Add new questions in each CIS – keep it
interesting for respondents and for users.
Recommendations that have not been
fully implemented in Europe
8. Provide academics with timely access to the
data.
9. In return for data access, insist that
academics provide a ten page evaluation of
the relevance of their research for policy.
10.Develop a strong interface between the
policy community and the National
Statistical Office to encourage the uptake of
CIS results for policy uses.
And two more….
• Create a panel data set (possibly a subsample)
• Provide data linkage to other data,
especially for outcome analyses
Thank-you!
For questions or more information
a.arundel@utas.edu.au
Detailed CIS reports available for
last 5 CIS surveys.