Innovation surveys: design, implementation,
lessons learnt
Micheline Goedhuys
Why do we need to measure innovation?

Scarcity of data in general, on innovation in
particular

Lack of policy tools for benchmarking

Insufficient monitoring and evaluation of policies

Nature of innovation calls for firm-level information
June 11 2008
DEIP, Amman June, 10-12 2008
2
Structure of session:
1.
Conceptual background
2.
Experiences with innovation surveys
3.
Methodological aspects
4.
Use of innovation survey data
5.
Opportunities and limitations to innovation data
collection: key issues
June 11 2008
DEIP, Amman June, 10-12 2008
3
1. Conceptual background



linear view that science, research and discovery
underlie innovation (science push)
innovation measured by science indicators:
 R&D
 engineers
 patenting
 bibliometrics, publications, citation indices
surveys (USA, 1960s) collecting R&D, patent data;
June 11 2008
DEIP, Amman June, 10-12 2008
4
1. Conceptual background
End 1980s, 1990s ‘activity approach’:
 investigating the ‘black box’
 innovation results from interaction firm-market,
learning, feedback (chain-link model of Kline and
Rosenberg 1986)
 need for indicators capturing non-R&D activities
and incremental change
 development of surveys asking firms about their
innovation process
June 11 2008
DEIP, Amman June, 10-12 2008
5
1. Conceptual background



Harmonisation of survey efforts in the
‘Oslo Manual’, 1992, 1997, 2005
basis for Community Innovation Surveys
innovation is measured as :
an activity (R&D, industrial design, acquisition of
machinery, external technology, training) and
an output (introduction of product or process
innovations)
Features: new-to-the-firm, significant improvements
June 11 2008
DEIP, Amman June, 10-12 2008
6
2. Experiences : CIS







CIS-1: 1990-92; first regional effort to collect
innovation data; 13 European countries,
CIS-2: 1994-1996; 17 countries
CIS-3: 1998-2000; more firms, more questions,
services, organisational change, 29 countries
CIS-light: 2000-2002, limited set of questions, 18
countries
CIS-4: 2002-2004: 29 countries, organisational
innovation and effect
CIS-2006: 2004-2006; 29 countries; no data
available yet
CIS-2008: 2006-2008
June 11 2008
DEIP, Amman June, 10-12 2008
7
2. Experiences : Latin America

Need of information to monitor the impacts of
economic reforms (trade liberalisation, privatisation,
deregulation, FDI,etc).
ARGENTINA
CHILE
COLOMBIA
MÉXICO
VENEZUE
LA
Survey
Number
I
I
I
I
I
Reference
period
1992-1996
1994-1995
1993-1996
1994-1996
1994-1996
Collection
period
1997
1995
1997
1997
1997
Agency
responsible
for survey
INDECSECYT
INE-SETPI
COLCIENCIA
-DNP
INEGICONACYT
OCEI

Source: Crespi, 2007
June 11 2008
DEIP, Amman June, 10-12 2008
8
2. Experiences : Latin America

Specific nature of innovation in Latin American
countries:
 Importance of incremental innovation;
organisational and marketing innovation;
 Importance of innovation embodied in machinery
and equipment (dissemination)
 Less private and more informal R&D
 Fragmented flows of information
June 11 2008
DEIP, Amman June, 10-12 2008
9
2. Experiences : Latin America


Need for changes to the survey instrument : Bogotá
Manual to complement OSLO Manual.
 from innovations to firm-level innovative
activities and technology efforts
 human resources, capabilities
 enlarged data need on organisational, delivery
and design innovations
lack of centralised agency, different questionnaires
and sampling methodologies
June 11 2008
DEIP, Amman June, 10-12 2008
10
2. Experiences : Latin America

Second wave of Innovation surveys:
10 countries; 2000-2001; Argentina, Brazil, Chile,
Cuba, Ecuador, Mexico, Panama, Peru, Trinidad
& Tobago and Uruguay

more of uniformity but without common
questionnaire and sampling methodologies

revision of Bogotá Manual and Annex to Oslo
Manual (2005).
June 11 2008
DEIP, Amman June, 10-12 2008
11
2. Experiences : Latin America


Third wave of Innovation surveys:
5 countries; 2003-2005; Argentina, Brazil, Chile,
Colombia and Uruguay
 A lot of exit and a core group of countries with
“consolidated” routines (but still with institutional
problems and financial issues).
 ECLAC-RICYT-OAS network (2006) to create a
harmonized “core” questionnaire (plus access to
micro-data)
June 11 2008
DEIP, Amman June, 10-12 2008
12
2. Experiences : Asia, Africa
In Southeast Asia:
 Malaysia (3), Taiwan (1), Singapore (1), Thailand
(2), China, India…
In Africa:
 South Africa (2)
 Planning to conduct an innovation survey in 20+
countries (NEPAD survey)
June 11 2008
DEIP, Amman June, 10-12 2008
13
2. Agenda:
ongoing debate to design innovation surveys to the
context of developing countries




concept of innovation : organisational, packaging,
delivery, design innovations, waste management
techniques, …
trade off between country/regional design and
benchmarking options
increasing policy relevance
inclusion of services and resource-based sectors
June 11 2008
DEIP, Amman June, 10-12 2008
14
3. Methodological aspects : questionnaire content







Basic information: name, location, industry,
ownership, year established…
Firm performance: sales, employment, …
Innovation activities: Investment, Training, intramural and external R&D, …and expenditures
Innovation outputs (product/process/organisational)
Sources of information for innovation
Cooperation for innovation
Government policy or incentives affecting
innovation
June 11 2008
DEIP, Amman June, 10-12 2008
15
3. Methodological aspects : questionnaire content

Objectives, goals or reasons for innovating

Impact of innovations on firm performance

Obstacles to innovation
June 11 2008
DEIP, Amman June, 10-12 2008
16
3. Methodological aspects



Organisation: national statistics agency, MOST,
universities, consultants
Reference period: 2 or 3 years (mostly 3)
Participation: voluntary, compulsary (in Latin
America)
June 11 2008
DEIP, Amman June, 10-12 2008
17
Venezuela
Uruguay
Thailand
Taiwan
South Africa
Slovenia
Singapore
Russia
Romania
M alaysia
Colombia
Chile
Brazil
Argentina
Turkey
Switzerland
South Korea
Slovak Republic
Poland
New Zealand
M exico
Hungary
European-17 (CIS)
Canada
Australia
0
NSA
June 11 2008
1
MOST/GOVT
2
UNIV/INST
DEIP, Amman June, 10-12 2008
3
CONSULT
18
0
str
Ca alia
EU na
- 17 da
(C
Po IS)
l
S and
Ar .Kor
ge ea
nti
n
Br a
az
i
Co Ch i l
le
lo
Ma mbia
la
Th ysia
ai
S- Aland
f ri
Ru ca
s
H
Yu ung sia
go ary
s
Ur lavia
ug
u
Me ay
xi c
o
Cz
e
Tu ch
r ke
Sw Slov y
iz
a
N. e rla k
nd
Z
Ve ealan
ne
zu d
Ta e la
Slo iwa n
Ro veni
ma a
ni
Sin Pe a
ga r u
po
re
Au
Number of Years
5
4
3
2
1
June 11 2008
DEIP, Amman June, 10-12 2008
19
3. Methodological aspects



Survey modalities: postal, PTEF follow up, personal
interview, telephone interview, online questionnaire,
CATI
Sector coverage: initially manufacturing,
increasingly services, resource based industries
Firm size: cutoff points: 5, 10, 20 or 50 workers
June 11 2008
DEIP, Amman June, 10-12 2008
20
4. Use of innovation surveys

by academics and researchers
 Innovation and firm performance
 Identify determinants/constraints to innovation
 Innovation strategies
 Regional and country studies
 Industry studies
 Innovation patterns over time
 Developing innovation indicators: measurement
issues
June 11 2008
DEIP, Amman June, 10-12 2008
21
4. Use of innovation surveys

for policy making:




June 11 2008
Indicators for benchmarking
Mapping innovation ; innovation in new sectors
Assessing trends
Monitoring specific policy instruments
DEIP, Amman June, 10-12 2008
22
4. Example:


European innovation scoreboard
Uses 20 Indicators
 Cross-country comparisons, industry comparisons
 changes over time
 consensus on policy action
uses CIS based indicators
 % SMEs with in-house innovative activities
 % SMEs that collaborate on innovation
 total innovation expenditures as % sales
 % new-to-market products/sales
 % new-to-firm products/sales
June 11 2008
DEIP, Amman June, 10-12 2008
23
5. Opportunities and limitations

Innovation surveys have become key research
inputs of modern innovation studies (Crespi, 2007)
the use of CIS data in academic research
50
45
40
amount of publications
35
30
cis1
cis2
25
cis3
20
15
10
5
0
1
2
3
4
5
6
7
8
9
10
11
12
13
years since reference year
June 11 2008
DEIP, Amman June, 10-12 2008
24
5. Opportunities and limitations

Heterogeneity across questionnaires and
methodologies remains and is even on the rise due
to broadening concept of innovation, scope, …

Lots of country studies, little cross-country
comparisons in developing countries

This limits the use of survey data as benchmarking
tool (e.g. Crespi, 2007)
June 11 2008
DEIP, Amman June, 10-12 2008
25
5. Opportunities and limitations


On the questionnaire:
 Need for the development of harmonized
guidelines with a core set of questions
 Optional policy-relevant questions can be added
for policy monitoring
Methodology for country benchmarking:
 Preferably common sampling methodology: size
cut-off point, industry coverage, …
 Compulsory common (length of) reference
period, and participation mode
June 11 2008
DEIP, Amman June, 10-12 2008
26
5. Opportunities and limitations




Dissemination of non-aggregated micro-data is
crucial
Assessing trends: need for panel data
Need of involvement of stakeholders from the start
Need for strong coordination mechanism
June 11 2008
DEIP, Amman June, 10-12 2008
27
Useful links:
For a download of the CIS-4 questionnaire:
http://www.berr.gov.uk/files/file9688.pdf

Oslo Manual:
http://epp.eurostat.ec.europa.eu/cache/ITY_PUBLIC/OSLO/EN/OSL
O-EN.PDF

Bogotá manual:
http://www.ricyt.edu.ar/interior//difusion/pubs/bogota/bogota_eng.pdf

NEPAD study:

http://www.nepadst.org/doclibrary/pdfs/innopolicy_aug2004.pdf

June 11 2008
DEIP, Amman June, 10-12 2008
28