judging the ‘impact’ of
development related research
can SIAMPI’s productive interactions help?
Evaluation of research impact
London, 17-10-2012
Jack Spaapen
2
Four issues
• The SIAMPI model
• Applicability of SIAMPI
• Options and challenges
• Relevance/potential for impact evaluation
3
What is SIAMPI and what are productive interactions
• SIAMPI is a European project (FP7) to find indicators for social impact
assessment (Research Evaluation, 20(3), September 2011)
• Productive interactions are those interactions between researchers
and relevant stakeholders that produce something: behavioral change
• Assessment focus on mutual learning instead of accounting
4
SIAMPI’s philosophy
• Impact is the wrong concept – too linear, too simple, too many
problems: attribution, temporality, missing links, reliable data hard to
find
• Social and technological innovation is the key
• Innovation is 75% social and 25% technological (Henk Volberda,
Erasmus University Rotterdam)
Social Innovation
• Broad concept ranging from societal infrastructure to conceptual
innovation
• Refers to socio-economic, cultural, legal, political spheres of society
• Core is quality of life areas: health, housing, work, (permanent)
education
• Issues: inclusion/cohesion, healthy aging, migration, urbanisation,
access to technology, opportunities for development (also H2020)
6
New coalitions to achieve S&T innovation
-
Triple helix, golden triangle: research, industry, society (government, NGOs, general
public), new collaborative arrangements: PPP (transdisciplinary research,
transepistemic communities – Knorr-Cetina, science 3.0 - Miedema)
-
Interdisciplinary input from research: natural science/technical fields, social sciences,
humanities (technical knowledge, content, use/behavior)
-
But also input from other expertise: politics, law, economists, ethics, consumerinterests, etc.
-
Co-creation of ‘ new knowledge’ and practical solutions = innovation, research by
design, iterative process (non-linear) -
-
Consensus about long term goals (“healthy aging”, “clean energy”), but in the meantime
shifting coalitions, different partners, different intermediate goals, different interests
-
Problems arise from not being used to work together (academics – industry, not
knowing what to formulate as research questions), institutional problems, political and
cultural problems….
7
SIAMPI’s philosophy [2]
• Life is not simple, not linear, research is only part of the solution
• Innovation is result of a variety of interactions between stakeholders
[researchers are stakeholders too]
• Through these (new) knowledge, expertise, experiences, demands, etc.
are shared / negotiated
• Let’s focus on that process, it’s complicated, but it can be done
8
RESEARCH IN CONTEXT
[@ TILO PROPP]
9
SIAMPI´s approach
• Analyse the mission of the research project/group
• Involve the relevant context from early on [audiences/stakeholders]
• Assessment (incl. indicators) as a joint effort of researchers and
stakeholders
• Increase commitment and awareness through focus groups
• Learning instead of accounting: focus on productive interactions
10
Productive interactions
Relations in a network that produce something of value, not only of economic value, also sociocultural, ethical, intellectual, technological, environmental, lead to behavioral change
1. Personal interactions : joint projects, advisory, consultancy, double functions, mobility
2. Interactions through media :
1. Texts : articles, books, catalogues, protocols, new diagnostics
2. Artifacts : instruments, exhibitions, models, designs
3. Support: contracts, subsidies, patenting, licensing, sharing of people and facilities
Intermediate effects/ instances of ‘social impact’ / good practices
SIAMPI indicators for productive interactions
personal interactions
between stakeholders
interaction between
stakeholders through
media
Financial / material
interaction between
stakeholders
•face-to-face meetings
•double functions, other
mobility arrangements
•phone conferences
•email
•social media
•videoconferencing
•public debate
•radio, tv, internet
•etc.
•academic journals
•professional journals
•non academic journals
•popular media
•exhibitions
•artefacts, models
•films
•master theses, graduate
projects
•standards, protocols
•social media
•etc.
•research contracts, public
and private, and mixed,
national, international
•facility, instruments
sharing
• start ups
•contribution “in kind”
(people)
•IPR arrangements,
patents, licenses
•Professional training
•Other stakeholder
interest
•etc.
12
Applicability
•
Tested in many different fields: health, social sciences,
humanities, ICT, nano research, architecture, engineering
sciences, law.
•
And in a few different countries: UK, NL, Spain, France
•
It helps to overcome some of the common problems with impact
evaluation (attribution / contribution; temporality / small steps;
robustness of indicators; commitment of participants
•
Focus is on improving collaboration and on social innovation
13
Research domain
Nano
ICT
Health care
Soc. Sciences and
Humanities
Country
NL, France
UK, NL, EU
NL
Esp, UK
Research type
Frontier, basic,
strategic
Basic, applied, TD
strategic, applied,
policy
Basic, strategic,
applied
Research mode
Academic, in
collaboration
with industry
Open to partnership
of knowledge
producers and users
Academic, open to
collaboration with
industry, government, patient
groups, professionals
Academic, open to
collaboration with
policy, institutions,
wider public,
industry
Productive
Interactions
Public
understanding,
ethical debates,
policy making,
products
Transport use,
security, interaction
between citizens and
government
Consultation,
collaboration,
regulations,
protocols, commercial exchanges,
PPPs, post
academic training
Informal links and
advice, formal
research contract
and collaborative
projects,
consultancy,
cultural events
Social Impact
Health, safety,
public
acceptance of
nano tech
Transport use,
security, interaction
between citizens and
government
Diagnostics,
treatments, safety,
general health,
policy advice
Policy tools and
techniques,
management
methods, cultural
goods and services
14
Options and challenges
• Options for development research
•
•
•
•
Mission oriented (scientific and societal goals)
Contextual, values all stakeholders alike
Helps to create commitment
Focus on mutual learning
• Challenges
• Needs out of the box thinking
• Changes in reward systems needed
• Data collection / new methods through internet
15
Example of impact in the context of NIVEL Utrecht
@Ad Prins
News media &
Communication &
Dissemination sites
60.00%
Evaluation of Health Insurance Reform
50.00%
Communication about cancer
40.00%
30.00%
Government and
Politics
20.00%
Health condition of asylum seekers
Science & Research
Monitor Social Participation of
Chronically Ill & Disabled
10.00%
0.00%
Capacity and referrals in primary
Mental Health Care
Monitor Asthma/COPD
General/Other: blogs,
searchresults etc
Health and Health
Care
Communication about cancer in
children
Local Health care effects Steel Mill
reception of reports per audience / domain (Google searches for reports / PDFs of 8 NIVEL
domains)
16
Example of impact in the context of LUMC Leiden university
© AD PRINS
Medical Journals
30
25
Chat, Blog's
20
Professional information
sites
15
10
5
University & Research
organizations
0
Dementia
Public or Patient
information sites
Epicondylitis
Thyroid Function
Libraries
Health Care Providers
Professional Organizations
Reception of medical guidelines for GPs (Google searches for three guidelines of
LUMC Dep. of Public Health and General Practice)
17
Relevance for ‘impact’ evaluation
• Aims at understanding what goes on in between science and
society and what is needed to reach ‘impact’
• Evades most common problems in impact evaluation
(attribution, temporality, robustness of data)
• Values small steps towards ‘ impact’ (intermediate impacts)
• Creates commitment of participants
• Enlightens interaction process by collecting evidence
18
EXAMPLE OF IMPACT SCORE CARD [NIVEL]
Type of
interaction
Indicator
Direct
Consultation rounds with stakeholders Adaptation in research agenda
User groups, supervising boards (PPP
Adaptation of research projects
consortia)
Indirect
Financial
Impact
Presentations to health care
professionals (conferences, meetings,
post-academic teaching)
Sustaining relations with
stakeholders, knowledge transfer
Collaboration in research
Mutual adaptation in research
projects
Annual plans
Mutual agreements with funding
agencies over future research
agenda
Implementation plans
Mutual agreements with funding
agency over future implementation
of project results
Reports & medical guidelines,
scientific publications
Knowledge transfer, Response
(uptake of knowledge) by a wider
variety of stakeholders than those
involved in direct interactions
Contracts, Licences, Project grants
Enabling completion of research
projects
Lump sum grants
Enabling independent research
19
RESEARCH, DEVELOPMENT and INNOVATION (RDI)
AS A LONG TERM NETWORK EFFORT
society
The network consists of a variety of
stakeholders working on a common
industry
Question,
issue,
problem
problem; but goals and people shift
research
Everybody produces knowledge,
everybody does research:
policy
transdisciplinary collaboration
The result is to be socially robust knowledge : scientifically reliable, socially valuable
20
DATA AND INDICATORS:
FOCUS ON INTERACTION AND INTERMEDIATE OUTCOME
representing productive interactions
between a variety of stakeholders
society
industry
Question,
issue,
problem
policy
= research, policy, industry, society
research
Data collection:
(i) personal interactions
(i) interaction thru media
(iii) financial or material support
(iv) intermediate output
(v) indications of social impact
(vi) new products, procedures, etc