Tracking the external impacts
of academic research
in long-term ways
and for HEFCE purposes
Patrick Dunleavy
London School of Economics and Political Science
See our ‘Impact of Social Sciences’ blog at:
http://blogs.lse.ac.uk/impactofsocialsciences/
Structure of this talk
1. Defining external research impacts
and exploring how they operate
2. Capturing useful evidence of external
impacts (starting now for the longer
term)
3. HEFCE’s maximalist view of external
“impacts”, and its case study method
Defining research impacts
in an evidence-based way
A research impact is: “a recorded or otherwise
auditable occasion of influence from university
research upon another actor or organization”.
a. Academic impacts from research are influences upon
other actors in academia or universities, e.g. as
measured by citations.
b. External impacts are influences on actors outside
higher education, that is, in business, government or
civil society, e.g. as measured by references in the
trade press or in government documents, or by
coverage in mass media.
Academic work and external impacts: a simple view
Primary impacts
– observable
occasions of influence
A
c
a
d
e
m
i
c
w
o
r
k
Academic
impacts
e.g.
citations
External
impacts
e.g. influence
counts
Academic/
university
outputs
Economic/
public policy/
civil society
outputs
Academic work and external impacts: allowing for
time lags and cumulation/ delay effects
Primary impacts
– observable
occasions of influence
A
c
a
d
e
m
i
c
w
o
r
k
Academic
impacts
e.g.
citations
Academic/
university
outputs
Dynamic
knowledge
inventory
External
impacts
e.g. influence
counts
Economic/
public policy/
civil society
outputs
A naïve (science-based) view of how an academic
discipline achieves external impacts
Single discipline processes
Wider
society
Media,
cultural
and civil
society
systems
Economic
and
business
systems
Discovery
Application
Public
policy
systems
How key forms of scholarship within each
academic discipline begin to achieve external
impacts
Impacts
Wider
Single discipline processes
Renewal
Integration
interface
society
Media,
cultural
and civil
society
systems
Economic
and
business
systems
Discovery
Application
Public
policy
systems
How cross-disciplinary influences add to and
mediate external impacts
Single discipline processes
Joined-up
scholarship
Bridging
Renewal
Integration
University
- local
integration
Impacts
interface
Wider
society
Media,
cultural
and civil
society
systems
Economic
and
business
systems
Discovery
Application
Academic
Service
Public
policy
systems
How the impacts interface shapes external impacts
Single discipline processes
Joined-up
scholarship
Impacts
interface
Media
Bridging
Renewal
Integration
University
- local
integration
Specialist
Media
Professions
Corporations
Entrepreneurs
Discovery
Wider
society
Media,
cultural
and civil
society
systems
Economic
and
business
systems
Consultants
Think tanks
Application
Academic
Service
Policy
communities
NGOs
Public
policy
systems
2. Capturing useful evidence
of external impacts
(starting now for the longer term)
Six metrics of academic/university
involvements with external impact
• External funding linkages, especially businessto-sciences/ technology departments
• Media and specialist media presence
• Government website presence
• Academic service in government
• Reported ‘knowledge transfer’ activities
• Academics’ perceptions of impacts
Universities’ key interactions with business in the UK
2007-08
2008-09
Change
% Change
697,030
731,734
34,704
5
27,051
28,111
1,060
4
834,627
937,373
102,746
11
64,292
64,025
-267
0
334,768
331,541
-3,227
-1
of which, number with SMEs
22,802
20,596
-2,206
-11
number with large business
10,499
10,360
-139
-1
Number of new patent applications
1,898
2,097
199
9
Number of patents granted in year
590
653
63
10
Total revenues (£000s)
66,271
124,368
58,097
47
Total costs (£000s)
21,003
27,794
6,791
24
2,223
2,289
66
3
89,497
154,451
64,954
42
Collaborative research (£000s)
Contract research
Total number of contracts
Total value of contracts (£000s)
Consultancy
Total number of contracts
Total income (£000s)
Patents
Intellectual property income
Spin-off companies
Number created
Estimated external investment received (£000s)
Proportion of research found in UK national press
from different disciplines (2008)
Health and Medicine
Science and Technology
Politics
Economics
Business and Finance
Psychology
Law
History
Sociology
Philosophy
English
Communication and Media
Architecture
Education
Arts
Humanities*
Computer science
Anthropology
Languages
Geography
0
10
20
30
Number of items
40
50
60
Proportion of research found on UK government websites
from different disciplines
Social policy and anthropology
HSS
Medical sciences
Medicine
Law and legal
Mixed
Health policy and public health
Criminology and criminal policy
Employment and work
Psychology and psychiatry
Development
Education
Public policy and policy studies
Transport policy
Politics and democracy
Management and media studies
Science and engineering
Technology
Economics
Geography
Design
Creative and arts
Archeology
0
5
10
15
20
Percentage of all research projects found
25
Academic Service: University staff involved in UK
central government quasi-government agencies
University
Wider
funding
government
bodies
bodies
Top academics
(managing universities)
Heads of department
Professors
Lecturers
Total
Number of academics
Source: Griffiths, 2010
28
15
58
0
101%
80
21
12
57
11
100%
131
All
bodies
24
13
57
7
100%
232
Academics’ reporting of ‘knowledge
transfer’ activities - top 11 activities
Knowledge transfer practice
% academics
involved 2009
% academics
involved 2008
type of
academic
activity
Attending conferences
87
56
general
Informal advice to
business
57
35
application
External lectures
65
34
application
Networks
67
32
integration
Joint publication
46
26
application
Advisory boards
38
22
service
Student
projects/placements
33
20
renewal
19
application
External visits
Formed/run consultancy
14
18
application
Contract research
37
18
application
Undertaken consultancy
43
17
application
Academics’ reporting of ‘knowledge
transfer’ activities – next 10 activities
Knowledge transfer
practice
% academics
involved 2009
% academics
involved 2008
type of
academic
activity
Been involved in
consortia
35
17
application
Joint research
49
17
application
Post-course placements
na
14
renewal
Prototyping and testing
10
na
application
Patenting
7
12
application
Licensed research
5
10
application
Standards forum
31
10
application
Spin out companies
4
7
discovery
application
Enterprise education
6
4
renewal
application /
External secondment
10
3
application
Social science and humanities academics’ perceptions
of external impacts (2008)
Area of external impacts
Public policy
Civil society
Public debates and
culture
and
business
Science and
technology
Level of
impact
Actual
Potential
Actual
Potential
Actual
Potential
Actual
Potential
Actual
Potential
Social
scientists
only
4.6
6.0
4.5
5.6
3.9
5.4
3.6
4.7
3.1
4.5
Academics
spanning
social
sciences and
humanities
All respondents
(including
academics across
all social science
and humanities)
3.6
5.5
4.1
5.3
5.0
5.9
3.1
4.0
3.4
4.1
3.4
5.1
4.1
5.1
4.6
5.6
3.0
3.8
2.9
3.9
Eight longer-term steps towards universities
boosting their external impacts
• Clean up ‘multiple branding’ problems
• Reappraise events programmes
• Develop an impacts file for all individual academics,
and then for departments
• Pull together databases and services to give improved
‘customer relationship management’ of research clients
– e.g. bring in CRM systems
• Start (and fill up) an online depository
• For all research on the closed-web, publish a useful
open-web version (see session D)
• Improve professional communication – starting with
multi-author blogs (see session D)
• Work better in networks (see Panel 2)
3. HEFCE’s expansive view of external
impacts, and case study method
PRIMARY EXTERNAL IMPACTS OF RESEARCH
Primary impacts
– observable
occasions of influence
A
c
a
d
e
m
i
c
w
o
r
k
Academic
impacts
e.g.
citations
Academic/
university
outputs
Dynamic
knowledge
inventory
External
impacts
e.g. influence
counts
Economic/
public policy/
civil society
outputs
WHAT MORE DOES HEFCE NEED? – 1. EVIDENCE OF CHANGED OUTPUTS
Primary impacts
– observable
occasions of influence
A
c
a
d
e
m
i
c
w
o
r
k
Changes in
organizational
activities
or outputs
DEMONSTRATED
CETERIS PARIBUS,
CONTROLLING FOR
ALL OTHER
INFLUENCES
Academic
impacts
e.g.
citations
Dynamic
knowledge
inventory
External
impacts
e.g. influence
counts
Multiple other
factors
Economic/
public policy/
civil society
outputs
WHAT MORE DOES HEFCE NEED? – 2. EVIDENCE OF CHANGED OUTCOMES
Primary impacts
– observable
occasions of influence
A
c
a
d
e
m
i
c
w
o
r
k
Changes in
organizational
activities
or outputs
Changes in
societal
outcomes
Academic
impacts
e.g.
citations
Dynamic
knowledge
inventory
External
impacts
e.g. influence
counts
Multiple other
factors
Economic/
public policy/
civil society
outputs
Economic/
public
policy/
civil
society
outputs
AGAIN
CETERIS
PARIBUS,
CONTROLLING
FOR
ALL OTHER
INFLUENCES
WHAT MORE DOES HEFCE NEED? – 3. PROOF OF POSITIVE SOCIAL BENEFITS
Primary impacts
– observable
occasions of influence
A
c
a
d
e
m
i
c
w
o
r
k
Changes in
organizational
activities
or outputs
Changes in
societal
outcomes
Evaluations of
outcome
changes
– against societal
values
Academic
impacts
e.g.
citations
Dynamic
knowledge
inventory
External
impacts
e.g. influence
counts
Multiple other
factors
Economic/
public policy/
civil society
outputs
Economic/
public
policy/
civil
society
outputs
+?-
HEFCE’S SCORING BANDS FOR
IMPACT CASE STUDIES’ ‘SIGNIFICANCE’/ VALUE
AND ‘REACH’/RELEVANCE
Four star
Three star
Two star
One star
Unclassified
Exceptional: Ground-breaking or transformative impacts of major
value or significance with wide-ranging relevance have been
demonstrated
Excellent: Highly significant or innovative (but not quite groundbreaking) impacts relevant to several situations have been
demonstrated
Very good: Substantial impacts of more than incremental significance
or incremental improvements that are wide-ranging have been
demonstrated
Good: Impacts in the form of incremental improvements or process
innovation of modest range have been demonstrated
The impacts are of little or no significance or reach;
or the underpinning research was not of high quality;
or research-based activity within the submitted unit did not make a
significant contribution to the impact.
medium
low
Significance (or value)
high
CHARTING ALL HEFCE’S CRITERIA
low
medium
Reach (or relevance)
high
CHARTING ALL HEFCE’S CRITERIA
3*
medium
excellent
low
Significance (or value)
high
4*
exceptional
low
medium
Reach (or relevance)
high
CHARTING ALL HEFCE’S CRITERIA
3*
medium
excellent
2*
very good
1*
low
Significance (or value)
high
4*
exceptional
good
low
medium
Reach (or relevance)
high
CHARTING ALL HEFCE’S CRITERIA
medium
excellent
▪ Quality of
Underlying
Research
2*
very good
U
unclassed
▪ Distinctiveness
1*
of Research
good
low
medium
Reach (or relevance)
▪ Impact
occurring in the
last 15 years
3*
low
Significance (or value)
high
4*
exceptional
Additional criteria:
high
Short-run steps to prepare
for HEFCE case studies
• Develop a large long list of possible case studies
• Collate grants, publications, contacts and
testimonials evidence from (senior) academics on all
potential cases, especially for 2005-11
• For ‘probable’ cases
- firm up outside evidence of outputs, activity
and outcome changes;
- push back pre-history to 1997 if needed
• Enlarge single-person cases wherever feasible
• Strengthen ‘sub-group’ case studies, by adding
members (if feasible, without diluting ‘quality’), and
re-branding for greater coherence by 2013
• Monitor potential cases closely from now to 2013,
and discuss with uninvolved senior case-writers
Thank you for listening
See our ‘Impact of Social Sciences’ blog at:
http://blogs.lse.ac.uk/impactofsocialsciences/
From there, download the free Handbook:
Maximizing the Impacts of Social Science
Research