Demonstrating research impact
in the REF
Graeme Rosenberg
REF Manager
Outline:
•
•
•
•
What is the REF?
What is impact?
How should it be demonstrated?
How will it be assessed?
What is the REF?
•
•
•
The REF is the new framework for assessing research
in all disciplines in all UK higher education institutions
It replaces the RAE
Its purpose is:
-
To inform research funding allocations by the four UK
HE funding bodies (approximately £2 billion per year)
-
Provide accountability for public funding of research
and demonstrate its benefits
-
To provide benchmarks and reputational yardsticks
The assessment framework
Overall quality
Outputs
Impact
Environment
Maximum of 4 outputs
per researcher
Impact template and
case studies
Environment data and
template
65%
20%
15%
Submissions
•
•
•
There are 36 discipline-based units of assessment
(UOAs)
Each institution decides which UOAs to submit in
Each submission in a UOA must contain a standard set
of information:
-
Staff details (REF1a/b/c)
Research outputs (REF2)
Impact template and case studies (REF3a/b)
Environment data (REF4a/b/c)
Environment template (REF5)
Assessment
•
•
Submissions will be assessed by 36 expert sub-panels
working under the guidance of 4 main panels
Panels will carry out the assessment according to the
published criteria and working methods
Sub-panel responsibilities
Main panel responsibilities
• Contributing to the panel
criteria and working methods
• Developing the panel criteria
and working methods
• Assessing submissions and
recommending the outcomes
• Ensuring adherence to the
criteria/procedures and
consistent application of the
overall assessment
standards
• Signing off the outcomes
Example of a quality profile
The overall quality profile
is comprised of the
aggregate of the weighted
sub-profiles produced for
outputs, impact and
environment.
Overall
Quality Profile
Quality Level
4*
% of Research
Activity
12 37 41 10 0
3*
12.8 32.8
2*
1*
U
Impact
Outputs
4*
3*
Environment
2*
1*
U
4*
3*
2*
1*
U
4*
3*
2*
1*
U
43
11.4
0
20
45
35
0
0
0
40
40
20
0
65%
20%
15%
Timetable
Criteria phase
2011
Submissions phase
2012-13
Assessment phase
2014
• Develop and publish
Guidance on submissions
(Jul)
• HEIs submit Codes of
practice
• Panels assess
submissions
• Launch the REF
submissions system
• Publish outcomes
Dec 2014
• Develop, consult on and
publish Panel criteria
(Jan 2012)
• Submission deadline
29 Nov 2013
What is impact?
•
Impact is defined broadly for the REF:
an effect on, change or benefit to the economy,
society, culture, public policy or services, health, the
environment or quality of life, beyond academia
•
•
The REF definition of impact is retrospective –
benefits or changes that have already occurred, not
future potential
Impacts can be manifest in a wide variety of ways,
may take many forms and occur in a wide range of
spheres, in any geographic location
Examples of types of impact
Enhanced professional
standards, ethics, guidelines
or training
Improved health or
welfare outcomes
Improved quality,
accessibility or efficiency of a
public service
Changes to the
design or delivery of
the school curriculum
More effective
management or
workplace practices
Production costs have
reduced
Organisations have
adapted to changing
cultural values
Research has enabled
stakeholders to challenge
conventional wisdom
Jobs have been
created or protected
Levels of waste have
reduced
A new product has
been commercialised
A social enterprise
initiative has been
created
Improved forensic
methods or expert
systems
Improved access to
justice, employment
or education
Research has informed
public understanding, values,
attitudes or behaviours
The policies or activities of
NGOs or charities have been
informed by research
New forms of artistic
expression or changes to
creative practice
Public debate has
been shaped or
informed by research
Improved business
performance
Enhanced preservation,
conservation or presentation
of cultural heritage
Policy debate or decisions
have been influenced or
shaped by research
Enhanced corporate
social responsibility
policies
Improved risk
management
Improved management or
conservation of natural
resources
Changes to
legislation or
regulations
Changes in
professional practice
Enhanced technical
standards or
protocols
Examples from the pilot exercise
 Improved treatment and quality of life for patients with
type 1 diabetes
 Detector technology used to screen liquids at airports
 Conservation of bumblebees
 Influencing policy on child maintenance and better
service for separated parents
 Enriched public appreciation of heritage at Hampton
Court Palace
Pilot examples are published at:
www.ref.ac.uk/background/pilot
Demonstrating impact
Impact template
20% of the
impact subprofile
Case studies
80% of the
impact subprofile
• Sets out the submitted unit’s
approach to impact
• Specific examples of impacts
already achieved
Case studies
•
•
•
•
One case study must be submitted per 10 FTE staff
(with a minimum of two cases in a submission)
Impacts that took place during 2008 to 2013;
underpinned by research since 1993
Submitted case studies need not be representative of
activity across the unit: pick the strongest examples
Case studies must be completed on a template maximum 4 pages
The case study template
1.
2.
Summary of the impact
Underpinning research:
-
3.
4.
What were the key insights or findings?
References to the research
Details of the impact:
-
5.
What was it? Who undertook it and when?
How did the research lead/contribute to the impact?
Who were the beneficiaries?
How did they benefit?
Sources of corroboration
Lessons from the pilot exercise
•
•
•
•
•
•
All the material required to make a judgement should be
included
The narrative should be coherent and clearly explain the
‘links in the chain’
Clear definition of who benefitted, and what had changed
Indicators should be meaningful and contextualised
Key claims should be capable of verification
Where the impact arises from public engagement:
-
How was the engagement activity based on the research?
Dissemination in itself is not impact – what was the
benefit?
Preparing impact submissions
•
HEIs seem to be at varying stages of:
-
•
Briefing staff
Identifying potential cases
Drafting case studies
Some issues seem to be:
-
How to write compelling narrative, and who should be
involved
-
Evidence gathering – retrospective and from many sources
Institutional memory – what if the staff have left?
Second guessing the panels to decide on the strongest
cases
How will impacts be assessed?
•
Sub-panels will assess impacts against the criteria of
reach and significance:
-
•
•
•
A holistic judgement about each case study
‘Reach’ is not a geographic scale.
Each case study will be examined by at least one
academic and one ‘user’ on the panel
Panels will make judgements on the merits of the
evidence provided – not further investigations
A sample of case studies will be audited to corroborate or
verify information
Underpinning research
•
•
Panels will assess the impact, not the underpinning
research
But each case study must be underpinned by research
that satisfies the following criteria:
-
produced by staff while working in the submitting HEI
-
meets the quality threshold of at least equivalent to 2*
generated outputs published between 1 Jan 1993 and 31
Dec 2013
made a material and distinct contribution to the impact
Assessment criteria
The criteria for assessing impacts are reach and significance*
Four star
Outstanding impacts in terms of their reach and significance
Three star
Very considerable impacts in terms of their reach and
significance
Two star
Considerable impacts in terms of their reach and significance
One star
Recognised but modest impacts in terms of their reach and
significance
The impact is of little or no reach and significance; or the impact
Unclassified was not eligible; or the impact was not underpinned by excellent
research produced by the submitted unit
* Each main panel provides further description of the criteria
Further information
www.ref.ac.uk
(includes all relevant documents)
Enquiries from staff at HEIs should be directed to
their nominated institutional contact
(see www.ref.ac.uk for a list)
Other enquiries to info@ref.ac.uk