REF 2021
What we know and thought we knew, in preparation for
the next Research Excellence Framework
Dr. Tim Brooks, Research Policy & REF Manager, RDCS
VFI, 19 January 2016
About the REF
• A periodic national
assessment of research
activity which:
• Provides benchmarking
information
• Ensures public accountability
for investment in research and
its benefit
• Enables the selective
allocation of recurrent
research grant (QR)
What was the REF?
• A process of expert review
which assesses research
activity – outputs, impact,
environment – over a period of
c. 5 years
• Organised by discipline, into
36 ‘sub panels’ covering a
‘Unit of Assessment’ and four
‘Main Panels’ to ensure
consistency
• Quality profiles rating research
Submissions
• Institutions decided if to
participate, and which UoA(s)
to submit to.
• Academic staff in post on the
census date the key building
block of each submission
• Employment details
• Any relevant personal
circumstances
Research outputs
• Outputs of research published
during the REF assessment
period
• Selected for submission on the
basis of quality
• Up to four outputs for every
member of staff, reduced
where personal circumstances
justify it
• Worth 65% of the final mark
Impact
• Impacts realised during the
assessment period,
underpinned by research done
by staff in the submitting
institution
• Worth 20% of the final mark
Environment
• Data about doctoral awards
and research grant income
• Narrative statement describing
the research environment,
strategies and plans,
structures and support.
• Worth 15% of the final score
Confidentiality
• Panel members and other staff
bound by confidentiality
agreements
• Arrangements in place to allow
institutions to partly or fully
redact any element of any
submission including case
studies
• Arrangements to have case
studies assessed by
individuals with national
security clearance
The next REF?
• Three weeks in November
2015:
• Green Paper
• Comprehensive Spending
Review
• Nurse Review
• And then the Stern Review …
The next REF?
• Rules and regulations not
likely to appear before spring
2017, and more likely
summer 2018.
• In most cases little or no
change e.g. UoA organization,
staff eligibility, output
requirements, environment
markers.
• But – possibility of submitting
all eligible staff, and this might
change links to outputs.
The next REF?
Open Access
• All journal articles and conference
contributions accepted for
publication on or after 1 April
2016 must be made available via
an open access repository (e.g.
ARRO) between acceptance and
3 months post publication.
• From 1 April 2017, policy tightens
to within 3 months of acceptance.
• Additional credit for exceeding,
and supporting the exceeding, of
the basic requirements.
The next REF?
Metrics
• Recent independent review of
metrics usage in research
assessment
• Confirms primacy of peer
review BUT encourages
judicious increase in metric
usage where appropriate
• Green Paper proposes an
interim metrics-based
assessment?
The next REF?
Timetable
• Spring 2017/Summer 2018: first
guidance published
• 31 July 2020: end of environment
& impact assessment periods
(impact of research from 1
January 2000)
• 31 October 2020: staff census
date
• 27 November 2020: submission
deadline
• 31 December 2020: last date for
The next REF?
Institutional strategy
• Go forward with the 15 UoAs
submitted to REF2014
• Add additional UoAs where
appropriate
• Submit more staff (CP - 55%
eligible staff auditable by 2017)
• Take steps to support quality
• First ‘stocktake’ during 2016
including impact cases
• Events and other activities to
drive preparations forward
Impact for
REF 2014
• Worth 20% of the final score
• An impact template (20%)
describing how the submitting
unit had enabled impact from
research during the assessment
period
• Case studies (80%) giving
examples of specific impacts
realised during the assessment
period, underpinned by research,
at least 2* in quality, produced by
the submitting unit in the previous
21 years
• Number of case studies
determined by FTE of staff in the
submission.
The next REF?
Impact
• Seen as a resounding success
in REF 2014
• Worth 25% next time? Or
40%?
• Abandon ‘impact template’
document or add into research
environment requirements?
• Increase number of case
studies required per FTE?
• ‘Reuse’ of previously
submitted case studies?
Defining Impact
• “an effect on, change or
benefit to the economy,
society, culture, public policy
or services, health, the
environment or quality of life,
beyond academia”
• Impacts within HE excluded
unless they extend
significantly beyond the
submitting HEI.
Case Studies (1)
• Summary: a brief introduction
to the specific impact(s) being
described
• Underpinning research:
Details of the key research
insights or findings, and
details of what research was
undertaken, when and by
whom
• References to the research,
demonstrating ≥ 2* quality
Case Studies (2)
• Details of the impact: and
explanation of how the
research made a distinct,
material contribution to the
impact; and the nature and
extent of that impact.
• Sources to corroborate the
impact: details of reports,
reviews, individual users or
other beneficiaries supporting
the claims made
Assessment
• Threshold test: is the quality of
underpinning research at least
2*?
• Impact case studies scored on
their significance and reach
• Context of the impact all
important.
RAND findings
• Links between underpinning
research and its contribution to the
impact must be very clear
• It shows if the author doesn’t believe
in the value of what they’re saying
• Assessors had to take a lot of the
content on face value; audit
requests more likely if they were
unconvinced by language or had
prior knowledge of the impact
claimed
• The case study says everything it
must – assessors will not always
follow links
Challenges
• Understanding what
constitutes eligible impact
• Identifying good examples
of impact arising from our
research
• Locating sufficiently robust
evidence retrospectively
• … but challenges that can be
overcome
Enabling Impact
• Who might benefit from the
research?
• How might they benefit from
the research?
• What can be done to
communicate the outcomes of
the research and engage with
potential beneficiaries?
• What resources will be needed
to support this?
Next steps
• Do excellent research, and
publish it
• Embed impact in your activities
• Identify synergies, and
collaborate, internally and
externally, inside and across
disciplines
• Develop and follow strategies
to support impact and develop
the research environment
• Supervise doctoral students to
completion, and win grants
More information
• REF 2014 rules & regulations:
www.ref.ac.uk/pubs/
• REF 2014 results &
submissions: results.ref.ac.uk
• REF impact case study
database: impact.ref.ac.uk
• REF evaluation reports:
http://www.hefce.ac.uk/rsrch/REFreview/
• RCUK Pathways to Impact
guide:
www.rcuk.ac.uk/innovation/impacts
/
Thanks for your attention
Any questions?
Dr. Tim Brooks, Research Policy & REF Manager, RDCS
tim.brooks@anglia.ac.uk