• These slides have been produced by the REF team, and
were last updated on 30 January 2012
• They provide a summary of the assessment framework
and guidance on submissions and the REF panel criteria
• They may be used freely for the purposes of briefing any
interested parties on the REF
• Should anyone other than the REF team edit these
slides, then any slides that you add or amend with
your own interpretation of the REF guidelines, should
not be presented with the REF logo and design. It
should be clear to the audience that such slides have
not been produced by the REF team.
The Research Excellence
Framework
Assessment framework,
guidance on submissions
and panel criteria
Presentation outline
•
Overview
•
REF panels
•
Staff
•
Outputs
•
Impact
•
Environment
Overview
Overview:
Purpose of the REF
The REF replaces the RAE as the UK-wide framework for
assessing research in all disciplines. Its purpose is:
•
•
•
To inform research funding allocations by the four UK
HE funding bodies (approximately £2 billion per year)
Provide accountability for public funding of research
and demonstrate its benefits
To provide benchmarks and reputational yardsticks
Overview:
Principles of the REF
The REF is a process of expert review. The assessment
is founded on the professional judgement of disciplinebased expert panels.
The conduct of the REF is guided by the principle of:
•
•
•
Equity: All types of research and forms of output in all
disciplines shall be assessed on an equal basis
Equality: HEIs are encouraged to submit the work of
all their excellent researchers
Transparency: The assessment criteria, procedures
and outcomes to be published in full
Overview:
Key changes since 2008 RAE
•
•
•
•
•
•
•
•
•
Inclusion of assessment of impact
Fewer UOAs/panels, operating more consistently
Strengthened equality and diversity measures
Revised eligibility criteria for staff
Addition of (limited) use of citation data in some UOAs
Removal of ‘esteem’ as a distinct element
Revised approach to ‘environment’ and data collection
Increased ‘user’ input on panels; and an integrated
role for additional assessors
Publication of overall quality profiles in 1% steps
Overview:
Guidance and criteria
Comprehensive information and guidance is set out in:
•
Assessment framework and guidance on
submissions (July 2011):
-
•
Panel criteria and working methods (Jan 2012):
-
•
Sets out the information required in submissions and
the definitions used
Sets out how panels will assess submissions
Further supplementary guidance will be published
during 2011-12 on technical and procedural matters
Overview:
The assessment framework
Overall quality
Outputs
Impact
Environment
Maximum of 4 outputs
per researcher
Impact template and
case studies
Environment data and
template
65%
20%
15%
Overview:
Submissions
•
Each submission in a UOA provides evidence about the
activity and achievements of a ‘submitted unit’
including:
-
•
Staff details (REF1a/b/c)
Research outputs (REF2)
Impact template and case studies (REF3a/b)
Environment data (REF4a/b/c)
Environment template (REF5)
A submitted unit may, but need not, comprise staff who
work within a single ‘department’ or organisational unit
Overview:
Multiple and joint submissions
•
•
•
Institutions will normally make one submission in each
UOA they elect to submit in
Joint submissions are encouraged where this is an
appropriate way of describing collaborative research
Multiple submissions only by exception and with
permission from the REF manager:
-
Where an HEI also makes a joint submission in that UOA
-
Where a sub-panel considers there is a case given the
nature of the disciplines covered. These sub-panels are
listed in the panel criteria statements.
Where HEIs have merged
In Sub-panel 28 where one submission is in Celtic
studies
Overview:
Timetable
2011
2012
2013
2014
• Panels appointed
(Feb)
• Panel criteria (Jan)
• Launch REF
submissions system
(Jan)
• Panels assess
submissions
• Guidance on
submissions (Jul)
• HEIs submit codes
of practice (final
deadline Jul)
• Draft panel criteria
for consultation (Jul)
• Pilot of submissions
system (Sep)
• Close of
consultation (5 Oct)
• Requests for
multiple submissions
(final deadline Dec)
• Survey of
submission
intentions
(Dec)
• Recruit additional
assessors
• Staff census date
(31 Oct)
• Submissions
deadline (29 Nov)
• Publish outcomes
(Dec)
Overview:
Publication of results
•
The primary outcome of the REF is an ‘overall quality
profile’ to be awarded to each submission:
-
•
Using the same scale as RAE2008, but in steps of 1%
To be published in Dec 2014
Further reports and feedback during 2015:
-
An overview report by each main panel, including
observations by each of their sub-panels
-
Concise feedback on submissions, to the heads of HEIs
-
A report by the Equality and Diversity Advisory Panel
The output, impact and environment sub-profile for each
submission
Submissions will be published (except for confidential or
sensitive information)
Overview:
Example of a quality profile
The overall quality profile
is comprised of the
aggregate of the weighted
sub-profiles produced for
outputs, impact and
environment.
Overall
Quality Profile
Quality Level
4*
% of Research
Activity
12 37 41 10 0
3*
12.8 32.8
2*
1*
U
Impact
Outputs
4*
3*
Environment
2*
1*
U
4*
3*
2*
1*
U
4*
3*
2*
1*
U
43
11.4
0
20
45
35
0
0
0
40
40
20
0
65%
20%
15%
REF panels
REF panels:
Main and sub-panel roles
There are 36 sub-panels working under the guidance of 4
main panels. Membership is published at www.ref.ac.uk
Sub-panel responsibilities
Main panel responsibilities
• Contributing to the main
panel criteria and working
methods
• Developing the panel
criteria and working
methods
• Assessing submissions
and recommending the
outcomes
• Ensuring adherence to the
criteria/procedures and
consistent application of the
overall assessment
standards
• Signing off the outcomes
REF panels:
Main Panel A
1
Clinical Medicine
2
Public Health, Health Services and Primary Care
3
Allied Health Professions, Dentistry, Nursing and
Pharmacy
4
Psychology, Psychiatry and Neuroscience
5
Biological Sciences
6
Agriculture, Veterinary and Food Science
REF panels:
Main Panel B
7
Earth Systems and Environmental Sciences
8
Chemistry
9
Physics
10
Mathematical Sciences
11
Computer Sciences and Informatics
12
Aeronautical, Mechanical, Chemical and
Manufacturing Engineering
13
Electrical and Electronic Engineering, Metallurgy
and Materials
14
Civil and Construction Engineering
15
General Engineering
REF panels:
Main Panel C
16
Architecture, Built Environment and Planning
17
Geography, Environment Studies and Archaeology
18
Economics and Econometrics
19
Business and Management Studies
20
Law
21
Politics and International Studies
22
Social Work and Social Policy
23
Sociology
24
Anthropology and Development Studies
25
Education
26
Sports-related Studies
REF panels:
Main Panel D
27
Area Studies
28
Modern Languages
29
English Literature and Language
30
History
31
Classics
32
Philosophy
33
Theology and Religious Studies
34
Art and Design: History, Practice and Theory
35
Music, Drama, Dance and Performing Arts
36
Communications, Cultural and Media Studies,
Library and Information Management
REF panels:
Additional assessors
Additional assessors will be appointed to extend the
breadth and depth of panels’ expertise:
•
•
Both ‘academic’ assessors (to assess outputs) and
‘user’ assessors (to assess impacts) will be appointed
Assessors will play a full and equal role to panel
members, in developing either the outputs or impact
sub-profiles. They will be fully briefed, take part in
calibration exercises and attend meetings:
-
Some appointments in 2012, where the sub-panel has
already identified a clear gap in expertise required to
assess outputs or impact
-
Further appointments in 2013, in the light of the survey of
institutions’ submission intentions
REF panels:
UOA descriptors and boundaries
•
•
•
•
UOA boundaries are not rigidly defined and all panels
expect submissions to cross boundaries with other UOAs
Sub-panel membership is broad and includes members with
interdisciplinary expertise
Assessors will be appointed to extend their breadth and
depth of expertise. Where there is a significant overlap
between UOAs, ‘joint’ assessors may be appointed to work
with more than one sub-panel.
The sub-panels prefer to assess submitted work within the
sub-panel but may, exceptionally, cross-refer parts of
submissions to other sub-panels for advice. The original
sub-panel remains responsible for recommending the
outcome
REF panels:
Main panel working methods
•
•
Each main panel will work with its sub-panels to apply
common assessment criteria
Panels will undertake calibration exercises and reviews
of emergent outcomes to make sure overall
assessment standards are applied consistently
REF panels:
Sub-panel working methods
•
•
•
•
•
The sub-panel chair will allocate work to
members/assessors with appropriate expertise
All outputs will be reviewed in sufficient detail to
contribute to the outputs sub-profile
Each case study will normally be assessed by at least
one academic member and one user
member/assessor
Sub-panel members may raise audit queries with the
REF team for investigation
The panel secretariat will minute panel procedures
Staff
Staff:
Category A and C staff
HEIs select which staff to include in submissions:
•
•
Category A staff: Academic staff with a contract of at
least 0.2 FTE, on the payroll of the HEI on 31 Oct
2013, with a primary employment function of either
‘research only’ or ‘teaching and research’
Category C staff: Staff employed by an organisation
other than an HEI with a contract or job role including
research, and whose research is primarily focused in
the submitting unit on 31 Oct 2013
(Category C staff will contribute to the quality profile
but not the volume measure for funding purposes)
•
Research assistants are eligible only by exception
Staff:
Codes of practice on staff selection
•
•
The funding bodies encourage HEIs to submit the work
of all their excellent researchers, and HEIs have legal
obligations affecting their staff selection procedures
Each HEI is required to develop, document and apply a
code of practice on the fair and transparent selection of
staff for the REF:
-
Must be signed off by the head of the institution and
submitted to the REF team by 31 July 2012 at the latest
-
The REF Equality and Diversity Advisory Panel will
examine these for adherence to the guidance
-
They will be published at the end of the assessment
process
Staff:
Codes of practice on staff selection
•
•
Codes should demonstrate fairness to staff by
addressing the principles of:
-
Transparency: clearly setting out the procedures for staff
selection, and communicating these to all eligible staff
-
Consistency: applying consistent procedures across the
institution
-
Accountability: clearly defining responsibilities for
decisions, with appropriate training for those involved
-
Inclusivity: promoting an inclusive environment, with
robust procedures for staff to disclose individual
circumstances
An Equality Impact Assessment should inform the code
and be kept under review at key stages of the selection
process
Staff:
Individual staff circumstances
•
•
•
Up to four outputs must be listed against each
member of staff
This can be reduced without penalty where
circumstances have constrained an individual’s ability
to work productively or produce four outputs during
the assessment period:
-
A wide range of circumstances will be taken into
account
-
With as much clarity as possible about how many
outputs may be reduced without penalty
-
To be treated consistently across the exercise
The allowances for maternity, paternity and adoption
leave have been revised following consultation
Staff:
Individual staff circumstances
Clearly defined circumstances
Complex circumstances
• Early Career Researchers
• Disability
• Part-time working, career breaks
and secondments outside of HE
• Ill health or injury
• Periods of maternity, adoption
and additional paternity leave
• Additional constraints related to
bringing a child into the family
• Mental health conditions
• Other caring responsibilities
• Gender reassignment
• Other circumstances related to
the protected characteristics or
employment legislation
Staff:
Clearly defined circumstances
•
•
•
•
‘Tariffs’ define the number of outputs that may be
reduced without penalty
These will be applied consistently by all REF subpanels
Circumstances may be combined up to a maximum
reduction of three outputs
Where an individual has a combination of clearly
defined and complex circumstances, these should be
submitted collectively as ‘complex’
Staff:
Clearly defined circumstances:
Early career researchers
Staff eligible for selection who started their careers as
independent researchers on or after 1 August 2009
Date at which the individual first met the
REF definition of an early career
researcher:
Number of
outputs may be
reduced by up to:
On or before 31 Jul 2009
0
Between 1 Aug 2009 and 31 Jul 2010
inclusive
1
Between 1 Aug 2010 and 31 Jul 2011
inclusive
2
On or after 1 Aug 2011
3
Staff:
Clearly defined circumstances:
Part-time working, secondments and career breaks
Total months absent between
1 Jan 2008 and 31 Oct 2013:
Number of outputs
may be reduced by
up to:
0-11.99
0
12-27.99
1
28-45.99
2
46 or more
3
For part-time working, the equivalent ‘months absent’
should be calculated
Staff:
Clearly defined circumstances:
Maternity, paternity and adoption leave
•
•
•
Changes were made to these arrangements, following
consultation
Individuals may reduce the number of outputs by one,
for each discrete period of:
-
Statutory maternity leave or statutory adoption leave
taken substantially during the period 1 January 2008 to
31 October 2013, regardless of the length of the leave.
-
Additional paternity or adoption leave lasting for four
months or more, taken substantially during the period 1
January 2008 to 31 October 2013.
Constraints relating to pregnancy, maternity, paternity,
adoption or childcare that fall outside of – or justify the
reduction of further outputs in addition to – these
allowances may be submitted using the arrangements
Staff:
Clearly defined circumstances:
Other circumstances in UOAs 01-06
•
Outputs may be reduced by up to two for the
following:
-
Category A staff who are junior clinical academics (and
have not gained a CCT or equivalent prior to 31
October 2013.
-
Category C staff who are employed primarily as clinical,
health or veterinary professionals
Staff:
Complex circumstances
•
•
•
•
•
EDAP will consider all cases of complex
circumstances
EDAP will make recommendations to the Main Panel
Chairs, who will decide
Sub-panels will be informed of the decisions and will
assess the remaining outputs without penalty
Information will be kept confidential to EDAP, Main
Panel Chairs and the REF team
ECU will publish a range of worked examples –
including EDAP’s rationale
Outputs
Outputs:
Eligibility
•
Outputs must be:
-
a product of research (as defined for the REF)
-
Authored/co-authored by the member of staff against
whom it is listed (regardless of where they were
employed prior to the census date)
first brought into the public domain between 1 January
2008 and 31 December 2013
Outputs:
Outputs ‘pre-published’ before 2008
•
A change in the guidance on pre-publication of
outputs has been made, following consultation:
An output first published in its final form during the
REF publication period, but ‘pre-published’ in 2007
– is eligible for submission to the REF, provided
that the ‘pre-published’ output was not submitted
to the 2008 RAE. Examples may include:
-
An online first article or preprint
A preliminary version or working paper
Outputs:
Range of output types
•
•
Outputs may include but are not limited to: printed or
electronic publications, materials, devices, images,
artefacts, products, buildings, confidential or technical
reports, patents, performances, exhibits or events
All forms of outputs shall be assessed on a fair and
equal basis
Outputs:
Co-authorship
•
•
Co-authored outputs can be listed against more than
one co-author when returned in different submissions
An output can be listed no more than twice within the
same submission and must be accompanied by a
statement explaining the substantial and distinctive
contribution of each of the submitting authors
Outputs:
Double-weighting
•
•
•
•
Institutions may request ‘double-weighting’ for outputs of
extended scale and scope
Requests for double weighting should be accompanied
by a supporting statement
Sub-panels will assess the claim for double-weighting
separately from assessing the quality of the output
If a sub-panel accepts a request for double weighting,
the output will count as two outputs in both a
submission and in the calculation of the outputs subprofile
Outputs:
Additional information on outputs
•
Panels may make use of additional information in the
form of citation data and details provided by the HEI
-
•
Sub-panels have requested further information on
the research process or content where this is not
evident from the output
Sub-panel will not make use of journal impact factors,
rankings or lists, or the perceived standing of the
publisher, in assessing the quality of research outputs
Outputs:
Citation data
•
The following sub-panels will make use of citation data:
-
•
•
•
Main Panel A: Sub-panels 1-6
Main Panel B: Sub-panels 7-11
Main Panel C: Sub-panel 18
These sub-panels will use only the citation data provided
by the REF team, sourced from Scopus (except for Subpanel 11 that will, in addition, use Google Scholar)
All other sub-panels will neither receive nor make use of
citation data
None of the sub-panels will use journal impact factors,
journal rankings or other forms of bibliometric analysis
Outputs:
Citation data
•
•
Where used, sub-panels will consider citation data as
follows:
-
As one indicator of ‘academic significance’. Expert review
remains the primary means of assessing outputs
-
Panels will assess all outputs on an equal basis
regardless of the availability of such data. They recognise
the limitations of citation data – especially for recently
published outputs – and will have due regard to potential
equality implications
-
In the relevant UOAs only, citation counts from Scopus will
be provided to panels on a consistent basis. HEIs will be
able to verify the outputs have been matched correctly,
and view citation counts on the submission system
The funding bodies do not sanction or recommend that
HEIs rely on citation data to inform the selection of staff
or outputs for their REF submissions
Outputs:
Assessment criteria
The criteria for assessing the quality of outputs are
originality, significance and rigour*
Four star
Quality that is world-leading in terms of originality, significance
and rigour
Three star
Quality that is internationally excellent in terms of originality,
significance and rigour but which falls short of the highest
standards of excellence
Two star
Quality that is recognised internationally in terms of originality,
significance and rigour
One star
Quality that is recognised nationally in terms of originality,
significance and rigour
Quality that falls below the standard of nationally recognised
Unclassified work. Or work which does not meet the published definition of
research for the purposes of this assessment
* Each main panel provides a descriptive account of the criteria
Impact
Impact:
Definition of impact for the REF
•
•
•
An effect on, change or benefit to the economy,
society, culture, public policy or services, health, the
environment or quality of life, beyond academia
Impact includes an effect, change or benefit to:
-
The activity, attitude, awareness, behaviour, capacity,
opportunity, performance, policy, practice, process or
understanding
-
Of an audience, beneficiary, community, constituency,
organisation or individuals
-
In any geographic location whether locally, regionally,
nationally or internationally
It excludes impacts on research or the advancement
of academic knowledge within HE; and impacts on
teaching or other activities within the submitting HEI
Impact:
Range of impacts
•
•
Panels recognise that Impacts can manifest in a wide
variety of different ways, may take many forms and occur in
a wide range of spheres
Examples of impact may include:
-
Impacts on public policy and services,
-
Impacts on society, culture and creativity,
-
Impacts on practitioners and services,
-
Impacts on the environment,
-
Impacts on the economy
Impact:
Examples of impacts: Main Panel A
Impacts on:
For example:
Health and welfare
Patient outcomes have improved
Society, culture and creativity
Public debate has been stimulated
The economy
Costs of healthcare have changed
Commerce
Business performance has improved
Public policy and services
Policy decisions informed by research
Production
Waste levels have reduced
Practitioners and services
Professional standards influenced by research
Environment
Changes to environmental risk management
International development
Quality of life improvements
Impact:
Range of impacts – Main Panel B
Impacts on:
For example:
The economy
Patient outcomes have improved
Public policy and services
Public debate has been stimulated
Society, culture and creativity
Public engagement in science has been stimulated
Health
Business performance has improved
Practitioners and
professional services
Policy decisions informed by research
Environment
Waste levels have reduced
Impact:
Examples of impacts: Main Panel C
Impacts on:
Examples:
Creativity, culture
and society
Enhancements to heritage preservation, conservation and
presentation
Shaping or informing public attitudes and values
Economy, commerce Improved products, processes or workplace practices
Enhanced corporate social responsibility policies
or organisations
Environment
Changes in public awareness or behaviour
Business operations have been changed to achieve environmental
objectives
Health and welfare
Development of policy or practice with regard to health services or
social care provision
Practitioners and
Influence on professional standards, guidelines or training
professional services Practitioner debate has been stimulated by research findings
Public policy, law
and services
Legislative change or effect on legal practice
Influence on policy (by government, NGO or private organisation)
Impact on democratic participation or access to justice
Impact:
Examples of impacts: Main Panel D
Impacts on:
Examples:
Civil society
Cultural life
Economic prosperity
Education
Policy making
Public discourse
Public services
Design of new products or services
Impact:
Submissions
Impact template (REF3a)
Case studies (REF3b)
• Sets out the submitted unit’s
general approach to enabling
impact from its research
• Specific examples of impacts
already achieved, that were
underpinned by the
submitted unit’s research
• One template per
submission – with a page
limit depending on the
number of staff submitted
• Covers the period 1 Jan
2008 to 31 Jul 2013
• Contributes 20% to the
impact sub-profile
• The number of case studies
required depends on the
number of staff submitted (1
plus 1 per 10 FTE)
• Impacts during 1 Jan 2008 to
31 Jul 2013; underpinned by
research since 1 Jan 1993
• Contributes 80% to the
impact sub-profile
Impact:
Case studies (REF3b)
•
•
In each case study, the impact described must:
-
Meet the REF definition of impact
-
Be underpinned by excellent research (of at least 2*
quality) produced by the submitting unit between 1
January 1993 to 31 December 2013
Have occurred between 1 Jan 2008 and 31 July 2013
(can be at any stage of maturity)
Submitted case studies need not be representative of
activity across the unit: pick the strongest examples
Impact:
•
•
•
•
Underpinning research
Case studies must be underpinned by research
produced by the submitted unit that has made a
material and distinct contribution to the impact
Case studies must provide evidence that the
underpinning research meets the quality threshold of
at least two star
HEIs may submit case studies of a confidential nature,
these may be redacted for publication or not published
at all
HEIs may request advance permission to submit case
studies containing sensitive material that could only be
assessed by individuals with national security clearance
Impact:
Types of evidence
•
Evidence of the nature and extent of the impact, in
terms of its reach and significance, should be
provided and may include:
-
Critical reviews in the media,
-
Verifiable influence on particular projects or
processes which bring benefits
Citation by international bodies,
Visitor or audience numbers and feedback
Business performance measures, sales, turnover,
Documented evidence of change to public
policy/legislation/regulations/guidelines
Impact:
Case studies (REF3b)
•
Each case study is limited to 4 pages and must:
-
Describe the underpinning research produced by the
submitting unit
-
Reference one or more key outputs and provide
evidence of the quality of the research
-
Explain how the research made a ‘material and distinct’
contribution to the impact (there are many ways in
which this may have taken place)
-
Explain and provide evidence of the nature and extent
of the impact: Who/what was affected? How were they
affected? When?
-
Provide independent sources that could be used to
verify claims about the impact (on a sample audit basis)
Impact:
Template (REF3a)
•
The unit’s approach to enabling impact from its research:
-
•
•
Context for the approach
The unit’s approach during 2008-2013
Strategy and plans for supporting impact
Relationship to the submitted case studies
Provides additional information and context for the case
studies, and can take account of particular circumstances
that may have constrained a unit’s selection of case
studies
To be assessed in terms of the extent to which the unit’s
approach is conducive to achieving impact of ‘reach and
significance’
Impact:
Assessment criteria
The criteria for assessing impacts are reach and significance*
Four star
Outstanding impacts in terms of their reach and significance
Three star
Very considerable impacts in terms of their reach and
significance
Two star
Considerable impacts in terms of their reach and significance
One star
Recognised but modest impacts in terms of their reach and
significance
The impact is of little or no reach and significance; or the impact
Unclassified was not eligible; or the impact was not underpinned by excellent
research produced by the submitted unit
* Each main panel provides a descriptive account of the criteria
Environment
Environment:
Data (REF4)
•
All submissions to include data on:
-
•
•
•
Research doctoral degrees awarded (REF4a)
Research income (REF4b)
Research income in-kind (REF4c)
Definitions are aligned with HESA returns. We will
provide HESA data to institutions to assist in preparing
submissions, and will use it for verification purposes
Sub-panels 8, 9, 19, 25 and 26 request specific
additional data, to be included within the environment
template (REF5)
Data will be considered by panels alongside the
qualitative information provided in REF5
Environment:
Environment template (REF5)
•
•
•
Each submission to include a completed template (with
page limits depending on the number of staff submitted):
-
Overview
-
Income, infrastructure, and facilities
Research strategy
People (including staffing strategy and staff development;
and research students)
Collaboration and contribution to the discipline
Panel criteria indicate appropriate forms of evidence, and
the relative importance of each section
No expectation that the environment relates to a single
department or organisational unit
Environment:
Assessment criteria
The criteria for assessing the environment are
vitality and sustainability*
Four star
An environment that is conducive to producing research of
world-leading quality, in terms of its vitality and sustainability
Three star
An environment that is conducive to producing research of
internationally excellent quality, in terms of its vitality and
sustainability
Two star
An environment that is conducive to producing research of
internationally recognised quality, in terms of its vitality and
sustainability
One star
An environment that is conducive to producing research of
nationally recognised quality, in terms of its vitality and
sustainability
Unclassified An environment that is not conducive to producing research of
nationally recognised quality
* Each main panel provides a descriptive account of the criteria
Further information
www.ref.ac.uk
(includes all relevant documents)
Enquiries from staff at HEIs should be directed to
their nominated institutional contact
(see www.ref.ac.uk for a list)
Other enquiries to info@ref.ac.uk