RESEARCH EXCELLENCE FRAMEWORK :
RESEARCH IMPACT ASESSMENT
LESSONS FROM THE PILOT EXERCISE
Professor John Marshall
Director
Academic Research Development
page 1
Thought for the day:
“We live in a moment of history
where change is so speeded up that
we begin to see the present only
when it is already disappearing”
R D Laing,
Scottish Psychiatrist
(1927-1989)
page 2
“Research Impact is as big a change
in REF as the introduction of the
original RAE was for HEIs”.
page 3
REF2014 – What we know
36 REF sub Panels (UoAs) down from 68 in RAE2008
REF2014 guidance document to be published in July 2011
Sub Panel Guidance document to be published in January
2012
Panel Chairs and Sub Panel members have been announced
The method of assessment has been changed to include the
a new measure relating to the impact of excellent research
(2* or above)
page 4
Research Excellence Framework (REF2014)
Assessment Criteria and Weightings
Outputs (65%)
Environment (20%)
Impact (20%)
REF emphasis is on Research Excellence. Only highest
quality levels (3* and 4*) will attract funding post 2014.
Contribution of research group activity over 15 year
timescale important for impact
Impact case studies : 1 per UoA plus 1 per 10 FTE
page 5
REF Research Impact Decisions
In the REF there will be an explicit element to assess
the 'impact' arising from excellent research,
alongside the 'outputs' and 'environment' elements.
Submissions will also include information about how
the unit has supported and enabled impact during
the assessment period.
page 6
REF Research Impact Decisions
The assessment of impact will be based on expert review
of case studies submitted by higher education institutions.
Case studies may include any social, economic or
cultural impact or benefit beyond academia that has
taken place during the assessment period, and was
underpinned by excellent research produced by the
submitting institution within a given timeframe.
Requires auditable evidence of impact.
page 7
REF2014 Research Impact Pilot Study
The REF team ran a pilot exercise which aimed to test the
feasibility of assessing research impact, and to develop the
method of assessment for use in the REF.
http://www.hefce.ac.uk/research/ref/pubs/other/re01_10/
The pilot exercise was undertaken in five units of
assessment (UOAs):
Clinical Medicine
Physics
Earth Systems and Environmental Sciences
Social Work and Social Policy
English Language and Literature.
page 8
Identified Good practice in REF Impact Case studies (1)
All material required to make a judgement about the impact was
contained within the case study and no additional information gathering
was required by panel members.
Clear identification of who the beneficiaries were or which
groups/organisations had changed something as a result of the research.
This could include ‘intermediary’ organisations as well as ‘end users’ or
audiences.)
A coherent narrative which explained clearly the relationship between
the research and the impact, and the nature of the impact or benefits
arising.
page 9
Identified Good practice in REF Impact Case studies (2)
Indicators that were meaningful and contextualised to support the case
being made, not used as a substitute for a clear narrative. The indicators
included precise information that was relevant to the case study.
A brief explanation of what was original or distinctive about the research
insights that contributed to the impacts.
Specific details about the names of researchers, their position in the HEI
and dates and locations of the research activity.
page 10
Identified Good practice in REF
Impact Case studies (3)
Specific and appropriate independent sources of corroborating
information were provided.
Focused and concise evidence, without generalised or overblown
statements or unexplained lists of publications or references.
Selected examples of case studies that scored highly and were
considered good practice in the pilot exercise are available on the
REF website at www.ref.ac.uk under ‘Impact pilot exercise’.
page 11
REF impact case Studies involving Engagement with the Public
There was a clear link between the department’s research and the
engagement activity.
Evidence was provided about dissemination (such as audience or visitor
numbers) as well as a clear explanation of the significance or benefits to the
audiences.
The activity went beyond ‘business as usual’ engagement (such as public
lectures), and attracted widespread interest or involved a programme of
activity which was often innovative and was sustainable or created
‘legacy’ resources.
In some cases there were additional, clearly identified benefits to
‘intermediaries’ (such as cultural institutions or curators), local
communities or to the economy.
page 12
Weaknesses in REF impact case studies:
Generalised, vague statements provided about key claims
without sufficient evidence, requiring panel members to
undertake further investigation to gather information or rely on
expert or prior knowledge.
Lack of clarity about the link between the underpinning research
and the specific impacts claimed.
page 13
Weaknesses in REF impact case studies (2):
Excessively long lists of unexplained publications or web
references, making it laborious for panels to identify salient
information.
Reliance on indicators that lacked context or meaning, for
example, numbers of hits on a website, without benchmarks
or further contextual explanation; or relying on anecdotal
evidence such as personal correspondence or quotes from
individual members of the public as evidence of impact.
page 14
GCU PREPARATIONS FOR REF2014
1. REF guidance/preparation of REF2014 submission
2. REF research impact case study development
3. New Current Research Information System (CRIS) to
support the development of the REF submission
4. New Research repository (REF publications
visibility/citations)
5. New REF Researcher staff development (via CREDO)
page 15
Where can I find more information about
REF2014 ?
The timetable for REF2014 with links to other material is
available at :
http://www.hefce.ac.uk/research/ref/timetable/
REF link on University Research Website:
http://www.gcu.ac.uk/research/researchatglasgowcaledonia
nuniversity/ref2014/
page 16
PART 2: WORKSHOP EXERCISE
Evaluating Impact Cases Studies from the 5 REF pilot areas:
Clinical Medicine
Physics
Earth Systems and Environmental Sciences
Social Work and Social Policy
English Language and Literature
Split into groups and discuss one area of interest to you (20 min)
Which case studies do you think are better and why ?
How would we develop case studies like these?
Rejoin larger group for close out discussion reporting
on best practice relevant to GCU
page 17