The Research Excellence
Framework
Data and Audit
May 2012
Presentation outline
•
•
•
Citation data
•
The submission system
•
Contextual data
REF 4 data
•
Research doctoral degrees awarded
•
Research income data
•
Returning data
Audit and data verification procedures
Citation data
“Those panels using citation data will do so
within the framework set out in ‘guidance on
submissions’ (paragraphs 131-136). In particular,
they will consider the number of times an output has
been cited as additional information about the
academic significance of submitted outputs. Panels
will continue to rely on expert review as the primary
means of assessing outputs, in order to reach
rounded judgements about the full range of
assessment criteria (‘originality, significance and
rigour’).”
(Panel Criteria and Working Methods, Part 1 paragraph 51)
Citation data
•
Will only be used by the following sub-panels:
Sub-panel 1: Clinical Medicine
Sub-panel 2: Public Health, Health Services and Primary Care
Sub-panel 3: Allied Health Professions, Dentistry, Nursing and Pharmacy
Sub-panel 4: Psychology, Psychiatry and Neuroscience
Sub-panel 5: Biological Sciences
Sub-panel 6: Agriculture, Veterinary, and Food Science
Sub panel 7: Earth Systems and Environmental Sciences
Sub-panel 8: Chemistry
Sub-panel 9: Physics
Sub-panel 11: Computer Science and Informatics
Sub-panel 18: Economics and Econometrics
Obtaining Citation data
•
Items matched to Scopus via the submission system
•
•
•
Via the web interface
•
When entering individual outputs
•
For whole submissions
On bulk import
Support from Elsevier available
Obtaining Citation data
Obtaining Citation data
Obtaining Citation data
Support from Scopus
•
•
•
Elsevier will provide technical support on the matching
process during the submissions phase
Initial means of support will be by email
We will publish further details of support arrangements
before the submissions period
Contextual citation data
•
Sub-panels using citation data will also receive contextual
data about citation rates to help inform their interpretation of
the citation counts.
Contextual citation data
•
•
Scopus have assigned journals to ~350 subject categories
Contextual data will be provided based on these subject
categories, for each year
•
•
•
•
The average number of times items were cited
The number of citations required to be in the top 1%,
5%, 10% and 25% of papers in that category
The data will also be provided at the level of UOA by
combining subject categories, after the submission system
closes.
Data will be provided for the years 2008 to 2012
Contextual citation data (2)
•
•
•
HEIs will be given the same data when the submissions
system opens, except:
Data will be provided for the years 2007 to 2011
UOA level data cannot be provided
•
•
•
•
Unable to assign Scopus subject categories to UOAs
until all REF data has been collected.
Data will be available as an Excel workbook to download
via the submission system.
Data will be available shortly after the submission system
opens
HEIs will have access to the same data as the panels after
the close of the submissions system
Questions?
Environment data
•
All submissions to include data on:
-
•
•
•
Research doctoral degrees awarded (REF4a)
Research income (REF4b)
Research income in-kind (REF4c)
Definitions are aligned with HESA returns
Data relates to whole unit of assessment - not just
submitted staff
Sub-panels 8, 9, 19, 25 and 26 request specific
additional data, to be included within the environment
template (REF5)
Environment data (2)
HESA data will be presented to institutions at three
stages:
-
Data for 2008-09, 2009-10 and 2010-11 in May 2012
-
Excel workbook containing data
Guidance note explaining how the data were
extracted
Data for 2011-12 in April 2013
Doctoral degree awarded data for 2012-13 will be
available from HESA when the 2012-13 HESA student
record is submitted
Research doctoral degrees
awarded
•
Taken from the HESA student record
•
All instances who successfully complete with:
•
•
•
D00: “Doctorate degree obtained primarily through
advanced supervised research written up as a
thesis/dissertation”
•
D01: “New Route PhD”
An award is only counted the first year it is given to an
instance
An award is not counted if the instance is not closed.
Subject mapping
•
•
Awards data are mapped to:
•
RAE 2008 UOAs
•
Cost Centres
Data linked between years back to 2007-08
•
To map awards to cost centres and RAE UOAs
•
To detect instances awarded multiple times
Research income
•
Taken from the HESA Finance Statistics Return (FSR)
•
FSR Table 5b: External research grants and contracts
•
Academic departments (Head 1)
•
Academic services (Head 2)
•
Less any income passed on to other institutions or
organisations as part of a collaborative project or
subcontracted work (Head 5)
Research income-in-kind
•
Data on research income in kind will be provided to
institutions from:
•
The UK Research Councils
•
Health research funding bodies:
•
•
•
•
The National Institute of Health Research (NIHR)
The Chief Scientist Office, Scottish Government
Health and Social Care Directorates
The Welsh Government’s National Institute for
Social Care and Health Research (NISCHR)
The Health and Social Care Research &
Development (HSC R&D), Northern Ireland.
Income-in-kind data template
Income-in-kind (£ by academic year)
Institution Department
Researcher
title
Researcher
surname
Researcher
initials
Facility
2008-09
2009-10
2010-11
2011-12
2012-13
Returning data
•
•
•
•
•
Data should be mapped to REF UOAs
Research degrees awarded for all students in those UOAs
should be returned, not just those with supervisors returned
to the REF
Income associated with all staff in those UOAs should be
returned, not just those who are submitted
REF4 data should align with corresponding HESA, RCUK,
health research funding bodies data as closely as possible
Validation checks will be run within the submission system
software before this data can be submitted
•
•
Returning data (2)
The total number of research degrees awarded submitted by the
HEI as a whole should not exceed the total number of degrees
awarded provided in the HESA data by:
•
•
The total amount of research income submitted by the HEI as a
whole should not exceed the total amount of income provided in
the HESA data by
•
•
•
•
More than 20 and 5% over the period 2008-09 to 2011-12
More 10 and 10% in any AY2008-09 to 2011-12
More than £200,000 and 5% over the period 2008-09 to 2011-12
More than £100,000 and 10% in any AY2008-09 to 2011-12.
More than £100,000 and 10% in any category of income over the
period 2008-09 to 2011 12
The total amount of research income in kind (REF4c) submitted by the
HEI as a whole should not exceed the total amount of income in kind
provided by the research councils and health agencies by:
• More than £100,000 and 5% over the period 2008-09 to 2011-12
• More than £50,000 and 10% in any AY2008-09 to 2011-12
Data adjustments
•
•
•
Institutions should contact the REF team as soon as
possible where they propose to return environment data
that falls beyond the set tolerance limits.
Requests should be directed to info@ref.ac.uk
All requests must be submitted by 1 June 2013 at the
very latest.
Questions?
Audit & data verification procedures
•
•
•
•
Audit and data verification of REF submissions is necessary to:
•
•
support principles of fairness and to adhere to good regulation
provide sufficient confidence in the quality and reliability of
data and assessment outcomes
Any part of a REF submission can be audited and institutions
should maintain accurate records in order to verify any information
provided as part of their submission.
Further guidance on audit and data verification will be published in
June 2012
Audit procedures will involve the following processes:
• Routine checks
• Structured and targeted audits
• Systematic data comparisons
• Panel instigated audits
Audit – Staff eligibility
•
•
•
A sample of randomly selected Category A and C staff will
be audited in 1 - 4 units of assessment for every institution
Institutions will be contacted promptly following the
submissions deadline to request evidence of staff eligibility.
Evidence may include employment contracts or other HR
personnel records
Audit – Individual staff circumstances
•
•
•
•
Submitted details for every individual with a clearly defined
reduction in outputs will be examined to check that the
reductions has been calculated correctly. Further information
may be requested.
All complex circumstances will be reviewed by EDAP and
further information may be requested to inform judgements.
A sample of staff with individual circumstances will be audited
from each institution to verify the circumstances declared.
Evidence will be largely based on:
• Staff contracts
• HR confirmation of leave
• Confirmation that appropriate evidence has been consulted
Audit – Research outputs
•
•
Audit of research outputs will include a systematic data
comparison of:
• Outputs that may have been returned to RAE2008
• Outputs that are listed more than twice within a
submission
Panels may also raise audit queries on research outputs,
these will be followed up, where necessary, by the REF
audit team
Audit - Impact
•
Routine audit will be carried out on a sample of impacts
from each HEI.
•
•
Queries will be informed by advice from the panels and
are expected to peak in March – June 2014
Queries are expected to relate to:
•
•
Claims made that can be corroborated through
independent sources
Eligibility of underpinning research
Audit – Environment
•
Targeted audit will be undertaken where:
•
The environment data (REF4a/b/c) is close to the
system limits
OR
•
•
Where a comparison at subject level or cost centre
suggests discrepancies in the allocation of data to
REF UOAs
If audited, HEIs will need to demonstrate how any additional
data is eligible; and/or justify how this has been allocated to
REF UOAs.
Audit – next steps
•
Guidance on audit and data verification is due to be
finalised and published on the REF web-site in June
Questions?