REF Briefing Event
RIBA, London
Thursday 31 May 2012
Introduction
On 31st May Professor Caroline Strange and Dr Tim Brooks attended a REF briefing event
organised by the REF team, the fifth and final event of a roadshow which took place around
the country during the second half of May.
Slides from the various sessions during the event (with the exception of the opening
introduction) are attached to the accompanying email. These notes are intended to be read
alongside them. As ever some of the most interesting information arose from responses to
questions asked by members of the audience, rather than from the material the REF team
had prepared.
Introduction to the Briefing Event (Graeme Rosenberg, REF Manager, HEFCE).
Survey on Submission Intentions
The Survey on Submission Intentions – which institutions will be invited to respond to by
letter by the end of July 2012, and which will be open for business between September and
December 2012, will collect information to enable the REF team and Main and Sub-Panels
to estimate the likely workload and areas of expertise they will need to be able to handle, for
example to inform the appointment of additional assessors. Information collected will include,
for each proposed UoA submission:
 staff volume (FTE for Category A staff; headcount for Category A and C staff).
 300-word statements on ‘areas of impact’, explaining the areas of impact in which
case studies are to be presented. In this, for example, stating that a case study will
focus on ‘policy impact’ is insufficient; the REF team is looking for more detail without
repeating the case study, e.g., ‘impact on national housing policy’.
 explanations of research specialisms within submissions. These may be the
names/areas of the research groups within a submission, or may make use of the
detail of UoA descriptors – different panels will have different expectations based on
their disciplines. There is no difference between a ‘common’ research specialism or
an ‘unusual’ one – while for the latter the panel may have to draft in additional
specialist expertise, they still need to know about the former to ensure they have
sufficient and appropriate panel membership to deal with it.
 expected volume of cross-referral requests
 languages other than English used in submitted outputs, and volume.
Collection of outputs
The REF team’s intention at present is that outputs will be collected by the same means as
for the RAE, that physical outputs will be deposited with them in a warehouse facility, while
electronic outputs with DOIs will be sourced directly from the publishers by the REF team.
(There was some suggestion, not expanded on, that non-DOI electronic format outputs
might be submitted electronically through the REF submission system rather than on CDROM, as was the case for RAE).
However, recent pronouncements by HEFCE on the desirability of open-access forms of
publishing and the possibility of mandating this in some way for the next REF has,
understandably, dismayed the publishers, and negotiations to arrange DOI-based access
have stalled. However, the REF Manager is confident these issues will be overcome.
Panel Criteria Guidance (Graeme Rosenberg, REF Manager, HEFCE, and Professor
Dame Ann Dowling, Chair, Main Panel B)
Codes of Practice (slide 7)
The REF’s Equality and Diversity Advisory Panel (EDAP) will produce reports on good
practice in the Codes of Practice it assesses, as soon as possible after the close of the
submission period for Codes of Practice (31 July 2012).
Institutions’ Codes of Practice will be published alongside submissions, after the assessment
process is complete. (If the RAE 2008 can be used as a model, for REF this is likely to be
some time in early 2015).
Clear Circumstances (slide 9)
The REF submission system will include a ‘clear circumstance calculator’ which will work out
the correct reduction for individuals with clear circumstances. (See the Submissions System
section slides).
Complex Circumstances (slide 10)
The REF submission system cannot calculate complex circumstances reductions.
REF 1b – the section where explanations of individual personal circumstances are made –
should be completed without referring to the member of staff by name. This will allow EDAP
to assess the claim anonymously. Similar practice should be followed in institutions.
If EDAP disagrees with an institution’s justification for a reduction on the basis of complex
circumstances, the ‘missing’ output will be rated unclassified. There is no facility for including
a reserve: the argument is that if reserves are included, this weakens any claim that
individual staff circumstances justify a reduction in the number of outputs required. Where an
individual is submitting four outputs, there is no need to declare either clearly-defined or
complex circumstances where these exist. The one exception is ECRs, whose status as
such must be declared.
If institutions have examples to which one of the existing case studies cannot be applied,
they are invited to ask the REF team to have EDAP prepare a further case study.
2
Staff who have retired and, after a period away, are brought back on a fractional contract,
can be submitted to REF. Their break in service counts as an ‘absence’ under the REF
rules, and should be treated as a career break as in other examples provided.
Staff who have held a teaching contract for some of the REF period cannot count their
period of teaching only as a ‘career break’ because REF rules consider a career break to
have taken place outside the HE sector.
Staff on management contracts, but who remain research active and could be submitted to
REF, must have an eligible contract – their management contracts will need amending.
Management activities in themselves do not constitute a legitimate reason for reducing
outputs required.
Sub-panel working methods (slide 14)
The calibration exercises to be run by each sub-panel, and within and between Main Panels,
should mean that different disciplines and subject areas are assessed equitable and evenly.
This should, for example, mean that within the Psychology sub-panel, applied psychology
research will not be disadvantaged by comparison with ‘hard’ Psychology.
The level of detail to which outputs will be scrutinised will depend on the time available. This
in turn is also dependent on the numbers and types of outputs submitted and the numbers of
assessors and members on the sub-panel. It is expected that most outputs will be read
twice, but some outputs, for example edited books and monographs, may only be read once.
Additional assessors (slide 15)
Additional assessors will be employed where needed to cover additional areas of expertise
etc. Additional assessors will either have an academic or user background, and will assess
either outputs or impact as a result. They will not, this implies, look at both.
Interdisciplinary research (slide 16)
Cross-referral may, exceptionally, take place and it can be either at the request of the
submitting HEI (if the panel agrees) or initiated by the panel. Cross-referral may be of a
single output, the work of a single individuals, or those of a research grouping – so there is a
great deal of flexibility.
Output Co-authorship (slide 19)
If in a single submission, i.e. UoA, the same co-authored output is, exceptionally, submitted
more than once (for a maximum of two authors) then a statement is required justifying the
inclusion of that output, describing the scale of the research and the distinct and substantial
contribution each author made. Where the same output is submitted to a different UoA, or to
the same UoA but in a different HEI’s submission, this is not required.
3
Citation data (slide 21)
The sub-panel for Computer Science will make use, in addition to the SCOPUS citation data,
information sourced from Google Scholar. Google Scholar citation data will not be part of the
citation information provided by the REF team through the submission system.
Some examples of impact (slide 26)
The examples shown on this slide are drawn together from all of the Main Panel criteria.
Case Studies (slide 28)
Ideally, evidence of impact should be quantitative, where this is available.
Panel members and assessors won’t follow up, for example, links to websites, to obtain
additional information. Case studies need to provide all of the information required to assess
the impact claimed, though of course a website may be included as a corroborating
reference. References and corroborative evidence will be sampled as a means of audit.
There was an interesting debate around how a piece of research can have both beneficial
and negative impacts. For example, research might lead to a new work-flow process which
brings greater efficiency, thereby having economic impact. However, the same efficiency
savings lead to job losses. No conclusion was reached as to whether the impact might be
‘marked down’ because the benefits were offset by negative consequences, but it was an
interesting discussion.
Of the panel, one academic member and one user member or assessor will look at each
case study.
Underpinning research (slide 29)
Panels will not, as a rule, request copies of the outputs referenced as underpinning
research. It may be that panel members want to look at some of these outputs, but in the
first instance they will try and source the outputs themselves through their own institutional
subscriptions etc. In the unlikely event that they cannot do so, submitting HEIs will be asked
to provide them as an audit query, but REF recognise and will ensure the need for sufficient
time to do this. However, HEIs are advised to ensure that outputs listed to corroborate
underpinning research are available!
The assessment of underpinning research is only to ensure that the research meets the 2*
quality threshold, and as such will be light-touch.
4
Data and Audit (Anna Dickinson and Vasanthi Waller, REF team)
Obtaining Citation Data (slides 5-9)
Citation data will only be available via the REF submission system for output information
entered against those UoAs making use of it. REF team members talk in terms of
‘requesting’ citation data, but the process is essentially instantaneous.
When citation data is requested, the system may return a number of responses, ranging
from ‘matched’ to ‘no match found’. In some cases, it may be necessary to take further steps
– e.g. to manually identify the correct output from a list provided by the system. HEIs are
required to have entered all outputs and attempted to match the outputs for citations at an
(unspecified) deadline ahead of the actual submission deadline.
Ideally, the output will be cleanly matched. The screen will show both the metadata for the
output originally submitted for matching, and the metadata identified within Scopus. It is not
necessary for institutions to have a Scopus subscription to view this data.
Scopus data provided through the REF submission system will not be “live”, but will be
refreshed on a weekly basis.
Contextual Citation Data (slides 13-15)
Contextual citation data – demonstrating, for example, citation ‘norms’ in a particular subject
area – will be made available to panels in early 2014. This will cover the period from 1
January 2008 to 31 December 2012. Equivalent data, covering the period 1 January 2007 to
31 December 2011, will be made available to HEIs in early 2013. The REF team believes
that this will be sufficiently in line with the data the panels will use, to make it valuable for
institutional planning.
In the assessment itself, panels will use a ‘snapshot’ of citation data taken on a particular
date (to be specified) after the submission deadline has passed, rather than citation data on
the day of assessment, for consistency.
Environment data (slides 16-17)
Unlike the RAE, research environment data (doctoral degrees awarded and research grant
income) relates to the whole UoA, rather than just the individual members of staff submitted
within it. (See paragraphs 166 and 171 of Assessment Framework and Guidance on
Submissions).
Data on doctoral degrees awarded for the 2012-13 academic year will be calculated by
institutions alongside their preparations for their HESA return, rather than by using their
HESA returns. One of the pieces of check documentation that will be returned by HESA as
part of the 2012-13 data submission will be the doctoral degrees awarded data, which will be
used by REF to check the data submitted directly to them. It is likely that this check
documentation won’t be returned by HESA until the end of 2013, too late for HEIs to make
5
use of in their REF submissions. Presumably the same check response applies to financial
data returned, though this was not specified; certainly we are not expecting to be able to
make use of HESA returns here either.
Research doctoral degrees awarded (slide 19)
Doctoral degrees awarded refers to awards made to home and overseas students only.
Awards made to students recorded in the ‘aggregate offshore record’ (students studying (to
date) wholly outside the UK who are either registered with the reporting institution or who are
studying for an award of the reporting institution) do not count.
MPhil awards do not count, whatever HESA may have previously said about its definition of
‘research degrees’.
Awards are only counted in the first year they are awarded – there are some instances of
institutions returning the same award to the same student in several years. Data going back
to 2007-8 will be used to check for duplicate awards. (No response was given to my query
about how this could be done, given we have previously been told that the reason for leaving
the 2007-8 research environment data out of REF was that it was not defined in the same
way, so using it for audit seemed contradictory).
For some UoAs, additional research environment data is requested in the REF5 narrative.
For Business and Management Studies (UoA 19) and Education (UoA 25), submissions are
required to include data disaggregating between PhD and Professional Doctorate awards.
Research income-in-kind (slides 22-23)
Research income-in-kind data originating from RCUK will be supplied in a few weeks’ time,
via the REF team. Income-in-kind data will be provided using the template provided in slide
23. In addition to the RCUK data, various income-in-kind data will be provided by various
health research funding bodies, e.g. NIHR, but no further information or timescale was given
for the supply of this information. Other research income-in-kind data is not eligible.
Returning data and data adjustments (slides 24-26)
HEFCE does not expect to see data returned against UoAs to which we are not otherwise
submitting; at the same time, we aren’t expected to return data in UoAs with no connection
to that data. It was pointed out that the rules and regulations for REF don’t prevent HEIs
from doing so, although the person posing the question acknowledged that tactically
allocating environment data to UoAs went against the spirit of the exercise. The presenters
agreed to take this point back to REF for additional guidance.
Various submission system limits have been put into place, which cannot be exceeded by
institutions. These limits are the HESA figures for the institution as a whole for 2008-9 to
2011-12 with various caps e.g. £200k and 5% over the period. Limits for 2012-13 cannot be
built into the system but the REF team will manually cross-check with HESA data for that
year. The REF team does expect to spot what was described as systematic over-reporting
that is within the system limits.
6
Where institutions are aware they wish to submit data exceeding the system limits, they can
request permission to do so from the REF team. (This replaces the instruction in the REF
guidance that HEIs should contact their funding council). HEIs should explain why the data
they wish to submit was not included in the HESA return. If the request is accepted, the REF
team will amend the system submission limits. Where UoA convenors identify the need to
make such requests, they should raise it as soon as possible with Caroline and Tim.
Audit – staff (slides 29-30)
The REF team recognise that, despite the terminology used in Guidance on Submissions,
contracts themselves do not always specify sufficient information to enable eligibility to be
determined. Supporting documentation such as job descriptions can be used.
There will be a FAQ indicating the content of information that should be provided in REF 1b
(individual staff circumstances) including for ECRs. Where evidence is requested through
audit of, for example, ill-health, the REF team will not expect to see primary evidence (e.g.
doctor’s notes). They will, however, require evidence that such evidence has been seen by
submitting HEIs.
For staff who come from another institution and whose individual circumstances relate to
their period of service there, we would need to obtain verification from that institution as to,
for example, part-time working, maternity leave and so forth.
Evidence for an ECR might be an extract from their CV.
The REF team seems to be relatively understanding that, particularly for older evidence,
some simply isn’t now available – e.g. HR records going back to 1992.
Audit – outputs (slide 31)
The REF team will be verifying/auditing outputs. However, it is expected that almost all such
queries can be handled by the REF team themselves, without needing the assistance of the
HEI.
Note however that for the audit of impact, it may be necessary to provide outputs referenced
as underpinning research (see below).
Audit – environment (slide 33)
In certain cases HEIs will be aware that their submission is very likely to be audited, for
example where environment data is close to system limits. HEIs are welcome to prepare
justifications for such occurrences, but are advised to wait to provide them, rather than preempting such a request.
7
Submissions System (Vicky Jones and Nick Soper, REF team)
Role of DCSG (slide 3)
The DCSG is the Data Collection Steering Group, which was formed in October 2010 to
advise on the data collection elements of the REF and oversee the development of the
submissions system.
[Screenshots of the data collection system were provided at the briefing event, as part of a
demonstration of the data collection system, and have not been supplied electronically. The
following came up during the demonstration].
The REF submissions system is effectively an evolution of the RAE submissions system.
There are a number of key differences:
 data entered into the pilot version of the system (which will be available to all HEIs
between September and November 2012) will be transferred to the actual version.
This transfer will be automatic and doesn’t require any intervention from HEIs.
 helptext will be built into the system, rather than being supplied in hard copy (though
a PDF version will be available to HEIs to print if required). The helptext will be
sensitive, i.e. asking for help from a particular page will bring up help specific to that
page (like the Je-S system).
 for the text-based parts of the submission (impact summary, impact case studies,
environment narrative) a PDF upload function will be available, replacing the built-in
text editing function.
 There will be an automatic ‘timeout’ after 30 minutes of inactivity, but the hope is that
part of the timeout routine will save any outstanding data, so nothing is lost.
Various different rights can be set up to grant different individuals varying levels of
permissions, and to different parts of the submissions. The system will be set up with the
HEI’s Main Contact as the primary administrator; it is advisable to create a second
administrator account for emergencies.
The submission system will contain a number of in-built verification processes. These will
resolve in three kinds of errors. Critical errors must be corrected before the information can
be saved. Submission errors do not prevent data being saved, but will need to be corrected
before submission can be allowed. Warnings flag up possible errors – e.g. an ISSN that
doesn’t seem to make sense – but do not prevent data being saved or submitted.
When importing and exporting data from the system in bulk, it is possible to override the
error messages, so that erroneous data is uploaded (which will then need to be corrected),
or for the erroneous data to be omitted from the import/export.
Tim Brooks
Caroline Strange
8 June 2012
8