REF Event – 22 September 2011
The Oval, Kennington
See Slides

More briefing events Spring 2012

Panel criteria and working methods consultation: 5 October 2011

Separate response for each of the main panels

July 2011: published in draft form

Will produce FAQs from briefing events and booklet on REF for users.
Consultation responses to www.ref.ac.uk
Part 1 – generic – sets out how all panels will operate
Part 2 – main panel criteria and guidance
The working methods of each main panel and its sub-panels
Can ask for greater clarity
Consistency across the 4 main panels –
Impact – A and B have extensive examples, panel C only general comments
View is that more examples is helpful.
In Panel A
 multiple submissions are not allowed.
 Panel A trying to encourage interdisciplinary research
 Need to demonstrate that areas are academically distinct.
 Southampton would want multiple submissions in Panel A
 Timing for decision on multiple submissions is too late.
 The same impact can be claimed by different UoAs if there are different
aspects to the impact.
 Impact is more likely to cross disciplinary boundaries.
 Do look at UoA descriptors – bits that are missing
In Panel B
 Allows multiple submissions but general engineering does not permit multiple
submissions
 Does allow research groups in a UoA. Main Panel A is looking for research
groups rather than multiple submission.
 Outputs

Criteria

Output types

Co-authorship

Double-weighted outputs

Additional information

Use of citation data
In Panel D
 Expectation about performance type outputs.
1
Co-authored outputs
 Panel A
but need to provide description about contribution
 Panel B
no description required except Physics more than 5 authors
 Panel C
will expect a description if output submitted more than once
 Panel D
as Panel A
Always give priority around the panel for single-authored outputs, particularly in the
social sciences.
If cross referred, an output will be sent to another panel, but this advice is only
advisory and the original panel still judges the output’s worth.
Co-authorship – is it clear what information is asked for in panels A and D?
Double-weighted outputs
 not clear in the guidance.
 Need double-weighted outputs across all panels.
 Not in Panel A.
 Monographs might be double-weighted.
 Possibility of putting in a reserve item.
Citation Data
 Panel A will be using citation data
 Panel B variable using Google Scholar
 Panel C variable
 Panel D not using citation data



Panels will get the raw citation counts and info on the reliability of the data.
Very concerning about use of citation data – should not be used without
contextual data.
What is normal in a given field in a given year?
Impact
 Read carefully – what is criteria and differentiate from what is advice – this is
guidance
 Each panel has described a range of impact
 Impact is an area where panels are really looking for feedback from HEIs
 Need evidence of these:

quality of output

relation between research and impact of impact
Panel C helpful
Must demonstrate that quality of research is a minimum of 2*
Impact template
 Panel B criteria possibly not consistent
 Criteria defined for reach and significance: a holistic judgement - don’t need
4* in both to achieve 4*
 Reach is not geographically defined – the spread of impact does not
necessarily indicate qualify of impact.
 Understanding the diversity of beneficiaries
 Can have research und3ertaken within the HEI in say 1994, person has left,
but research has been continued with other staff at the HEI.
2
Environment
 Asking for a lot less than in RAE 2008
 PGRs and research income
 Panel B is asking for additional data
Template for Environment
 Overview, strategy, infrastructure, collaboration and contribution to the
disciplines
 Criteria of vitality and sustainability
 Panel B asking for FTE of students
 Some panels are asking for FTE of support staff on submission date
 Need to ensure that HESA data is being collected and applied consistently by
the panels
Responses must be made by mid-day 5 October online.
Separate response for each main panel.
Responses will help refine the criteria and working methods.
Supporting and Promoting Equality and Diversity through REF
Lessons learnt from RAE 2008
RAE evaluation suggested need
 More consistence across panels in criteria and processes relating to
individual staff circumstances
 Better (REF specific) training
 Proactive proforma-based procedure for disclosing individual staff
circumstances (we need to ensure we get the disclosures)
 Better communication of Codes of Practice
 More timely appeals process (must be soon enough for a staff member to be
submitted)
 To use next exercise to promote equality and diversity more
Examples of good Codes of Practice on Dianne Berry’s E&D website from the RAE.
New equality law since RAE 2008.
Overall approach in REF
 Equalities embedded

Equality and Diversity Advisory Group/Panel (EDAP)

Promotion of E&D through environment template
 HEIs required to develop, document and apply Code of Practice on staff
selection to:

Ensure work of ALL excellent researchers, regardless of
circumstances

Help compliance with equality legislation

Avoid inadvertent discrimination
There will be a review of effectiveness of measures for REF
Changes in HESA staff records in 2012 to include info and staff circumstances
3
Individual staff circumstances
 Age
 Disability
 Marriage and civil partnership
 Pregnancy and maternity
 Race
 Religion or belief
 Sex (including breastfeeding and childcare)
 Sexual orientation
 Part time and fixed term employment status
Codes of Practice
 Must be signed off by Head of HEI and submitted to REF team by July 2012
 Will be reviewed by EDAP in advance of submission deadline
 Will be published with submissions at end of process
The Guidance on Submissions provides detailed guidance
 Return – code of practice that could not be applied effectively and fairly
Look at how your RAE 2008 code of practice matches the new guidelines and the
new legislation.
Information will be placed on the E&D website next week.
Underlying Principles
Code should demonstrate fairness
 Transparency
how accessible is it to staff
Keep it succinct
Staff need to be aware of it, on maternity leave
 Consistency
across the HEI
clarity as to which staff situations are considered
 Accountability
what training has been conducted since Equality Act
Oct 2010
Have all UoA convenors been trained
 Inclusivity
What messages are managers sending out re
submission to the REF?
What needs to be included
 Information on staff and committees responsible for staff selection, including
terms of reference, role definitions and training undertaken
 Information on criteria to be used, how communicated, feedback to staff,
appeals process and timescales
 Don’t have to say the actual threshold level ie 2*, but must give contextual
information about the level you are seeking. How are these levels being
communicated to staff?
 Look for FAQs to provide to provide clarity
 If you have an appeals process, this will need to be detailed. Timescales that
staff need to be aware of
Equality Impact Assessments – these are required
 Required to conduct EIA on policy and procedures for selection staff to
determine whether selection policy may have differential impact on particular
groups
4




Should inform Code of Practice and be kept under review as submissions
prepared.
Should be informed by analysis of range of data, including (eligible and
submitted) staff data in respect of protected characteristics for which data
available.
Should be reviewed in light of mock exercise, appeals, and final submission
Further information on ECU website
Impact assessment at the start of determining the policy of selection of staff. This
assessment can be reviewed regularly and at final submission. Funding Councils
require publication of impact assessment.
How to do this will be on ECU website for REF.
Equality and diversity team may have templates for this.
Question:
Citations do have a gender bias. The Funding Council does not
condone use of citations for selecting staff for submission.
Individual Staff Circumstances
 Clearly defined circumstances (clearly defined periods of time)

ECRs, part-time ‘working’, maternity, paternity of adoptive leave,
secondments or career breaks
 More complex circumstances

Disability, ill-health or injury, mental health conditions, constraints
relating to pregnancy or maternity in addition to clearly defined period
of leave, caring responsibilities, gender realignment, other
circumstances related to protected circumstances
(not easy to work out time lost to research)
Staff disclosure
 REF team advise proactive approach (to ensure that all staff feel able to
declare restrictions regarding reduced number of outputs)
 Suggest giving every member of staff a form to complete
 Use a central group to assess cases and advise of decisions
 HEIs will need to consider confidentiality in procedures for staff disclosure
(ethics issue)
Consideration of individual staff circumstances
 Clearly define circumstances
 More complex circumstances

Considered by EDAP who will advise main panel chairs on whether
submitted number is appropriate
Submission of information
In REF1b
(200 words for clearly defined circumstances)
(300 words for complex cases)
For complex cases:
 Example cases on ECU website to provide guidance on how to structure
information re EDAP’s likely response
 By March 2011
 The initial set will be supplemented if further cases are brought to their
attention
5
Reduced outputs
Proposed tariffs allow for reduction in outputs in relation to:
 ECRs – determined by date become independent researcher during the
census period
 Other clearly defined circumstances – reduction in contracts hours / FTE
worked over census period
Need to look at consultation document for E & D
Guidance on submissions
Systems going to be using
Survey of submission intentions
Oct 2012 deadline – Dec 2012
 Collect information to plan panel’s workload
 Following this may need to appoint additional panel members
 Proposal to use this submission intention to request multiple submissions –
need to provide detail to support the request.
 Response from REF team by end January.
None of the submissions intensions are binding
Only one change to ask for multiple submissions
Will be asking:
 Total volume of submitted staff
 Impact areas (and details of highly confidential impacts)

This is to help identify expertise needed to judge the impact
 Highly confidential ie MOD, defence
 Research specialism (and volume)
 Outputs in welsh / other languages
 Work where cross-referral will be requested?
Submitting outputs
 REF team will attempt to source all submitted journal articles and conference
proceedings in electronic format
 Working with publishers (PLS) to gain agreement to access as in 2008 RAE
 For all other outputs types, and where not available for publisher, we will
require the output in either electronic form or a physical copy / appropriate
evidence
 Electronic wherever possible
Citation data
 All in A, some in B & C, none in D
 Using Elsevier and Scopus – raw citation counts
 In September 2012 can load your outputs on pilot REF system and verify
outputs and see citation count, system updated probably weekly to update
citation counts
 Panel will be looking at contextual information to go alongside citation counts.
For REF 2014, the citation counts will be taken just after submission
 Citation only to be used as a positive indicator.
6
HESA data
Want to align HEFCE data with HESA data
Data will be provided to us from the HESA data
Discrepancies in data
Proposal allow
 5% in total for all UoAs across all years
 10% in total for all UoAs, per year / income category
 These limits may also have a number element
 If an HEI expect this would be unachievable, they should have early
discussion with the relevant funding body.
Impact case studies
Each study must
 Describe and reference the underpinning research
 Explain the link between the research and the impact
 Explain and provide evidence of the nature
Impact stays with the institution, so research must have been done whilst staff is at
institution, but that member of staff could have died, not being submitted. Panel may
need access to the research.
No particular ‘model’ of impact generation is assumed.
The HEI need not have been involved in ‘exploiting’ the research
The contribution may be the sole research contribution, part of a collaboration or a
whole body of research
 Direct, indirect, linear or non-linear
No research = no impact
Panels are not going to follow up references. Case studies need to provide all the
details to justify the impact.
Sources will be used for audit purposes only – to verify claims but not additional
evidence.
Sources could include:
 Details of factual statements by ‘users’ that the HEI has collected, and can be
made available to the REF team
 Documentation, web-content or report (whether in the public domain or
available from the HEI)
In section 5 – do not put details of actual people, only titles
After REF 2014, HEIs will be able to redact info in the case studies or confidential
case studies before they are published on the web.
7
Summary of Briefing Sessions
Equality Headlines
Option 2 for maternity – strong favour for this
1 maternity leave = 1 reduction in outputs
14 months too high
Panel criteria
Greater justification for sub-panel differences and panel differences
Multiple submissions in particular
Co-authorship requirement for text boxes in panels A & ?
Discussion re double weighting – option to submit a reserve across all the panels
How will underpinning research for impact be judged?
What is the disciplinary justification for using citation data?
How will reach and significance be combined and judged?
Guidance on submissions
Request for multiple submissions earlier
May not have full details for submissions intentions on case studies for impact
REF team will batch up all the questions and use this to form FAQs on the web.
What should the focus be for briefing next Spring?
8