REF 2014 and the Business &
Management community
Mike Pidd
Chair: sub-panel C19 (Business & Mgt Studies)
Department of Management Science
Lancaster University Management School
UK
M.Pidd@lancaster.ac.uk
2
Overview
 Purposes of the REF (the
Research Excellence
Framework)
 REF versus RAE 2008
 Panels
 Staffing




Outputs
Environment
Impact
Timetable
You can always blame the REF
3
REF purposes: official statement
 FUNDING: inform research funding allocations by
the four UK HE funding bodies (approximately £2
billion per year)
 ACCOUNTABILITY: for public funding of
research so as to demonstrate its benefits
 QUALITY INDICATORS: provide benchmarks
and reputational yardsticks
4
REF principles: official statement
 Expert, peer review panels
 Equity: all types of research and forms of
output across all disciplines shall be assessed
on an equal basis
 Equality: HEIs encouraged to submit the work
of all their excellent researchers
 Transparency: assessment criteria and
procedures, and outcomes to be published in
full
5
RAE 2008: how did it work?
Research outputs: 4 per person (70%)
Research
environment
(20%)
e.g.
Research income
PhD students
Staff development
Peer
review
Esteem &
impact
indicators
(10%)
Quality
profile
Weighted and aggregated across each submission
Nossex
Business
School
Quality level
4*
3*
2*
1*
u/c
% research activity
10
40
35
15
0
6
REF 2014: how will it work?
Research outputs: 4 per person (65%)
Research
environment
(15%)
e.g.
Research income
PhD students
Staff development
Peer
review
Impact
(20%)
Spot the
differences
Quality
profile
Weighted and aggregated across each submission
Nossex
Business
School
Quality level
4*
3*
2*
1*
u/c
% research activity
21
38
37
4
0
7
Overview of REF assessment framework
Overall quality
Outputs
Impact
Environment
Maximum of 4
outputs per
researcher
Template and case
studies
Template and data
65%
20%
15%
8
Key changes since RAE 2008




Includes assessment of impact
Removed esteem as a distinct element
Structured approach to research environment
Fewer UOAs/panels operating more consistently
 Only 4 main panels (main panel C for BMS)
 No separate panel for Accounting & Finance
 Strengthened equality and diversity measures
 Revised eligibility criteria for staff
 Some UOAs will make (limited) use of citation data
 ... But not in BMS (C19)
 Increased ‘user’ input on panels; and an integrated role for
additional assessors
 Publication of overall quality profiles in 1% steps
9
Panels and sub-panels
10
REF 2014: roles of panels
 MAIN Panel C




Panel criteria & working methods
Oversight during assessment process
Consistent application of overall assessment standards
Sign off final profiles
 SUB-PANEL C19: Business & management studies
 Limited, specific variations in criteria & working
methods
 Conduct the assessment
 Recommend final profiles
 Add extra assessors & users during 2012/13
11
Structure of main panel C: REF 2014
UK academic members (sub-panel Chairs):
Janet Finch
Chair
Manchester/Keele
Cara Aitcheson
Sports-related studies
Edinburgh
Hastings Donnon
Anthropology & dev studies
Queens Belfast
Gillian Douglas
Law
Cardiff
Colin Hay
Politics & int studies
Sheffield
Peter Neary
Economics etc
Oxford
Alan Penn
Architecture, built env & planning
UCL
Mike Pidd
Business & mgt studies
Lancaster
Andrew Pollard
Education
Inst for Education
Keith Richards
Geog, env studies & archaeology
Cambridge
John Scott
Sociology
Plymouth
Peter Taylor Gooby
Social work & social policy
Kent
12
Structure of main panel C: REF 2014
International and user members
Trevor Barnes
Geography
UBC
Frans Berkhout
Innovation & sustainability
VU Amsterdam
Francois Borguignon
Economics
Paris School of Economics (prev
Chief Econ, World Bank)
Paul Finch
Architecture & planning
Commission for the Built
Environment
Herbert Kritzer
Law & politics
Univ Minnesota
Jone Pearce
Business & mgt studies
UC Irvine
Mark Robson
Various
Bank of England
Sue Rossiter
Education
NFER
Martin Walsh
Charity sector
Oxfam
Paul Wiles
Social work & social policy
Oxford (formerly Home Office)
Sharon Witherspoon
Social policy
Nuffield Foundation
Sub-panel C19: business & mgt studies
Mike Pidd1
Lancaster
Chair & member of Main Panel C
John Arnold
Sheffield
Martin Laffin
Durham
Jan Bebbington
St Andrews
Alan Marsden
Ex-Arup
David Blackaby2
Swansea
Kathrin Moeslin
Erlangen-Nuremberg
Jane Broadbent3
Roehampton
Peter Naude
Manchester
Robert Blackburn
Kingston
Andy Neely
Cambridge/Cranfield
Chris Brooks
Reading
Caroline Oades
ACCA
Colin Eden
Strathclyde
Richard Thorpe
Leeds
Paul Edwards
Birmingham
Ian Tonks
Bath
Guy Fitzgerald
Brunel
Caroline Tynan
Nottingham
Keith Glaister
Sheffield
Terry Williams
Hull
Mark Jenkins
Cranfield
Hugh Willmott
Cardiff
2: Joint with Economics sub-panel
3: Deputy Chair
13
14
Staff eligibility & other rules
15
Staff: Eligibility & requirements
 Category A staff
 Academic staff with a contract of at least 0.2 FTE,
 on payroll of the HEI 31 Oct 2013
 primary employment function of either ‘research only’ or ‘teaching
and research’
 Category C staff:




Employed by an organisation other than an HEI
Have contract or job role including research,
Research primarily focused in the submitting unit on 31 Oct 2013
Will contribute to the quality profile but not volume for funding
 Research assistants are only eligible by exception
 Institutions MUST produce an Equality & Diversity Code
 Four outputs per person unless special circumstances
16
Staff: Individual circumstances
Reduced # outputs allowed
Clearly defined (rules)
Complex (EDAP)
 Qualified as ECR
 Part-timers
 Maternity, paternity or
adoption leave
 Secondments of career breaks
outside HE




Disability
Ill health or injury
Mental health conditions
Constraints relating to
pregnancy or maternity in
addition to maternity leave
 Childcare or other caring
responsibilities
 Gender reassignment
 Other circumstances relating
to characteristics protected by
the Equality Act 2010
17
Output tariffs: individual circumstances
Early career researchers
Date at which the individual first met the definition of an Outputs may be
ECR:
reduced by up to:
Between 1 August 2009 and 31 July 2010
1
Between 1 August 2010 and 31 July 2011
2
After 1 August 2011
3
Part-timers and career breaks
Equiv months absent
from work (1 Jan 2008 to
31 Oct 2013):
For part-time staff, average
FTE worked (1 Jan 2008 to
31 Oct 2013):
Outputs may be
reduced by up to:
14 – 27.99
0.601 – 0.8
1
28 – 48.99
0.301 – 0.6
2
49 or more
0.3 or less
3
NOTE: % availability need not be same as % used in Volume calculation
18
Other individual circumstances
 Maternity, paternity & adoption
 Outputs may be reduced by 1 for each period of
statutory leave over the REF period.
 Also by 1 for each extra period of such leave lasting 4
months or more.
 Complex circumstances
 REF EDAP (Equality & Diversity Advisory Panel)
 Recommend any reductions to main panel Chair, who will
decide on any reductions
 Sub-panels
 Must abide by main panel Chair’s decision
 Will not be told of the circumstances
19
Outputs
20
Eligibility of outputs
 Must be a product of research (see REF defn)
 All forms of output are welcome and will be
treated equally
 Into public domain between 1/1/08 and
31/12/13
 BUT may include pre-published work (e.g. online
first or working paper if (and only if)
 Pre-published in 2007 but in print form after 1/1/08
 Not included in RAE 2008.
21
Co-authored outputs
 Recognised to be increasing as collaboration grows in Main
Panel C disciplines
 Where outputs are co-authored/produced, submitting
author expected to have made a substantial contribution
 Sub-panels wish to receive the fullest picture of a
submitting HEI’s research, hence
 Co-authored/produced outputs may be submitted twice in the
same UOA by the same HEI
 MUST be accompanied by a statement explaining the substantial
and distinctive contribution of each of the submitting authors
 Still unclear about pre-published, co-authored work if not used by
submitting author/institution in RAE 2008.
22
Double weighted outputs
 May be requested for outputs
of extended scale & scope
 MUST have supporting
statement
 Sub-panels will assess the
request separately from its
quality
 If accepted: counts for 2 items
 1 reserve allowed for each
double weighing request
23
Environment
24
Environment assessment criteria
 Vitality: reflects the existence of a thriving, dynamic and
fully participatory research culture based on a clearly
articulated research strategy, displayed both within the
submitting unit and in its wider contributions, and in terms
appropriate to the scale and diversity of the research
activity that it supports
 Sustainability: understood in terms of the extent to
which it is capable in the future of continuing to support
and develop such research activity as defined in the quality
levels, both within the submitted unit and the discipline
more generally
25
Research environment: 2 aspects
1. Narrative (template) with 5 sections
i. Overview
ii. Research strategy
iii. Staffing strategy & development, including research
students
iv. Income, infrastructure & facilities
v. Collaboration & contribution to the discipline
2. Quantitative data: based on HESA stats: e.g.
doctoral degrees, income ...
26
Impact
27
Impact defined
 An effect on, change or benefit to the economy, society,
culture, public policy or services, health, the environment
or quality of life, beyond academia
 Includes an effect, change or benefit to:
 The activity, attitude, awareness, behaviour, capacity, opportunity,
performance, policy, practice, process or understanding
 Of an audience, beneficiary, community, constituency, organisation
or individuals
 In any geographic location whether locally, regionally, nationally or
internationally
 Excludes impacts on research or the advancement of
academic knowledge within HE; and impacts on teaching or
other activities within the submitting HEI
28
Examples of impact: main panel C
Impacts on:
Examples:
Creativity, culture and
society
Enhancements to heritage preservation, conservation and
presentation
Shaping or informing public attitudes and values
Economy, commerce
or organisations
Improved products, processes or workplace practices
Enhanced corporate social responsibility policies
Environment
Changes in public awareness or behaviour
Business operations have been changed to achieve
environmental objectives
Health and welfare
Development of policy or practice with regard to health
services or social care provision
Practitioners and
professional services
Influence on professional standards, guidelines or training
Practitioner debate has been stimulated by research findings
Public policy, law and
services
Legislative change or effect on legal practice
Influence on policy (by government, NGO or private
organisation)
Impact on democratic participation or access to justice
29
Research impact submission & criteria
Impact template
How is impact facilitated? Context, approach(es) to
impact, strategies & plans, relation to case studies
Impact
case 1
Impact
case 2
Impact
case 3
Impact
case 4
Impact
case n
CRITERIA
Reach: understood in terms of the extent and diversity of the
communities, environments, individuals, organisations or any other
beneficiaries that have benefitted from or been affected (not geographic,
but relative to potential domain)
Significance: understood in terms of the degree to which the impact
has enriched, influenced, informed or changed policies, opportunities,
perspectives or practices of communities, individuals or organisations
30
Impact cases
 Maximum of 4 pages per case.
 Must demonstrate that the unit’s research made a
distinctive contribution to the impact claimed during the
REF period 1/1/08 to 31/12/13
 Evidence for the impact claimed must be cited
 May include qualitative, quantitative or material
 Should be independently verifiable wherever possible
 Case studies should not describe activity alone, but should
make clear links between activity and impact claimed
 Research on which claimed impact is based
 Must have taken place at the submitting HEI
 Should meet the definition of 2* (internationally recognised)
 Underpinning research published since January 1993
31
REF 2014 Timetable
2011
2012
2013
2014
• Panels appointed
(Feb)
• Final panel criteria
and methods (Jan)
• Launch REF
submissions system
• Panels assess
submissions
• Guidance on
submissions (Jul)
• HEIs submit codes of
practice (Jul)
• Recruit additional
assessors
• Publish outcomes
(Dec)
• Draft panel criteria
for consultation
(Jul)
• Pilot of submission
system (Sep)
• Staff census date (31
Oct)
• Requests for multiple
submissions (by Dec)
• Submissions
deadline (29 Nov)
• Survey of submission
intentions (Dec)
32
REF 2014: important dates
Date
31/12/13
Significance
End of REF period for outputs, impact template
& cases and research underpinning impact
29/11/13
31/10/13
31/7/13
Submissions due
Census date for staff
End of REF period for research students,
research income & environment
Start of REF period for outputs, environment &
impact template & cases
01/01/08
01/01/93
Start of period for research underpinning
impact
Questions & discussion
34
Likely sub-panel workloads (based on 2008)
Sub-panel
FTE staff/member
16
Architecture & planning
45
17
Geography & architecture
67
18
Economics & econometrics
47
19
Business & management studies
159
20
Law
100
21
Politics
62
22
Social work & social policy
50
23
Sociology
47
24
Anthropology & development studies
31
25
Education
81
26
Sports-related studies
38
Source: REF team
35
HEFCE QR Funding weighting
4*
3*
2*
1*
0
09/10 based on RAE 2008
7
3
1
0
0
11/12 based on RAE 2008
9
3
0.294
0
0
Based on REF 2014
?
?
?
0
0
Distinct possibility that QR weighting for 2* will be zero after 2014
36
Structure of main panel I: RAE 2008
Main panel I: Chair David Otley
34: Economics & econometrics
(David Greenaway, Nottingham)
35: Accounting & finance
(Andy Stark, MBS)
Jane
Broadbent
David
Blackaby
36: Business & management
(Mike Pidd, Lancaster)
Ray Paul
37: Library and information mgt
(John Feather, Loughborough)
Sub-panels did the work
Main panel had oversight