FUNDING COUNCILS’ REVIEW OF RESEARCH ASSESSMENT
Liverpool JMU Response
General Comments
JMU welcomes the review of the RAE, in particular the opportunity it represents to allow a wider and more
relevant view of research to emerge.
The process and approach and approach of the RAE at present is problematic. A system, in which academics
undertake and quality assess their own work, is fundamentally flawed. Citations are largely by academics, grant
applications reviewed by academic peers, academic papers are refereed by academics. Not only is the current
peer panel approach narrow in terms of what is valued but it is devalued by those benefiting from the results
actually producing them.
We need to reflect on the notion of expert and move from a system when academics are assessing the
academic worth of the work submitted to one in which expert and possibly lay involvement assesses the quality
of work that is of value to the UK as a whole.
a. What is the purpose of the RAE?
The purpose of the RAE should be to inform the allocation of resource to research activity within universities
consistent with the needs of the UK in the 21st century. This role is difficult to disentangle from other resource
streams for research (such as the Research Councils) so we will focus here on the particular role of the funding
councils within the dual support system of funding background activity and allowing sufficient flexibility in
research funding for new or unfashionable areas to be supported. We would argue that this implies widening the
scope of the review to take into account broader issues than those narrowly associated with restricted views of
academic excellence. Whilst a clear requirement is a system that supports world class innovative research,
wherever it is found, it is also necessary to ensure that the research activity underpinning knowledge transfer in
all its various forms (teaching of students to spin out) is also supported. The RAE should therefore be capable
of identifying leading edge research and research which has genuine relevance to the knowledge economy.
We would argue for a strong focus on the needs of the funding council’s for research assessment and the
avoidance of overloading of the assessment process with the needs of other stakeholders. If other stakeholders
find the outcomes of RAE’s useful, that is to be welcomed but it would be a mistake to impose on an already
difficult and onerous task requirements that might distract from appropriate and accurate assessment. The
funding councils should be clear about exactly what any RAE measures but, if they do this, cannot be held
responsible for misuse by other agencies and should not compromise their needs in an attempt to make good
the shortcomings of others.
JMU is of the view that given its role in dual support the methodology for RAE support should be separated from
that of other funding agencies. Co-ordination of activity should be a matter of policy and strategy not an inbuilt
part of the data collection exercise. It is of course possible to use and exploit data from other sources but that
should not prejudice the role and function of the RAE itself.
b. How often should research be assessed?
The current time between assessments seems appropriate. We do not feel that there is any compelling reason
why research in different subjects should be assessed at the same time. A rolling programme of assessment
would be reasonable. We would argue that much larger groupings of subjects could be assessed together as
this would avoid some of the issues of fragmentation, territoriality and inconsistency of assessment between
UoA’s found in the current system.
c. What is excellence in research?
Excellence in research must be related to the reasons why we undertake research. Excellence is not an isolated
quality of research but can only be judged against a view of why the research is needed. Excellent research is
research that fits that need. Having said this it is possible to identify an important general characteristic of
excellent research – it is of use either directly in its own right or it is of importance in the development of new
knowledge, that is, it is useful to other researchers in the development of further knowledge. In terms of
excellence on this latter characteristic it would be appropriate to look judge against international norms.
It is vital that a broader understanding of the nature of research within universities and the reasons that research
is undertaken becomes part of the assessment process. At present the view of research is too heavily focussed
on a model of research support derived from big science (implying spatial concentration of research is in itself a
good thing) and on the requirement to compete against international elites supported under completely different
funding arrangements. We would note that such a model does not take into account, for example, the funding
requirements of low cost research subjects in the modern world of wide access information resources, emerging
areas of research and the emergence of distributed (often interdisciplinary) research groupings. We would note
that the exploitation off knowledge developed within the UK and elsewhere in the world requires a wide-ranging
base of research of national excellence in addition to leading edge research. As it is clear that the funding
councils should be nurturing research of national standing either the RAE should be capable of identifying where
it takes place or a mechanism for its support that does not require an RAE should be in place. In a similar
manner the cost effective role of research in academic job satisfaction (and hence recruitment, retention and
motivation of staff) and in broadly based knowledge transfer should also be recognised in any funding
mechanism for research RAE.
Excellence should be judged by fitness for purpose against both international norms and these broader aims for
research.
d. Should research assessment determine the proportion of funding directed towards each
subject?
No. Research assessments should inform such decisions but the level of support for different research areas
should be strategic need. An important caveat here is that research areas should be very broadly drawn (e.g.
bio-sciences, information technology) so that micro management is avoided and the allocation of funding is still
appropriate to the needs of dual-support. We would argue against allocations based purely on historic, external
or volume measures.
e. Should each institution be assessed in the same way?
This is an odd question. We assume that assessment is of the quality of research (we would argue for the
quality of research of individual researchers) not institutions. One possibility is to assess different research in
different ways in which case, de facto, institutions would be assessed differently dependent on the nature of the
research they submitted. All assessment should be consistent, fair and on an equal basis for comparable
research.
f.
Should each cognate area be assessed in the same way?
Yes.
g. Discretion.
Institutions should be free to opt in or out of assessment as they see fit. As responsibility for funding is currently
devolved to institutions it is difficult to envisage a system that would not devolve responsibility for submission to
institutions other than a system that allowed no discretion. For example a reasonable system might well be to
assess all individuals and allocate funding on an individual basis, aggregated to subject or institutional level.
This would avoid many game-playing, equal opportunity and ratings issues.
h. Equality of Opportunity.
This is a vital issue that the current RAE fails fundamentally to address. The current system enforces a narrow,
blind and unhelpful view of research excellence that is predicated on the research career progression of a very
particular type of individual. Any system that does not broaden both the notion of research and criteria for
excellence will inevitably systematically disadvantage a number of different groups within universities.
i.
Priorities.
Not burdensome.
Rigorous.
Transparent.
Expert Review
We have already noted our concerns over the use of peers. We welcome the adoption of the term expert review
and support the view that such a panel, properly constituted could respond to the questions of excellence in
terms of purpose and use, and consequently view research quality more rationally, usefully and accurately.
We would argue that the assessment should combine the past, based on evidence of output and achievement,
with plans against which expected performance can be evaluated.
The data used will reflect the purpose of research support that we have discussed above. Criteria should be
output rather than input based and hence concentrated on achievement, contracts delivered, knowledge
transferred and publications written.
We acknowledge that the significance of the level of assessment might vary depending on the discipline, lone
scholar to research team. We are however emphatically opposed to the assessment being at the level of
Institution.
We would support, subject to further work, assessment of individuals.
We have noted above our views on subject level assessment.
Expert review is well understood and predictable, but well known to be subject to the bias of self-similarity (“good
researchers tend to be like me”). The range of reviewers thus needs to reflect the reasons why research is
being supported and not just included those who are selected on the basis of problematic judgements of
academic excellence.
Algorithm
We would not support the sole use of algorithms in the assessment of research.
Such an approach would almost certainly result in the distortion of research activity to fit the algorithm used.
The use of metrics, where available and appropriate, in conjunction with other methods is reasonable. We
would argue strongly that if research income is to be used as a metric that it should only be used alongside a
value for public money metric.
Self Assessment
Self-assessment has positive attributes, as it would permit institutions to adopt their own approach and level. Its
major strength lies in the value of the process itself which would include strategy development, monitoring and
review. Many institutions currently undertake self-assessment as part of their normal research monitoring
activities. It would however be subject to several of the problems associated with criteria, validity and verification,
and the overall validation process, with its associated difficulties of expert and peer review, alluded to above.
Fundamentally however, we feel that it is, taken alone, an inappropriate approach in circumstances where it is
the sole determinant of a system which allocates substantial funding.
Historical Ratings
We strongly oppose the adoption of an historical approach. It is basically flawed as it takes no account of the
current quality of research, its strategic value or timely responses to changes in the distribution of research
excellence at any level.
Any assessment system should be responsive to change and there is little or no evidence to suggest that the
cost benefits of a static assessment system would be outweighed by the disbenefits of allocating funding in the
wrong places due to the inevitable time lags caused by such a system. There is no evidence that quality of
research or strategic need changes at such a slow pace that such a system could be justified even when
modified by use in conjunction with other systems.