THES article I UoA 21 & the RAE
The RAE – Environmentally Unfriendly?
Just before Christmas, the Research Assessment Exercise (RAE) results were released. Overall, this
five-yearly nation-wide assessment of the strength of research in UK universities revealed the best
results yet. For Environmental Sciences however, in the main it was very bad news indeed. It is
resulting in the closure of schools, and appears to be in the process of shutting the subject down in
most, except the very few, large universities. There are bigger underlying issues here about how
universities are funded, and whether the model applied to Higher Education from the 1980s until the
present day is appropriate or in the national interest. However, this article will focus on this quiet
crisis which is going on in UK higher education only inasmuch as the RAE is part of it.
The Exercise
Under the RAE, 59 peer review subject panels were set up to review 69 subject areas (Units of
Assessment or UoAs), and these considered in total 2597 submissions from university schools,
departments and research groups. Each of these submissions was graded from 1 to 5* (see Table I)
where 5* is internationally excellent, and 1 is virtually no national excellence. The procedures that
would be followed were published in advance, and form the basis on which the exercise was
conducted. The immediate outcomes of the exercise are threefold:
(1) each submission has a rating, and hence (it is assumed) an indication of research strength and quality.
(2) most of the major research funders (e.g. The UK Research Councils) use these ratings (either directly or
indirectly) as an arbiter of whether and to what extent they will fund research or research proposals from a
given school/department/university. The degree to which subjects are funded also becomes an issue when an
entire area comes out as Environmental Sciences has.
(3) there is immediately an easy way to categorise departments and universities by the use of such scores, hence
many of the national newspapers produce league tables of universities.
The RAE looks at research outputs over a five-year period, so RAE2001 looked at research from
1996-2001. The 2001 exercise is the third national exercise, and followed previous exercises in
1996 and 1992. Although the UK Research Councils advised that RAE2001 should be the last RAE,
in their own words they reported1 that “..it [the RAE] was having the effect of stifling rather than
promoting research..”, the vice chancellors of the universities (“Universities UK”) lobbied strongly
for its retention. (“Universities UK” is the former Committee of Vice Chancellors & Principals
(CVCP) augmented by those of the new universities). It is thought that they did this because the ex
CVCP institutions were loathe to lose a system of research funding over which they [or their
universities] had large amounts of control. Even a casual inspection of research funding of UK
universities shows that these original institutions receive the overwhelming share of funds, and that
the RAE has strengthened that position further.
The main strengths2 (or the “jewels in the crown”) of the RAE are:
(1) it promises comparability, consistency, continuity and credibility across the spectrum of UK research, so that a
5* in anthropology or history, should be comparable to a 5* in chemistry or environmental sciences.
(2) it is supposed to be an “absolute” rather than a “relative” process, the grading that a submission receives is not
supposed to depend on that received by any other, nor on the identity of the department making the submission
There are also potential weaknesses. Unless the quality control procedures exercised by the
organisation running the RAE are rigorous, both of the above strengths can be lost, and the results
1
Reference from THES in office box
Paraphrased and summarised from RAE2001: Assessment panels’ criteria and working methods. RAE Team
publication
2
1
THES article I UoA 21 & the RAE
of the whole exercise compromised to a greater or lesser degree. As both the health of UK science,
and (literally) people’s livelihoods can depend on the outcomes of this exercise, it is incumbent on
HEFCE3. to take great care in how the RAE is set up and run. It is a situation where things must
both be done, and be seen to be done. For these reasons, reported problems in any aspect of the
running of this exercise should be the subject of rigorous investigation, review and if necessary
modification. It is perhaps important to stress that the RAE as a salient peer review system must
also be seen to work well. All UK university scientists are very dependent on peer review processes
like the RAE. We are part of these processes as both reviewers and the reviewed. Our papers are
published by peer review, our grants gained by peer review, in fact much of any individual’s future
is determined by peer review. It is important then, that there is confidence in these processes. If the
RAE is unsatisfactory it can bring the whole peer review edifice into question.
History
After each of the previous RAEs there has been serious controversy over one particular subject Environmental Sciences (UoA21). In RAE1996 it was judged by a joint panel with UoA20 Earth
Sciences. After that exercise there was a mass meeting of the subject area (under the auspices of
CHES – The Committee of Heads of Environmental Sciences) at which HEFCE effectively agreed
to change the panel status (from joint to single subject) and hence address issues around the
composition of the panel. In RAE2001 it again was judged in a joint panel with UoA20 (although
two members, including the chair, were eminent environmental scientists). This time, UoA21 came
out by almost any parameter, as apparently the weakest of all UoAs in the exercise. So weak in fact,
that the average grade for this UoA is below the threshold for government funding. With the
possible exception of Nursing, this must be the only UoA to effectively close down its own subject
area. Although not strictly correct, if the numerical codes in Table I are used for RAE grades, the
average for UoA21 comes out at 3.26 – this compares to averages for the whole exercise of 4.10, or
from UoA20 of 4.38. (On average Environmental Sciences was more than 2 RAE grades lower than
that for the whole exercise).
As Higher Education in the UK (or at least sectors of it) has been significantly under-funded over
recent years, the funding that comes with the RAE has become a major component of the operating
plans of many Schools and Universities. This is especially true for the new universities which were
floated on a single day in 1992 with no reserves, just their collateral assets (buildings etc.). Hence,
although the UoA21 results have had major impacts in many institutions with Environmental
Sciences groupings being dismantled and in some cases staff being made redundant, these effects
are enhanced in the new universities.
Quantitative Analysis of the results
As the implications of the results of UoA 21 are so great for the RAE generally, and this area of UK
science particularly, some scrutiny of the results is warranted. Moreover, it seems appropriate that
as a judgement is being made about a science area, the tools of science should be used to prepare
that judgement. Accordingly, below a statistical approach is presented which establishes that the
results of UoA21 are different to those from the rest of the RAE and UoA20, and a logic tree is then
used (see Figure 2) to investigate reasons for the differences. The sample of university
schools/departments submitted to any UoA are self selected, and universities chose how to “carve
up” their research effort to this exercise. These choices were only revealed when the submissions
were in. There has been a suggestion that the sample of institutions which comprised UoA21 in this
exercise does not well represent the actual sample which is UK Environmental Sciences. It is true
3
The exercise is the joint responsibility of the national funding bodies - in England it is the Higher Education Funding
Council of England – HEFCE, in Scotland and Wales, SHEFC and HEFCW respectively, and in Northern Ireland it is
the Department of Higher & Further Education, Training, & Employment for Northern Ireland. In this article this group
will simply be referred to as HEFCE
2
THES article I UoA 21 & the RAE
that one of the major players (Lancaster) chose to submit elsewhere, and some other institutions did
likewise, but in reality, by far the majority of Environmental Science schools were represented4.
The results of UoA21 are anomalous
In any collection of different numbers (including RAE grades) one number has to be smaller than
the rest. The fact that UoA21 is lowest in the overall ranking of all UoAs is therefore, in itself, not
necessarily a problem. What is a major problem however, is that both in terms of the distributions
and medians, (see Figure I) UoA21 is significantly different from both the results of the rest of the
exercise, and from those of UoA20 – Earth Sciences, (which was dealt with by the same panel). It
is important to realise what this implies - the chances of the UoA21 results coming from the body of
the RAE results as a chance outlier is ~ 1 in 75,000. This result is important for HEFCE and
everybody else. That this difference exists at all points to a loss of one or both of the main strengths
of the exercise.
What if the results are “correct”?
Figure 2a shows a logic tree for analysing the reasons for the difference between UoA21 and the
rest of the RAE. Logically, this highly significant difference could reflect either a real difference
between UoA21 and the rest of the RAE, or could be something which is nothing to do with the
samples themselves. For now, we will restrict ourselves to the first option. If in addition the
difference above is not due to some sort of artefact in the samples, (formally a sample limitation),
then there are only two options to explain these results:
(a) either Environmental Sciences in the UK is indeed the weakest of all subjects – but this does not address the fact
that the data is so different from that from the rest of the RAE, nor that outcomes (grades) given to most of the
UoA21 institutions cannot be reconciled to the definitions of those grades. Whilst the basic HEFCE definitions of
grades (see Table 1) are so inconsistent with the situation “on the ground”, it is extremely difficult to interpret the
outcomes as credible
(b) or, paradoxically, the UoA21 panel is the only panel which worked correctly. Figure 3 shows the grades from the
whole exercise. The distribution displays considerable skewness (distortion toward the higher grades). This is in
itself problematic, and should ring alarm bells. It has been speculated that this shows how much UK academia has
improved in terms of its research. However, it is very difficult to derive such a skewed distribution by that
mechanism. It is more likely to indicate that the (overwhelmingly single) subject panels “talked themselves up” –
i.e. came out with higher grades across the board for their subject areas. The lack of effective quality assurance
procedures is the most likely cause. Although unpalatable to celebrated UK academics, the shape of this
distribution is potentially good evidence that HEFCE has not taken sufficient care to ensure comparability across
the range of subject coverage, and hence has compromised both the main aims of the RAE.
Could the difference be due to a sampling problem?
We could alternatively assume that the difference between UoA21 and the rest of the exercise is due
to a limitation in the samples. Figure 2a lists a number of possible artefacts gleaned from
conversations with Heads of UoA20 and 21 submissions, as well as others in the area.
Chance has already been quantified as negligible above (see Figure 1). Many expressed the view
that UoA21 was either harder or easier than UoA20. However, this is logically inconsistent either
way. It seems clear that provided we do not subscribe to conspiracy theory (that some knew in
advance that UoA21 would be harder or UoA20 easier) then all would have favoured the easier
option.
One factor, (it is widely held), underlying the difference between the results of UoA21 and those of
the rest of the exercise or UoA20 is that UoA21 was primarily composed of new universities.
Hence, because these have not had the sustained high density funding that is necessary to initiate
4
There are about 45 undergraduate Environmental Sciences type courses in the UK, and possibly 90% have a
substantial research base. Of these maybe ~15% did not submit into UoA21.
3
THES article I UoA 21 & the RAE
and sustain a powerful research base, then UoA21 should be much weaker than all the others. It is a
beguiling idea, and one which at first glance is obviously correct and generally applicable across the
RAE, see Figure 4, (Table 3 contains the definition of the university age or “Type” variable).
However, were this idea the underlying reason for this particular difference, one would expect to
see all UoAs with a similar new university composition to that of UoA21 with similar outcomes.
Such a comparison (see Figure 5) shows significant difference between these six other UoAs5 and
UoA21. This effectively rules out sample composition (in terms of new universities) as the primary
reason for the difference between UoA21 and the rest of the exercise.
Size of department was the other factor that many people from the Earth and Environmental
Science communities cited as crucial. This is alarming because if size were such an important
factor, this contradicts specific advice from both from HEFCE6 “..There is no reason, in principal,
why a submission from a small group, or comprising small groups, cannot obtain a high rating…”,
and also informal advice from the panel prior to the submission. Several Earth Sciences colleagues
pointed to the Oxburgh Review of Earth Sciences which resulted in the closure of many Earth
Science departments to form a few super-departments. They posed a question like “Did you not
realise that this will happen to Environment as well?” Interestingly, most scientists now realise that
“Big is not Beautiful” when it comes to creativity in research, but still many academics and civil
servants who are involved at the core of the UK research funding scene are bedazzled by this idea.
Although that group of people is relatively small, they have a very powerful hold on the purse
strings and pop up in the RAE, Research Council peer-review committees, anywhere in fact where
there is money or influence. Even a cursory look at grants awarded from The Natural
Environmental Research Council reveals that most of the grants go to members (past or present) of
their own committee structure. This is not a conspiracy – just the recognition of a culture. Where
there is under-funding of a research area, the competition for the scarce funds accelerates and this
culture springs up. Once the main players are identified, then an equilibrium situation will result
which is difficult to break, e.g. there is an annual meeting between the Head of NERC and the heads
of what he consider the chief “client” departments. There has even been a word developed for this
type of culture – “grantsmanship”. So it should not be surprising that “Big is Beautiful” should
prevail as an attitude within this part of the UK Science community. Figure 6 shows that size of
submission is a strong indicator of grade across the whole exercise – but it is steeper in the science
area, and steepest of all in UoA21. Whilst in certain individual cases very small submissions did
obtain high grades7, overall the picture is clear.
Whilst this type of analysis may be interesting, it is simply trying to re-invent whichever wheel the
UoA21 panel themselves used. As it is unlikely we will ever know what they actually did, there is
limited use in pursuing this further with any of the other HEFCE data. The immediate conclusion
for this analysis is that despite their advice and RAE guidance to the contrary, clearly size was an
important factor, and that this is so in UoA21 more than in any other UoA.
This leaves us with only two conclusions from the assumption the significant difference between
the UoA21 and the rest of the RAE is due to a real difference in the samples. They are:
a Size of submission gives some insight into UoA21 panel operation, and was certainly more important for this
panel than for others
b..New university composition of submissions is not the reason for oddity of UoA21 results
5
UoA26 General Engineering was not different, it was also a joint panel, see Figure 5.
verbatim from RAE2001: Assessment panels’ criteria and working methods. RAE Team publication
7
The submission from Exeter University contained just one person and obtained a grade of 4.
6
4
THES article I UoA 21 & the RAE
These conclusions seem rather interesting, because it seems none of the widely thought
explanations for the anomalous UoA21 results hold up in the cold light of day. That they are
anomalous is beyond doubt, and is in itself evidence that the Quality Assurance systems put in place
by HEFCE have failed. Alongside the indicative evidence from the distribution of results for the
whole exercise, it is potentially very bad news, and it should be acknowledged as such by
academics and HEFCE alike.
A peer review process that went astray?
However, in the end, it is a peer review process that has made a judgement. To challenge such a
process is difficult unless there is some other independent evidence that supports the statistical
analysis of the HEFCE data. Happily, in this case, such data exists. Our peers (on the panels) who
reviewed UK academia in the RAE are my colleagues in this and other UK universities. Those
universities that chose to submit into UoA21 did so alongside their other submissions into other
UoAs. The required data comes from the predictions and results to all UoAs of those universities
which submitted into UoA21.
If the predicted grades across all UoAs could be obtained for those institutions who made
submissions into UoA21, the difference in predicted and actual grades could then be tested for
UoA21 vs the rest. I have obtained some of this data. There are very stringent confidentiality
conditions attached to the raw data (Public Relations considerations of the donor universities), and
hence it appears in condensed form in Table 2. RAE grades were coded from 1-7, where 1 = RAE
grade 1, and 7 = RAE grade 5* and it is the differences between predicted and actual grades that are
presented. So a difference of 0 means an accurate prediction, -1 means optimistic by 1 grade, etc.
From the condensed data it seems that this sample of universities were mildly optimistic in their
predictions of UoA grades (not including UoA21) by ~0.2 grades, whereas for UoA21, this
optimism swells to ~ 1.2 grades. The accuracy of the predictions of UoA21 and the other UoAs are
different with a confidence of 99.95%. (The standard deviation data from the two samples also
indicates a similar story with respect to accuracy of predictions). This data at last begins to shed
some light onto the conundrum which is the UoA21 results – could it be that the members of the
panel were themselves not representative of those they were peer-reviewing? It is an uncomfortable
fact that the institutions of only 2 members of this panel actually submitted into UoA21, with at
least one of the significant institutions in the sector eschewing it. This hardly makes for
“ownership” of the subject area. It is also a fact that most of the panel members were earth
scientists. It seems to me that by definition peer review must include “peers” on both sides of the
fence – the evidence here is that HEFCE have failed in this fundamental underpinning of the RAE
in this area.
Did the UoA21 panel malfunction?
There is one other area where we should probably go, and as the data is leading there it seems
reasonable. Figure 2b contains the logic tree for the assumption that there is no genuine difference
between the samples. This does not imply any sort of conspiracy or lack of integrity by any panel
member per se., Instead, it is an exploration of the role of the “culture” in the decisions that have
been taken. However, it is very bad news for the peer review process that is the RAE in this area.
There are three possibilities for difference in grades which are not to do with differences in research
quality between the samples. The chance of the UoA21 result being a genuine outlier have already
been assessed (see above) as negligible. Equally the possibility of some architectural feature of the
RAE exercise which somehow discriminates against UoA21 seems remote. The joint panel issue is
probably the only relevant point here, and this has been alluded to above in terms of subject
5
THES article I UoA 21 & the RAE
ownership. It is difficult to formulate any quantitative statistical test which might shed light on this.
Which means in short order we come back to some aspect of the function of the UoA21 panel.
Of the remaining possibilities, subject bias has been alluded to above, and may have played a part.
The process adopted by this panel may have been different from that adopted by other panels –
certainly we are aware that this panel sent out a great deal of material – maybe more than any other
in the RAE - for advice. (Understandable, given the subject range of the submissions and panel
expertise). There is opinion that this process alone would give depressed assessments because those
receiving the material will have been judging full submissions in their own areas, and hence might
unconsciously compare the UoA21 “bit” submissions as of lower quality that the “full” submissions
from their home panel. Inappropriate criteria may have been applied, or appropriate criteria applied
too harshly. I certainly have no information on what went on in the panel to express an opinion one
way or the other. Finally, personal or political bias. I personally have confidence in the integrity of
those panel members I know, but also acknowledge that bias need not be conscious, and think the
discussion on “culture” above is relevant.
The final issue here is again one of HEFCE (and the UoA21 panel) appearing to breach their own
procedures. There is of course the general Quality Assurance failure apparent by the significance
level of the difference between UoA21 and the rest, as well as what appears to be the failure in
UoA21 to provide what is normally thought of as peer review. However in an interesting twist
HEFCE seem to have indicated in a formal response to a request for information under the Data
Protection Act that in UoA21 they did not follow their published procedure in assessment of
submissions. These procedures state8 “..The work [my emphasis] of each researcher will be
individually graded…..and these grades will form the principal basis for the determination of the
overall grades for each of submissions specified research groups..”. To do this UoA21 sent out
forms to reviewers to assess each of the individuals submitted (interestingly, not only their papers,
but the researchers themselves). This then leaves a paper trail to enable panels to meet the Quality
Assurance requirement that2 “..Panels must be able to show in all cases how their quality
judgements relate to the evidence before them and to the criteria which they have stated they will
apply..”. Without this paper trail it could be said that the primary basis of the RAE gradings were
“discussions around a table”. However, in their own words on this very point, HEFCE have stated
“Neither the funding councils nor the panels retain any of the panel’s comments on the work of
individual researchers”. Yet without these written documents (which we know existed, because
some of us had to fill them out), the UoA21 panel cannot meet their own Quality Assurance
requirement (above). We also now learn that institutions were compared to assist in the
determination of final grade – this makes the RAE in UoA21 a relative rather than absolute
exercise, which in itself breaches yet another of the underpinning principals of the exercise. (One
wonders how many structural underpinnings need to be breached before the entire edifice falls.)
The aftermath and way ahead
In the aftermath of the exercise, schools struggle to reconcile the grades they have with the
definitions of those grades. The feedback which at the outset was promised to be “..summarising the
reason for the rating awarded, with reference to the published criteria..” has been delayed and is still
awaited – but we hear that it will not be particularly useful to help us find out how (in the main) our
submissions did so badly. Many institutions are in the position of having “failed” in UoA21 after
major investment, and are not able to find out exactly why they have failed, or more importantly
how to correct it for next time. Meanwhile the sector shrinks as schools close and institutions
redirect scarce resources elsewhere. There is no appeal on the RAE, unless a member of the
judiciary agrees with the thrust of this article and overturns the results of UoA21 in the courts. I
believe that the arguments (statistical and otherwise) for doing so are compelling.
8
Guidelines to submissions in UoA21, RAE Team
6
THES article I UoA 21 & the RAE
What though for next time? Although there are many suggestions that could be made, it seems to
me there are some clear lessons that stand out from the rest.

However the peer review system for the next RAE works, the peer reviewers must be representative peers of
those they are reviewing, which minimally means that their own institutions should be in the group they are
reviewing.

The goal-posts must be declared before the exercise runs, and there should be real transparency before, rather
than artificial visibility afterward. Prior declared procedures and principals should be inviolate.

HEFCE must institute effective quality control on all aspects of the RAE process from definitions of the
subjects, through to timing and nature of the feedback. There seems to be strong evidence that HEFCE quality
assurance procedures have to some degree failed, and it is alarming how there seem so many structural routes
left open by which results of panels can be tainted. Joint panels are a potential weakness.

Apart from the under-funding of this UK science sector, it seems to me that there are urgent issues for
government in the funding of universities. It cannot be in the national interest that powerful committees like the
Universities UK can overturn a recommendation by the Research Councils on the future of university research
funding which seem to be motivated by greed. It is also of concern that the lack of funding in this area is
encouraging the domination of the research funding nexus by relatively few universities, effectively preventing
the entry of other universities into the funding stream.

There is clear evidence that the results from UoA 21 are flawed (just by their difference) and that is a serious
issue for large parts of the Environmental Science sector in the UK
Acknowledgements
I would like to thank the large number of colleagues in the Earth and Environmental Science
community who have taken the time to talk to me, and pass sensitive institutional data to me.
Especial thanks go to my Colleagues in the School, especially Professor Vic Truesdale, Drs Andrew
Rendell, Peter Grebenik and Tim Shreeve. Finally I also thank the various statisticians both here at
Brookes and elsewhere who exercised patience with my questions, and Dr Nicola Madge who spent
time making me think a little more like a social scientist!!
7
THES article I UoA 21 & the RAE
LIST OF TABLES, FIGURES AND LEGENDS
Table 1 Definitions of the RAE grades
Table 2 Difference between predicted and actual grades for all UoAs from those
universities† who submitted to UoA21.
† Not all of the 34 institutions gave any or all of this data
Table 3 Codes used to define “Type” variable for university age
Figure 1 Results of Kolmogorov-Smirnov (distribution) and Mann-Witney-U
(medians) tests for difference between the results of UoA21 and the rest of the RAE
results, and the UoA20 results.
Figure 2(a) Logic Tree for assumption that there is a difference in the quality of
research between UoA21 and the rest of the RAE.
Figure 2(b) Logic Tree for assumption that there is no difference in the quality of
research between UoA21 and the rest of the RAE.
Figure 3 The distribution of grades‡ from the RAE2001.
‡ For codes used see Table 1.
Figure 4 The effect of age of university on RAE outcome§
§ For list of codes see Table 3. For this analysis non-teaching Research Institutes, Charities, non-university colleges or special
projects (e.g. University of Highlands & Islands project) are not included.
Figure 5 (top) a plot of the average “Type variable” for each UoA. (bottom) Results
of Kolmogorov-Smirnov tests for difference between each of those areas with a Type
variable of 1.78 or less with UoA21 (Type variable = 1.54)
Figure 6 The affect of the size of the submission on RAE outcome#
# The definition of “science” has been taken to include: UoAs 1,6-9, 13-21, 25-32.
8
THES article I UoA 21 & the RAE
Table 1 Definitions of the RAE grades
RAE grade
5*
5
4
3a
3b
2
1
Number
5.5
5
4
3.5
3
2
1
Definition (paraphrased)
International Excellence in > 50% of the work
International Excellence in up to 50% of the work
National Excellence in ~ 100% of the work
National Excellence in > 66% of the work
National Excellence in > 50% of the work
National Excellence in up to 50% of the work
National Excellence in virtually none of the work
Table 2 Difference between predicted and actual grades for all UoAs from those
universities† who submitted to UoA21.
† Not all of the 34 institutions gave any or all of this data
NULL: No difference between (actual RAE grade-predicted RAE grade) for UoA
21 and (others-UoA21)
mean UoA21
-1.219
Data is in the form of (actual RAE gradepredicted RAE grade) for all UoAs that each of
mean (others-21)
-0.222
the UoA21 submitting institutions submitted
stdev UoA21
0.826
to. The distribution of these differences should
stdev (others-21)
0.071
be normal around a mean of zero. The test is
21 count
21
whether the differences for UoA21 are
(others-21) count
11
significantly different than those for the other
t coeff
3.963
UoAs. Result is that they are different at
t crit (99.95%)
3.646
99.95% confidence level.
Table 3 Codes used to define “Type” variable for university age
Age of Institution
(in more or less current form)
1990- (the “New” universities)
1960-1989 (the “Robbins” universities)
1900-1959 (the “Redbrick” universities)
-1899 (the “Old” universities)
Type variable
1
2
3
4
9
THES article I UoA 21 & the RAE
Figure 1 Results of Kolmogorov-Smirnov (distribution) and Mann-Witney-U
(medians) tests for difference between the results of UoA21 and the rest of the RAE
results, and the UoA20 results.
Results for UoA21 seem different from
those of other areas and RAE generally
Medians of UoA 21
different (99.999%
CL.) from rest of
RAE (~1 in 74,000).
Also different
(99.97% CL.) from
UoA20 (~1 in 3000).
Distributions of
UoA 21 different
(99.98% CL
.)
from UoA20, rest of
RAE (~1 in 5000)
Kolmogorov-Smirnov
Mann-Witney-U
YES
10
THES article I UoA 21 & the RAE
Figure 2(a) Figure 2(a) Logic Tree for assumption that there is a difference in the
quality of research between UoA21 and the rest of the RAE.
Results for UoA21 are different from
those of other areas in RAE

NO
YES
Difference does not
reflect genuine
difference in samples
Difference reflects
genuine difference in
samples
Sampling
limitation
Genuine
Difference
Chance
21 Easy
Newer
1 in 74,000
logically fails
21 still different
Other
size
11
In UK,
Research in
Environment
al Sciences of
lowest
quality.
Environment
al Sciences
only panel to
function
correctly RAE flawed
THES article I UoA 21 & the RAE
Figure 2(b) Logic Tree for assumption that there is no difference in the quality of
research between UoA21 and the rest of the RAE.
Results for UoA21 are different from
those of other areas in RAE

NO
YES
Difference does not
reflect a genuine
difference in samples
Difference reflects
genuine difference in
samples
UoA 21 Panel
Chance
0.001% likely
RAE process
overall
Criteria
Criteria
Process
People
Subjects
Appropriate, but
harsh
Inappropriate
Inappropriate
Personal or
political bias
Subject Bias
12
Joint
Panels
THES article I UoA 21 & the RAE
Figure 3 The distribution of grades‡ from the RAE2001.
‡ For codes used see Table 1.
Results of RAE 2001
Fr
eq
ue
nc
y
800
600
400
200
0
1
2
3
3.5
4
RAE grade
13
5
5.5
More
THES article I UoA 21 & the RAE
Figure 4 The effect of age of university on RAE outcome§
§ For list of codes see Table 3. For this analysis non-teaching Research Institutes, Charities, non-university colleges or special
projects (e.g. University of Highlands & Islands project) are not included.
3000
Av
ran 2000
k
1000
4.00
Av 3.00
typ
e 2.00
1.00
0
0.00
0
1
2
3
4
Average type
0
2
4
RAE grade
14
6
THES article I UoA 21 & the RAE
Figure 5 (top) a plot of the average “Type variable” for each UoA. (bottom) Results
of Kolmogorov-Smirnov tests for difference between each of those areas with a Type
variable of 1.78 or less with UoA21 (Type variable = 1.54)
6.00
av.
typ 4.00
e
2.00
0.00
0
20
40
60
80
UoA
UoA
Name
11
Other Studies & Professions
allied to medicine
Built Environment
Library & Information Management
Art & Design
Comm, Culture & Media Studies
33
61
64
65
15
Results distribution
different from that
of UoA21
CL 80%
CL 95%
CL 95%
CL 99%
CL 80%
THES article I UoA 21 & the RAE
Figure 6 The affect of the size of the submission on RAE outcome#
# The definition of “science” has been taken to include: UoAs 1,6-9, 13-21, 25-32.
Av 50.00
era 40.00
ge
# 30.00
sta 20.00
ff
10.00
Whole Exercise
Science
UoA21
0.00
0
4
2
RAE grade
16
6