ESRC Teaching and Learning Research Programme
Research Capacity Building Network
The RCBN Consultation Exercise:
Stakeholder Report
Chris Taylor
taylorcm@cardiff.ac.uk
Cardiff University School of Social Sciences
August 2002
Occasional Paper Series
Paper 50
ESRC TLRP Research Capacity Building Network
Cardiff University School of Social Sciences
Glamorgan Building
King Edward VII Avenue
Cardiff CF10 3WT
Tel. 029 2087 5345
www.cardiff.ac.uk/socsi/capacity
Fax. 029 2087 4678
CONTENTS
1.
Introduction _____________________________________________________1
2.
The state of educational research ____________________________________3
2.1
2.2
2.3
2.4
3.
Constraining the capacity for high quality, relevant, educational research __13
3.1
3.2
3.3
3.4
3.5
3.6
4.
Response to recent critiques__________________________________________________3
The quality of educational research ____________________________________________5
The impact and relevance of educational research________________________________10
Conclusion ______________________________________________________________11
Introduction _____________________________________________________________13
The missing impact of educational research ____________________________________13
Constraining advances in educational research __________________________________16
Methodological limitations of educational research ______________________________19
The nature and composition of the education research community ___________________28
Conclusions _____________________________________________________________31
Building educational research capacity_______________________________32
4.1 Introduction _____________________________________________________________32
4.2 Research ‘capacity-building’ ________________________________________________33
4.3 Improving relevance and impact of educational research __________________________38
4.4 Knowledge generation in education research____________________________________41
4.5 The organisation and composition of the educational research community, its academic
departments, and research teams__________________________________________________44
4.6 Building individual research capacity _________________________________________55
4.7 Research training and continued professional development ________________________61
4.8 Conclusions _____________________________________________________________66
5.
Summary and conclusions _________________________________________68
6.
References ______________________________________________________71
1. Introduction
1. Introduction
One of the early objectives of the ESRC Teaching and Learning Research Programme
(TLRP) Research Capacity Building Network has been to undertake an extensive
consultation exercise in order to identify the priorities for research capacity-building
and to generate a database of expertise from across the UK educational research
community. Our approach to this consultation exercise has been no different to other
social science research. It has involved four main elements. The first was to interview
key stakeholders from across the education field, from policy-makers and
practitioners to funders and researchers. The second part of the consultation exercise
has been to survey, as far as possible, the UK educational research community in
order to identify current expertise in research and future research training needs. The
third element is to undertake a review of the ‘best’ education research literature, as
determined by publications returned to the 2001 Research Assessment Exercise
(RAE). The final element to our consultation exercise involves continuous discussion
with researchers and research projects with regard to their individual continued
professional development.
This report provides an account of the first element of the consultation exercise, the
stakeholder interviews. So far twenty-five key stakeholders, each representing the
major constituencies of the UK education community, have been interviewed about
their general perception of educational research in the UK. In particular, they were
asked about the current state of educational research in the UK, why it is like this, and
how educational research could continue to move forward.
The stakeholders were chosen to represent the major constituencies of the education
world, particularly in relation to teaching and learning policy-making, practice and
research. These included representatives from the following: national and local
government – comprising elected members, policy-makers, and researchers; research
funding agencies; education research organisations – such as the British Educational
Research Association (BERA), the National Foundation for Educational Research
(NFER), the National Educational Research Forum (NERF), the Office for Standards
in Education (OFSTED) and the ESRC Teaching and Learning Research Programme
(TLRP); HE educational researchers – including TLRP project leaders, non-TLRP
researchers, and Research Assessment Exercise (RAE) panel members; the teaching
profession – including the General Teaching Council and the Teacher Training
Agency; educational research journal editors; and stakeholders with an international
perspective. It is worth noting that these stakeholders did not include representatives
from the ultimate user community, parents, students and other learners. They would
have offered a very different perspective based upon the immediate needs of the
education system. Although their perspective might be useful it would be difficult,
thereby, to get a broad overview of education research, both nationally and over time.
Many of the discussions with stakeholders focussed upon qualitative and quantitative
methodological issues, including the relationship between the two. Such discussions
began in response to the high profile given to the limited use of quantitative research
across the entire social science research field. Because of this it may be worth
outlining the methodological backgrounds of the research-active stakeholders
themselves: two have built a reputation for their quantitative research – one in
education and one in sociology; possibly three or four of the stakeholders would see
1
1. Introduction
themselves as having done both quantitative and qualitative research; while the rest of
the stakeholders, the vast majority, would consider themselves as qualitative
researchers. The balance in the methodological focus of the stakeholders closely
reflects the balance of the wider educational research community.
This report is divided into three sections. The first section outlines the stakeholders’
views on the current nature of education research and reflects on their perception of
its quality and relevance – two of the major areas of criticism from the recent past.
The second section examines the many factors that were raised during the stakeholder
interviews that they believe underlie the current state of education research in the UK.
They range from structural factors, relating to the organisation of educational research
over the last three decades, to the culture of the education research community, and,
in particular, the identities attached to qualitative and quantitative research
methodologies.
Irrespective of the stakeholders’ concerns regarding the current state of educational
research there was consensus in the belief that progress in educational research was
necessary. The final section draws together their views on the kind of issues that they
believe need to be addressed, and begins to outline the kind of activities that would
lead to building the capacity and impact of educational research in the UK.
2
2. The state of educational research
2. The state of educational research
2.1 Response to recent critiques
A discussion of the stakeholders’ perceptions of educational research cannot begin
without reference to the recent high-profile commentaries on its current state (for
example: Woodhead, 1998; Hillage, Pearson, Anderson and Tankin, 1998;
Hargreaves, 1997 and 1999; and Tooley and Darby, 1998). Many of the stakeholders
when asked about the current state of educational research began, without prompting,
by referring to and addressing these. This illustrates the importance of these apparent
‘critiques’ for the education field. They have clearly had a major impact upon these
stakeholders and prompted them, if they had not already done so, to reflect upon the
state of education research in relation to their own experiences.
Although published reaction to these commentaries has tended to be defensive of the
current state of education research (see for example, Hodkinson, 2001) the typical
response of stakeholders was actually to find some agreement in their criticisms. Even
when the stakeholders expressed particular concern for the motivation behind the
critiques some good is still seen to have come about from them,
‘I think a lot of that has been politically motivated and in some cases and in
many cases has been unfair [...] I mean all these things…there was no doubt in
my mind that there was a definite kind of, I'm not sure conspiracy is the right
word, but a concerted and co-ordinated effort to attack educational research and
educational researchers that in many cases was unjustified and I think a lot of the
current concerns around capacity building go back to those criticism and people
say "well ok we've got to strengthen educational research, we've got to make it
more relevant and so on". And maybe some good has come out of it.’ (HE
researcher and TLRP Team Leader).
There was often doubt expressed about the evidence base of the critiques. In
particular, Tooley and Darby’s (1998) review of articles published in four established
education journals was seen as a poor attempt at illustrating some of the key problems
with education research – although many of the criticisms are accepted,
‘The Tooley report was such bad research, that while he made a lot of points that
might be true it was not convincing.’ (HE researcher and RAE panel member).
The problem of a perceived lack of evidence for such critical claims was that the
‘genuine’ and useful arguments raised are getting lost or being ignored,
‘Trouble is, as far as I can see, David’s [Hargreaves] critique of educational
research, which is much more well founded than Tooley’s tends to get confused
up with Tooley’s. So in a response to one, the other got sucked into it
unfortunately. Tooley’s is hardly worth mentioning.’ (HE researcher and UCET
Executive Committee member);
‘So that…. I’ve said some of this…. it became very difficult to criticise
educational research because of the half-baked attacks being made on it.’ (HE
researcher).
3
2. The state of educational research
Indeed, while trying to avoid lining up with the many criticisms of educational
research, many stakeholders, particularly the researchers themselves, saw a lot of
validity in David Hargreaves’ concern over the relevance of this research to practice,
‘It always seems to me with David Hargreaves that he is extremely good at
getting things on the agenda, in order to do that he always makes a case,
nevertheless, there is always something in what he says.’ (HE researcher and
RAE panel member);
‘I think well certainly I and other people I know actually go along with a great
deal of what David Hargreaves says, namely that there should be at least two
kinds of education research. There’s nothing wrong with blue skies research but
it actually does move you towards the subject discipline, I mean the parent
discipline. So you end up taking your applied discipline back into the fold of the
parent discipline and the application, the people wanting the application are left
hungry as it were. So David’s suggestion that…well it was more than a
suggestion….the argument that educational research ought to be applied and
ought to have some kind of application in the classroom is something that I think
quite a few educational researchers would go along with.’ (HE researcher and
UCET Executive Committee member);
‘Sometimes you just look at it and it looks bizarre… I am starting to sound like
David Hargreaves. He had a point actually. The butt of jokes in education
departments, who think he is terribly crass to say this, but I think some of the
stuff that you look at is rubbish. I won’t name anyone, but I don’t see the
practical relevance of this and I am not saying that everything has to have a
practical relevance, but some stuff is just so beyond the place, so esoteric.’
(Education researcher and BERA Executive Council member).
Similar views, particularly from the policy-making domain, were expressed regarding
the criticisms, but not the conclusions, made by Chris Woodhead about the general
usefulness of research,
‘Well strangely enough I had some sympathy for some of the things Chris
Woodhead said about it, because a lot of educational research frankly doesn't tell
you a lot that you don't already know. Though sometimes you question whether
in fact it’s been done very thoroughly... I think the money we spend is not spent
wisely at all. I do share the criticism. I do not share the view that you do not need
educational research. I think its highly skilled work like anything else and its
essential if you're going to spend public money wisely for good effect.’ (Chair of
a LEA Education Committee and Local Government Association executive
member).
However, in respect to this particular criticism a number of the HE researchers were
keen to argue that policy-makers tend to take a naïve view of the importance of
education research in its contribution to what becomes ‘common knowledge’. This
particular point is discussed in more length later in this report.
The question, then, is to what extent should these criticisms be ignored or considered
as a significant contribution to any capacity-building exercise?
‘The educational research community…. they’d had a rather difficult choice on
its hands. It either had to bury its head in the sand and continue working as it
4
2. The state of educational research
was and just hope that these attacks would just go away or respond to them in
some way. I do remember when there were similar attacks made on initial
teacher education, the Hillage group and co. UCET at the time, this was before
the new universities… it was very much an old university system, an old
university club… UCET in a discussion about how to respond to these attacks,
finally agreed in a quite large meeting, actually, that because they were so
obviously ill-founded, they had no empirical base whatsoever, that if UCET
responded to them they would give them respectability. So they decided not to
respond.’ (HE researcher and UCET Executive Committee member).
While many of the arguments and criticisms raised in these high-profile commentaries
are accepted to some degree there is a considerable lack of evidence to justify their
claims. Consequently, the RCBN’s consultation exercise on the current state and
future needs of the education research community is timely. However, it is worth
noting the irony that Tooley and Darby’s commentary generated the greatest
opposition to its findings although it made the greatest attempt of those cited above to
base these upon some empirical evidence. Indeed, little attention is paid to the
examples of educational research that Tooley and Darby identified to celebrate
excellence.
2.2 The quality of educational research
When asked for their views on the current state of educational research, the majority
of stakeholders, from all perspectives, expressed concern regarding its general quality.
While a few of the stakeholders were keen to show educational research in a positive
light,
‘There is a lot of good education research.’ (HE researcher and TLRP Team
Leader),
or that there is great variety in its quality,
‘There is some very good research done and some very poor research done.’
(Education researcher and BERA Executive Council member);
‘I mean I have had for some time a concern about educational research that
because it comes from such a variety of institutions and individuals that it
probably has greater variation in quality than research in almost every other
area.’ (HE researcher and RAE panel member),
the majority of stakeholders tended to emphasise the poor quality of educational
research,
‘[there was a] a kind of hunch that there was a problem and in particular the
problem had been painted in lurid but not necessarily valid colours by various
critics. That research in the field was going nowhere, it's not accumulating a
body of knowledge, there were various deficiencies in those that conduct it and
so […] My own view was that there was hardly any research in contemporary
terms that helped us fill the gap between on the one hand people’s policies and
practices and on the other hand desired outcomes for attainment and learning
[…] There is a lot of stuff talking about policy, but it wasn't helping anyone.
5
2. The state of educational research
When you bring it down to the technical level very large amounts of it was
technically so bad as to be embarrassing. No discussion about validity and
reliability of data for example. No attempt to theorise.’ (HE researcher and
member of NERF)
‘I mean I to be honest have been surprised sometimes with what I see as the
relatively low level of sophistication in some of the research that’s done. I went
to a meeting recently, for example, about a project on homework and there was
an academic group who had been commissioned to do this. I started asking one
or two questions – “have you done this?”, “have you tried to join these 2 things
up?” and they hadn’t and it almost hadn’t occurred to them to do this’ (OFSTED
research manager);
‘Lots of low level cottage industry stuff. Too little understanding that it’s
actually skilled, time consuming business not to be engaged in lightly or
amateurishly. That sounds a bit hard but as somebody that edits 2 journals and
sits on 4 or 5 editorial boards I see a lot of the stuff which people see as their best
shot. If you’re submitting something for a journal you’ve really worked it over.
Some of the stuff that I see.…thank Christ somebody has brought this for
somebody else to look at never mind considered to be published.’ (HE
researcher);
‘I mean even the people who are doing the current EPPI reviews, the review
groups, keep coming back and saying how shocked they are by the poor quality
of the literature.’ (DfES Officer).
Such views of educational research have to be considered in their context. It is clearly
the case when discussing the future and capacity-building for educational research
that the majority of the stakeholders started from this rather pessimistic view.
However, this did not mean that they thought all education research is poor. Indeed,
from an international perspective UK educational research may be seen as relatively
good, although lessons could be learnt from a number of other countries, such as the
US and Sweden,
‘But as a sort of starting answer to that a general observation would be that
across the thirty countries of the OECD which covers the EU countries, the
central European countries, Hungary, Poland, Czech Republic, North America
including Mexico and Japan, Korea, Australia, New Zealand. Across those
countries the UK is pretty good in terms of it quality and the standing of its
educational research […] So there's a sense in which and I'll come back to the
things that I think are not…. where the UK doesn't perform so well. But there's a
sense in which what that highlights is perhaps not so much that things are fine in
the UK, but some of these phenomena are really general. That the general
standing, the organisation of education and the general standing of research
about education is something that is by no means peculiar to Britain.
Two countries that come to mind where you might say well there seems to be
more both… in Britain look for either higher quality or different way of
organising the research that seems to be already interesting… I don't think is
matched in this country, is the two countries would be the United States and
Sweden. In the United States, although arguably given all sorts of indicators
about the quality of education whether this results in tangible gains to learning
outcomes is quite another matter. But there are major research programmes
which link very directly to policy in the States and its clear as well that the sheer
amount and volume of educational research in the States is far higher than
6
2. The state of educational research
anywhere else… In Sweden, the organisation of a central school board which has
been very instrumental in data gathering and in commissioning research and
organising research as an offshoot of the Ministry with close connections to
schools and school districts is an example of where there seems to be a much
closer integration that has gone on for a very long time between research and the
worlds of policy and practice.’ (OECD researcher).
Similarly there was a general view that education research in the UK is getting better,
particularly in terms of its standing as an independent research discipline,
‘The state of educational research?. Well over the last 20 years it’s certainly
changed dramatically […] So the conception that was held then of education
research, this is about 20 years, was clearly of something that had to map onto, if
you like, main subject researchers […] It’s quite ironic because again at that time
without the RAE as a driver the only real leverage that one had to move you out
of one area of research into another was your own professional development,
your own CV. Now since that time of course, there have been all sorts of
changes and shifts and attacks on education and educational research. I think
they start partly with the educational research community itself, feeling, to my
mind somewhat embarrassed about the nature of its own work because most of
us have actually come from a main subject discipline – philosophy, sociology,
psychology. And we are aware what we’re doing now is actually different to
what was in the mains subject and indeed that’s often the reason they’ve come
out of the main subject, the parent subject […]So one the one hand then you have
subject specialist who moved into education at least in part because they want to
apply their subject being compared to those who are not applying their subject in
the universities. The RAE was a godsend in a way because I identified education
as a different unit of assessment. You’ve got a status and it can then be compared
with the parent disciplines […] Similarly with the QAA, policy and agencies.’
(HE researcher and UCET Executive Committee member).
In terms of the latest RAE there is a feeling that the state of educational research has
perhaps improved, although fewer submissions may make it appear this way,
‘We actually have a smaller number of submissions now. Partly that is because
people have fallen off the bottom and have decided not to bother, but there have
also been some mergers of institutions so there are not as many institutions
around as there used to be to do research. The best research was always very
good and the best research is still very good.’ (HE researcher and RAE panel
member).
However, whether this progress has kept up with the pace of progress in other
research disciplines is uncertain,
‘There have been signs of progress, but perhaps not as much as I suspect there,
but I don’t really know this, as I expect there has been in other areas. I think that
is to some extent because of the enormous extra demands. I don’t mean that
people in education work harder than people in other areas, it is a sort of
multiplicity of demands that seems to have grown through time and of course
there has been an extension of the population of people who are required to do
research.’ (HE researcher and RAE panel member).
7
2. The state of educational research
As this stakeholder argues, this may be due to the particular circumstances or context
in which a great many educational researchers have to work. This is a point that is
raised by a number of the stakeholders, and something that is discussed in more depth
later in this report.
The researchers’ working environment, and many other constraints, are frequently
seen as hindering the output of good quality educational research. For example, the
sheer size and complex nature of the education world makes relationships between the
many factions difficult to strengthen,
‘There’s also distrust isn’t there in the educational world about educational
research. Chris Woodhead said there’s a distrust of it and yet it must be one the
few areas of major public policy where there is that distrust. I mean I don’t think
the military use the weapons without research, do they? I’ve got a feeling that I
think we are well behind in our thinking of how valuable research should be. It
should one of the most valuable tools for education affecting everybody in the
country.
Interviewer: Where do you think that distrust comes from?
I think it comes from sloppiness from the educational world really. One thing
about the educational world is that it’s never had a discipline about money has it
and priorities. It has had a sort of assumption that academics or teachers know
best. It’s isolated….it’s one of the few communities in the educational world
which is so isolated from each other. Primary school teachers know nothing
about what goes on in the secondary school and vice versa and that’s an
example. And the FE world is a world on its own and the university worlds. In
fact what you should be doing is having an educational world where people are
moving across the sectors.’ (Chair of a LEA Education Committee and Local
Government Association executive member)
Consequently, the potential for improvement is there but in practice seems to be
lacking,
‘In some ways I think we’re, we’re in a healthy period, in principle, but in
practice we’re not right, realising the potential of that healthiness…. We’re in a
stage where we have got a more liberalised conception of educational research
methodology and there is more openness to a whole range of approaches […] So,
I think, in principle, we could have quite a rich, eclectic, you know, radically
triangulatory type of stance. I suspect we’re not quite yet there and there are still
people who plump for one thing or the other.’ (HE researcher).
As this stakeholder hints at, one of the key limitations may be in researchers’ limited
approach to research methods. A point shared by another stakeholder,
‘On one sense I think it’s quite healthy and that there’s a lot of activity and quite
a lot of variety in terms of themes but in terms of methods and the kinds of
things that people are trying to do, I suppose two points. One is it strikes me that
there isn’t much methodology innovation going on and... I suppose another
comment is that, yes, there is a lot of qualitative work and there is less
quantitative work which is in any way sophisticated, that is, there are lots of
people doing surveys but when you actually read the outputs from those, they are
8
2. The state of educational research
really not much more than people being able to push some buttons on SPSS.’
(HE researcher).
A number of stakeholders referred to the lack of research ‘skills’ among the research
community, for example,
‘Actually the situation is so bad that people don’t realise they haven’t got the
skills they need which is even worse than having people who are struggling but
aren’t sure where to get the skills they need. I say that because when we
interview teams here for research contracts, contracts which clearly demand
multi-level modelling, they might demand other things as well but they clearly
demand multi-level modelling and you get people happily presenting away about
ways of doing it which don’t include any of that kind of approach and they don’t
seem to realise there’s a gap. Even when you quiz them they don’t seem to
understand what you’re getting at.’ (DfES officer).
This leads directly to the issue of the current capacity to undertake good quality
educational research,
‘There aren’t the people in the system to provide the research that we want, we
know that.’ (DfES officer).
Others were more precise about this lack of methodological skills. In this example it is
the lack of methodological, and theoretical, rigour,
‘What I think the weakness of a lot of the proposals were and the reasons they
weren't funded was the kind of lack of theoretical rigour and methodological
rigour and I think that’s where educational research is weak.’ (HE researcher and
TLRP Team Leader).
It is at this point that many stakeholders start to differentiate between qualitative and
quantitative research methods and methodologies. There was a clear perception that
there is a lack of quantitative research, but the issue of quality and rigour was raised
equally for quantitative and qualitative research,
‘As a journal editor what worries me slightly is that you have so much qualitative
stuff. It’s very very unusual to get anything quantitative and there’s beginning to
be a pattern with qualitative papers. I’ve noticed it not only in my own journal,
but in my case… I don’t allow them to come through the journal, but in other
journals the stuff that I read… the pattern is a literature search, then without any
real critical analysis straight into what the dataset… how it was generated and
then a quick discussion and that’s it. And that's a kind of paper by numbers and
you see more and more of them. Now and of course no-one then goes back to
them and starts critiquing them because they are too busy doing the same thing
themselves.’ (HE researcher and UCET Executive Committee member).
Fuller explanations behind these widely held views about the quality of educational
research are discussed in the second part of this report. However, underlying all the
possible causes of poor research, or at least the constraints preventing higher quality
research, was the belief that research training and understanding, or lack of, played an
important role. This can be seen in the initial process of defining research questions,
9
2. The state of educational research
‘I think the problem with the poor quality end of your continuum is that very
often people try to do things that are not possible and it is inevitable that it is
going to be low quality stuff.’ (HE researcher and RAE panel member).
But more generally concern was raised regarding the form of research training and
learning that many educational researchers have been through,
‘It is certainly a range of some sort from very high quality social science
research, which sometimes has the problem of being irrelevant to the particular
issues that practitioners and perhaps policy makers have, through the kind of
research, which I like to see, which is probably kind of messier but nevertheless
is both as rigorous as we can make it but also has as a priority addressing
questions that are of important to practitioners and policy makers right through to
people who have come in to education departments rather than social science
departments and they have come in with all sorts of backgrounds. The common
thing is that they have had some experience of teaching in the compulsory period
of education or maybe pre-school or something like that but not with any social
science methodological or sociological background.’ (HE researcher and RAE
panel member)
2.3 The impact and relevance of educational research
One of the major criticisms of educational research has been about its relevance to
policy and practice. As a result this was an issue frequently raised by the stakeholders.
While many agree with the general criticism about the relevance of educational
research, both in policy-terms,
‘I think the quality of the policy-oriented research has been poor’ (OECD
researcher),
and in practice-terms,
‘There’s a lot of good things that must come out of that that could be very useful
to inform teachers. That would be probably my criticism. There’s not enough of
that that trickles down in a non-policy form to the practitioners. It maybe that
you work out its great to get kids to learn their tables and not much good to get
them writing essays, they’re better off playing in the garden with their brothers
and sisters or whatever. That may not be enough in those sorts of answers to
provide a policy but it still could inform teachers about what’s worth doing.’
(OFSTED research manager),
the general feeling was that this is a system-wide issue and not something that could
be addressed solely by the researchers themselves,
‘I think there's a great deal to be done in terms of improving the quality of that
research, but I don't think the responsibility lies with…again things we'll
probably talk more about now…I don't think the responsibility lies entirely or
perhaps even mainly in the fact that researchers have not been imaginative
enough or good enough to rise to the challenge. I think the whole organisation of
educational research has been largely outside a policy process. (OECD
researcher).
10
2. The state of educational research
Indeed, from discussing this issue with policy-makers the application of research to
policy-making and/or practice is not straightforward nor just the responsibility of the
researchers,
‘Of course, when I say by using the research, I mean clearly its not using the fine
detail of the research by any stretch of the imagination. We are reacting to
general messages, which come floating through those things after a period of
time. For example, when we considered the basic way that we were going to go
about the business of improvement, which in [LEA], which is focused on school
self development, we did look at some important major research. We took their
evidence, if you like and believed their conclusions. It would be really important
to stress we are operating at the level of those, you know, kind of views mediated
three of four times through ‘popularisers’ really, rather than strictly speaking the
academics.’ (LEA Director of Education)
As far as the HE researcher stakeholders are concerned they would argue that most
educational research is relevant, or that an argument for its relevance could be
generated,
‘One of the criticism has always been that educational research isn't relevant to
practice. Now I just don't think that's true of the vast number of proposals I've
seen. Most of the proposals that we look at are very relevant to practice. They
are concerned with ultimately trying to understand what happens in schools, so
that it can be improved for example or trying to understand schools organisation
so that we can make them more effective organisations.’ (HE researcher and
TLRP Team Leader).
From discussions with the stakeholders it appears the problem may be in the process
of transforming research findings into useful and relevant knowledge. For example, it
may be that researchers actually need to take more care when suggesting how relevant
their research is to policy-makers and/or practitioners,
‘One of the things that I am always concerned about is the way that educational
researchers express the claims that they make and I do think that with an
experienced educational researcher, the way that they make claims for their
exploratory studies is pretty dicey. I would never do that and I would always be
absolutely clear in the findings that I could have quite a lot of confidence in the
findings but only in the context in which we had worked.’ (HE researcher and
RAE panel member)
Again, the issue of relevance to policy and practice is discussed in more detail later in
the report. This also includes the role of the practitioner, in its widest sense, in the
research process.
2.4 Conclusion
Given that many of the stakeholders tended to agree, at least in principle, with some
of the criticisms of educational research made by Hargreaves, Woodhead, Hillage et
al., and Tooley and Darby, it is perhaps not surprising that they focussed themselves
upon the poor quality research that is generated. However, all the stakeholders would
agree that there is a great deal of excellent and decent research being produced. The
11
2. The state of educational research
perspective offered by the stakeholders was, without being pessimistic, to argue that
educational research in the UK needed to, and, incidentally, could, be enhanced.
How educational research could be improved is the focus of the rest of this report.
The next section outlines the factors, as discussed by the stakeholders, which may
explain the feeling that educational research needs to improve. This also begins to
highlight the level of feeling for change and the relative importance of capacitybuilding to the educational research community.
12
3. Constraining the capacity for high quality, relevant, research
3. Constraining the capacity for high quality, relevant, educational research
3.1 Introduction
The stakeholders discussed a number of examples and explanations that they believe
underlie the issue of quality in educational research. These included structural
problems ranging from the organisation and institutional processes of educational
research currently in the UK, to the composition of the research community itself.
Conversations emerged around the education research culture, influenced by the
structural problems mentioned above, and, in particular, the many and varied paths to
becoming an educational researcher. These factors gave way to issues relating to
research training and the nature of continued professional development amongst the
education research community. However, every stakeholder eventually turned these
conversations around to methodological issues, and the use of, or lack of, particular
research methods. Almost, without exception, this revolved around the distinction and
relationship between quantitative and qualitative methods.
This part of the report is presented in four sections. The first section begins by turning
to what appears to be a recurring concern regarding the relevance of educational
research to policy-making and practice. In particular, this presents the stakeholders’
views on the capacity for researchers to make an impact on both policy and practice.
The second section explores the structural factors that were reported to constrain, or
limit, transforming the potential for good quality research into reality. This is
followed by a discussion of the key methodological issues raised by the stakeholders.
The final section then focuses upon problems associated with the education research
community itself, its response to the constraints described above and its composition.
3.2 The missing impact of educational research
‘So I think a lot of problems in this country, a lot of educational policies are not
based on any research or any facts and we don't actually have any mechanism
ready for using research properly. We spend a lot of money on it… I think we
waste a lot of money on educational research. We should target much more. We
need educational research and we need to make sure it causes us to decide our
policies based on the facts that are coming out rather than just…I think its not
clear, its not planned and its not used properly and its not acted on when it comes
through either” (Chair of LEA Education Committee and LGA executive
member)
This stakeholder raises the concern that policy-making, in particular, is not based
upon ‘facts’, or evidence, generated from educational research. The explanations
given for this are varied, even within this one paragraph. For example, they include
the suggestion that researchers are not undertaking relevant research, the problem of
turning research into a form that policy-makers can react to, and finally that even
when research gets articulated into policy-making circles it can often be ignored.
This latter point was one that HE researchers were keen to express. For example,
13
3. Constraining the capacity for high quality, relevant, research
‘What you keep coming up against… is that most civil servants have no regard
for research at all and so you can set these things up but whether actually
anybody’s going to take that much notice of them.’ (HE researcher)
However, the reluctance of policy-makers to ‘believe’ in research may be changing,
‘Its not one that is entirely one that…if whether the difficulties and shortcoming
that it’s the research community’s fault. For example that policy people have
often been very rude about research and not looked to research to answer any
questions. There is much more of a belief in gut instinct than evidence for
example. But that may well be changing.’ (OECD researcher)
This improvement had also been seen in relation to the devolution of government
across the UK.
‘that [the relationship between research and policy-making] has improved a bit
but I think it varies a lot in the four countries of the UK actually. I mean, I think,
by just talking to the kind of people who are in Scotland, I mean, you know
much about the Welsh context, I don’t know so many people there, but my guess
is that it’s actually much easier to have an effect on policy making. Certainly in
Scotland and I suspect also now in Wales, than it is in England and I kind of
draw the analogy between, say Scotland and Republic of Ireland, where if you
give a talk on education, the Minister always comes, you know, well I mean, you
know, you would be very lucky to get even a kind of a junior, junior, junior
minister in England, because they themselves don’t particularly value it. I think,
the nature of the policy-making community’s quite important. I mean, it might
have something to learn, in terms, of capacity by looking, say at Scotland, what’s
happening in Scotland and whereas I know people are doing that, but also what’s
happening in Ireland. I mean, that isn’t to say that, you know, lets say they have
better policies in place but perhaps a better way of working with researchers.’
(HE researcher).
The degree to which research has a part to play in the policy-making process is only
one aspect of research impact. The relationship between research and practice is of
equal importance. And, again, the HE researcher stakeholders were keen to emphasise
the reluctance of practitioners to use research in their work,
‘I think we have very large numbers of teachers in the UK who don’t feel
research is valuable, have never had really that much contact with it or think of it
as theory and therefore not useful, so they’re, they’re not even on that starting
point, where they want to work with researchers because they think of
researchers a load of rubbish, that’s going to make their life more difficult.’ (HE
researcher).
Stakeholders from the practitioner side of education responded to this by stressing the
difficulty for practitioners to find the time to change their practice based upon new
research findings,
‘I mean what’s happened as far as teaching is concerned, my view is, that
teachers have become, there is a significant measure, forced into not trusting
their judgement and because the pressures on their time have been so much
greater, they have had significantly less time to reflect upon their own practice.
So, there is a measure which, more and more teachers imply or ask, have I got to
14
3. Constraining the capacity for high quality, relevant, research
do this? And secondly have I got time to do it? Those are the questions which
face them. So their openness to new suggestions, is being significantly reduced,
although actually it continues to amaze me, that if you say something good, on
the whole they still, a significant proportion of teachers will respond positively,
if they can see there is something in it.’ (LEA Director of Education).
Stakeholders with significant responsibility for developing teaching practice in
schools in effect admitted to their limited knowledge of research findings. In this
example a Director of Education for an LEA highlighted the rather ad hoc use of
research evidence,
‘Our knowledge of what is available in terms of education research is limited, it
is just whatever we pick up from various sources by keeping generally abreast of
things from the TES, you know the Institute in London, briefing papers and so
on. We pick up bits of research, we then have an overview of it.’ (LEA Director
of Education).
It was also implied in discussion with such stakeholders that, where utilised, research
evidence was often accepted on trust. In other words there was little critical review of
research evidence in this part of the impact process.
However, as the opening quotation of this section implied, HE researchers were not
devoid of responsibility in the way educational research could have greater impact
upon policy and practice. Not only was there criticism that researchers were not
undertaking research in policy-relevant areas, there was also the suggestion that many
HE researchers were not staying abreast of developments in teaching, thereby limiting
the relevance of their research to practice,
‘Actually academics are not very good at being up to date with policy which is
not terribly surprising but more seriously they are not always very good at being
up to date with school practice. That’s more worrying.’ (DfES Officer)
The recommendations, made by the stakeholders as to how research can be made
more relevant and have greater impact to policy-making and practice, are explored in
the final part of this report. However, it is necessary to start highlighting some of the
stakeholders’ concerns about the difficulty for research to be seen to have an impact
on policy and practice. As this stakeholder implies, knowledge generation can be a
very complex process, often detached from individual research projects,
‘One of the problems that I have seen is that it always makes an impact quite a
long time after the research is done, in fact it makes an impact once it has
become part of the common sense understanding and therefore the fact that it
comes from research has been lost sight of.’ (HE researcher and RAE panel
member).
This statement also reflects how difficult it is for educational research to be seen as
relevant because of the size and complexity of the educational world. The next section
starts to explore a number of structural constraints on research capacity-building
identified by the stakeholders.
15
3. Constraining the capacity for high quality, relevant, research
3.3 Constraining advances in educational research
The last point made in the previous section noted how difficult it is for educational
research to be seen as relevant. It provided an example of how factors, a long way
removed from the control of the individual researcher, play a part in inhibiting the
potential for educational research to be realised. The most obvious, and almost
irremovable, obstacle to this is the size of the education system. This was raised by
the following stakeholder,
‘I think you know one of the problems is you’re dealing with a huge organisation
here. Because what you’re actually dealing with is something which consists of
OFSTED, the DfES and 25000 schools and each with an average of… I don’t
know 100, 200 pupils. You’re talking huge… It’s a huge machine and the wheel
turns slowly.’ (OFSTED Research Manager)
Again, the implication here would be that it is difficult for research to be seen to make
an impact upon policy and practice, simply because change in the education system
takes place at an incredibly slow pace. The notion of a wheel turning slowly, or more
precisely, the constraint and hindrance of time, was present throughout many of the
discussions with stakeholders. Another example of this was given in relation to the
demands made in the commissioning of research. As a consequence research quality
may be affected,
‘Well I think to some extent that is possibly driven by time. It certainly is with
the thing you’ve just mentioned. We wanted to get the thing done very very
quickly. It was a little bit on the hoof to be honest. But the more… as you know
the more complicated a piece of research the more careful you have to be. It’s no
good… obviously I have seen this already in… where people are the qualitative
end of this sort of sector are doing things that are a little bit quantitative where
they end up not quite taking the data that they need.’ (OFSTED Research
Manager).
In the above example the importance of time in the commissioning of research was
seen as possibly limiting the quality of the research. A number of stakeholders also
raised the possibility that the process of commissioning or funding research was also a
constraint. This claim was even made for funding bodies who are not directly related
to the policy and practice domains,
‘It [the funding body] has a limited amount of money to give out and therefore
its job is to create as many hurdles as possible at which researchers could fall in
order that it doesn't have to give them the money and that doesn't build research
capacity… they [the Joseph Rowntree Foundation] will give you feedback and
they will help you put together a good research proposal and then when you're
doing the research they will create for you an advisory group whose job is to
help you. So, it’s… when you look at the major funder of educational research
behaving in that capacity destroying way that create real problems I think.’ (HE
researcher and member of NERF)
The underlying suggestion here is that a competitive research environment is perhaps
not conducive to capacity-building since it tends not to be supportive or co-operative.
Consequently, the funder, as this example illustrates, is more interested in identifying
the research that it does not want to fund, and hence is not immediately interested in
16
3. Constraining the capacity for high quality, relevant, research
improving the proposals or research it funds. In this example the stakeholder suggests
that the approach followed by the Joseph Rowntree Foundation not only reduces the
obstacles put forward by other funding bodies but also can directly assist in capacitybuilding.
However, it would be wrong to suggest that this was a typical reaction to the funding
and commissioning agencies. One stakeholder, for example, with experience of the
‘other side’ of this process disagrees,
‘I think that’s what it probably looks like from the outside. I can assure you
having sat on the grants board for four years that it wasn’t like that from the
inside. I think the board members saw their developmental role as very important
and they weren’t just looking for ways of rejecting bids although sometimes
faced with a mountain of bids in front of you and you know you’ve only go it a
certain amount of money you do that. But I think on the whole we were looking
to give feedback on bids that would actually allow people to substantially rework
them and come back with better bids… So I think the board members actually
had quite a strong commitment to that… to a developmental role […] But I think
the basic problem at the end of the day was that in practice people didn’t come
back with better bids.’ (HE researcher and former ESRC grants board reviewer).
This stakeholder does acknowledge that their ‘developmental’ role was, in practice,
limited. But they also go on to suggest that the success of their approach was, at least
in part, actually the responsibility of the researchers to improve their own applications
and bids.
The problems associated with competitive research funding were also reflected in the
stakeholders’ thoughts on the peer-review system. The use of peer-review in the
funding process was seen as a generating a particular tension, again, not conducive to
capacity-building,
‘So the job of a peer reviewer is to sit in judgement. The job of a peer reviewer is
not to help the researcher develop the work particularly […] The anonymisation
of it, this is a delicate one because of course there are excellent reasons why a
reviewer should be free to say what they really think without thinking what
happens when I bump into this chap at BERA or in Sainsburys next. That’s true.
But it also creates opportunities for people to do the dirty on other people and
you take my original field, in the UK there are maybe twenty active, really active
researchers, so it’s a very tight knit group. So, if you put a proposal in it's going
to be reviewed by one of nineteen other people. Now, you are a rival for each in
many ways for each of those other nineteen. It's in their interests to actually say
this is not terribly good work or it's perceived to be. I know, when I act as a
reviewer it's very difficult for me not to fall into that way of thinking.’ (HE
researcher and TLRP Team Leader)
However, a typical response to the problems associated with any peer-review system
was that there was no better alternative to this. As one stakeholder representing a
funding agency pointed out,
‘I mean peer review's not perfect, but nobody's come up with a better one yet.’
(Head of a UK research funding agency)
17
3. Constraining the capacity for high quality, relevant, research
Although there were no suggestions how the peer-review system could be changed a
number of stakeholders suggested that at the core of the peer-review problem was the
culture of educational research. For example, comparisons were made with other
academic disciplines that are seen to be more co-operative as a community in this
process. In particular, economics was considered a good example of this approach,
and one where real benefits, in this instance more research funding, were seen,
‘So what we should be like is like the economists who when they referee each
others proposals always say how wonderful they are and then economics
manages to capture a whole lot more research funding, whereas in education
people are too busy stabbing each other in the back and that is something to do
with the system and it's something to do with the culture change into moving into
a much more collaborative and supportive culture and I think we've fallen into
the trap, I think all areas are like this to a certain extent, but I think education
maybe somewhat worse than many others, into the trap of thinking that quality is
something that comes out of competition and quality assurance is about finding
problems and faults as opposed to saying well actually no quality comes out of a
more collaborative approach and quality assurance is about developing strengths.
I know that sort of sounds like worn words, but it's the balance of how things are
done.’ (HE researcher and member of NERF)
Although there was not general consensus among the HE researchers there were
strong views that the educational research community is too ‘destructive’, possibly
due to the level of competition for funding,
‘There are cultural issues as well if you're looking at academic led research in
terms of there's too much of a culture in the field which is a negative destructive
culture. It's too competitive and the system of peer reviewing doesn't help that
and the fact that there are too many researchers chasing too little money doesn't
help that.’ (HE researcher and TLRP Team Leader),
But many stakeholders discussed problems with the educational research culture in
methodological terms, highlighting the naïve approach that many researchers take to
research,
‘I’m not saying everybody but I am saying that there is, there is a culture and
alright a culture is not a monolithic thing but there are strands in that culture
which are, which are about all sorts of things and, I mean, it’s, it’s a very, very
interesting one. I mean, I think the quality of some people in education is highly
questionable. I can point to you, I could point the people walking around with
PhDs, who demean the title, right.’ (HE researcher).
And, again, this point was sometimes discussed in relation to other academic
disciplines,
‘I suspect in some disciplines the immediate response is “right go off and read
this, this and this and see how it fits in.” Whereas I think in education its “yeah
that’s great why you don't you go out and talk to a few a teachers and see
whether it will work or why don't you try a pilot study.” Maybe that’s the applied
nature of the discipline you know that you don't immediately go and reference
your ideas to the body of knowledge that you need to build on. So maybe saying
that something in the culture is not an empty… not just simply saying that it
18
3. Constraining the capacity for high quality, relevant, research
doesn't happen, but it is to do with the culture of the discipline.’ (HE researcher
and TLRP Team Leader)
The issue of the ‘naïve’ educational researcher is closely related to methodological
problems and limitations, and it is the use of methods and methodologies that
dominated conversations with the majority of stakeholders. The next section begins to
outline their concerns in more detail.
3.4 Methodological limitations of educational research
The shortage of ‘quantitative’ research in education, and the social sciences more
generally, has been given a very high profile in recent years (see for example, Major
2001 and Marshall 2001). However, the stakeholders interviewed for this consultation
exercise were very aware that this was only one feature of the methodological
limitations to educational research. The complexities of many of the stakeholders’
arguments is best summarised by the following extract,
‘There are whole areas of social science where for various reasons people have
got into the mode disproportionately of doing non-cumulative small-scale
usually flawed pieces of quote unquote qualitative work, they'd have an
interesting problem, perfectly interesting idea, really plausible you know good
ideas about it and they thought they could test the ideas by just chatting to 20
kids or kids in this classroom versus that classroom and they'd never heard of
Hawthorne effects or they'd never thought about intentionality and social
mechanisms or whatever, I'm slightly exaggerating for the sake of the point. I
mean I think that there are too many examples and education is not the only one
where in social science people have got an interesting problem, some interesting
ideas of what the answers might be, but they can't do the research necessary to
quote unquote test their propositions because they don't have the skills. So, what
they do is they do the research that they can do to match the skills that they've
got. So, they body swerve around the direct way of addressing the question and
they have some second best or third best take on it that they get from using the
only technique or methods that they can use and that’s a characteristic weakness
of a lot of social science and often because people have got this hang up about
quantitative work disproportionately that’s the bit they avoid. And they kind of
work the other way around. They other way around is you just get the ASR
phenomenon of the 1960's, you get people who don't think about causality, who
think variables do things and who just do this hey path analysis is the technique
at the moment and they never stop to think wait a minute we have to talk to some
people to answer this question or whatever.’ (Head of a UK research funding
agency).
This stakeholder, for example, covered many issues surrounding the use of research
methods and methodologies. The first criticism raised the relatively large number of
research projects that were ‘non-cumulative’, ‘small-scale’, and ‘flawed’. This led the
stakeholder to question the extent to which such research can identify causal links in
social phenomenon. The argument given was that often the researchers do not have
the skills to undertake the type of research that will actually address their research
question or problem. The assumption made by this stakeholder is that such work
requires quantitative skills. However, they add that quantitative research can treat the
19
3. Constraining the capacity for high quality, relevant, research
social world too much as numbers and variables, and hence is unable to fully uncover
causal links also.
Another stakeholder, arguing that most researchers do not undertake research in any
logical or scientific manner, questioned the rigour of research,
‘What's wrong with educational research is its not fully articulated links with
parent disciplines or some deeply felt doubt about its aims and missions, it’s just
that the vast majority of the people in it nowadays are almost wholly incapable of
behaving in a logical technically competent scientific way.’ (HE researcher and
member of the NERF).
These stakeholders argued that research too frequently ignores the issue of causality,
or do not set out to fully test their assertions in a satisfactorily ‘scientific’ or ‘logical’
fashion. They would argue that research could, therefore, be lacking in what it can
achieve. However, another common criticism of research, which is conceivably worse
than the ‘missing’ elements of research, is when the conclusions may bear no
relationship to the evidence presented. In particular, the stakeholders believed that
research over-generalised, largely because the majority of educational research is
considered as small-scale,
‘There is a gulf between your evidential base and your conclusions, so
sometimes it is like that, quality issues, but other times you cannot understand
the relevance of the whole area of study. I would just say it is always a problem
about generalising from a small sample.’ (Education researcher and BERA
Executive Council member).
These issues of causality, being able to test propositions, and the process of
generalisation raised by many stakeholders are, as suggested, more complex than
simply the problem of a lack of quantitative research. But nearly every stakeholder
reported that there was a lack of quantitative research and that this has straightforward
consequences in the quality and relevance of research,
‘There is a widely acknowledged absence of quantitative research of particular
kinds, especially, there's a weakness, there's a relative absence and there's no
mechanism for addressing that currently.’ (HE researcher and TLRP Team
Leader).
One consequence of this shortage is that there are simply too few researchers to pass
such skills on to their peers or the next generation of researchers,
‘It’s a real worry in this department that you know in a department of 3,500
students and 190 staff there isn’t one strong quantitative lecturer.’ (HE researcher
and UCET Executive Committee member).
This shortage was also reported for LEAs,
‘There are a lot of LEAs with vacancies because they just don’t seem to be able
to find the people, because the people are not necessarily there.’ (Former LEA
researcher).
20
3. Constraining the capacity for high quality, relevant, research
This has a double impact. First, there are too few researchers in LEAs with the skills
to analyse the large quantity of numerical data now generated for the school
improvement agenda. And secondly, there is little or no capacity within LEAs to be
able to interpret, or ‘consume’, existing quantitative research findings.
The relative shortage of quantitative research was also associated with the limited use
of large-scale datasets in educational research. The application of such data to
educational research is so limited that there are too few examples of its potential,
‘Yeah, that’s the other thing I was going to come to because I think, that is, that
is a feature that you don’t see people using those [large data sets] very much.
You don’t see, for example, the use of the general household survey. I guess that
it’s partly that education research in this country has become, become so policy
focused that people don’t think about using those data sets because they’re
looking at the latest development, rather than thinking, you know, further ahead
or further back, and so people don’t necessarily think that those will be
appropriate. I mean, I think, that’s it’s not just the people can’t necessarily
handle that kind of data, I think it’s also that people don’t think of it as having
any usefulness because they’re so concentrating on, you know, what’s the latest
policy development we can research and so you tend not to look at big data sets,
to shed light on that.’ (HE researcher).
The perceived relative absence of quantitative research need not detract from the
development of sophisticated quantitative techniques, often unique to educational
research. In particular, a number of stakeholders highlighted the developments in the
use of multi-level modelling and the analysis of complex datasets. But there was a
great fear that even though such developments were being made there were too few
researchers who could actually use these techniques. There was also the concern that
the requirements of these sophisticated techniques meant that they had become overly
technical and far removed from most researchers’ minds,
‘I have a suspicion, more than a suspicion, I think there are quite a lot of signs
that we are losing the quantitative side, just at the point when we’re getting more
sophisticated techniques, which I think we’ve got to be very, very vigilant
because, there is a sort of technicist mentality that gets seduced by… essentially
arithmetic or numbers, you know, and, yeah, you’ve got to be very careful about
that.’ (HE researcher).
This would suggest that the situation is not just one of a shortage of quantitative
research, but simply a shortage of quantitative research skills, required to be able to
understand, and critically review, quantitative research. The following stakeholder
suggests that in LEAs where there are the people to undertake quantitative data
analysis they do not have the research background to review the methods they are
employing,
‘What I am saying is that although there are a lot of people doing quantitative
work, with school data and pupil data, a lot of them are actually implementing
central Government analysis. They are not necessarily people who have an
analytical background in research and quantitative research. They are not
necessarily being as critical about the methodologies and are not as aware of the
limitations of the methodologies as maybe they ought to be, so you know, there
21
3. Constraining the capacity for high quality, relevant, research
is a bit of an uncritical implementation of government strategy, but the
government is not worried about that.’ (Former LEA researcher).
Another stakeholder while discussing the presence of sophisticated quantitative
research made a similar point. They argued that there were too few researchers with a
sufficient level of competence in quantitative methods,
‘I think that it’s not that there isn’t any quantitative work going on, it’s just that I
get the sense that it’s people haven’t necessarily, even people who seem
reasonably competent haven’t necessarily reached the levels of competence that
they might have done in qualitative. You compare the kind of competent
researchers and qualitative, compared to the, competent research and
quantitative, there seemed to me to be fewer of those.’ (HE researcher).
Nevertheless, as this stakeholder argues, just because there is an almost accepted
shortage it did not mean that all research should be quantitatively-based in the future,
‘I do feel that we are short of good quantitative researchers in this country and
that is a particular need. What I don’t want us to do is to somehow float away
from that into thinking that it’s quantitative everywhere that is needed.’ (HE
researcher and TLRP Team Leader).
Although there is great concern regarding the shortage of researchers with quantitative
skills and knowledge this is not the methodological limitation of educational research.
Instead, a number of stakeholders conceived the main problem surrounding
quantitative research as one of balance between that and the use of qualitative
research methods,
‘Where I think that has failed in a way is when it has not kept the proper balance,
is there are plenty of questions that have to be addressed through quantitative
methods.’ (HE researcher and RAE panel member).
Many of the stakeholders commented on the rise of qualitative research in education.
Although all stakeholders welcomed this there was this concern that it has come at the
expense of quantitative research,
‘I think there has been this huge increase in qualitative research in education over
the last 10 or 15 years or whatever. I think that's basically very good, very
healthy. It's provided a huge new perspective on work, but I think part of the fall
out has been the drop of interest if you like in quantitative research.’ (HE
researcher and TLRP Team Leader)
As suggested at the beginning of this section there was also a consensus amongst
stakeholders that ‘qualitative’ research, although in a healthier state than ‘quantitative’
research, has its own limitations,
‘On the other side there's an awful lot of qualitative workers where the quality is
not very high, as it were, for various reasons.’ (Head of research funding
agency);
‘In qualitative research because certainly when you start to read some of this, and
clearly some of it is very good, and some of people working it, have led the field
22
3. Constraining the capacity for high quality, relevant, research
themselves. I mean, that’s, you know, that you can take that group out but below
that there seemed to be quite a lot of people who are doing qualitative research
which isn’t terrible good and that maybe because they haven’t ever been very
well trained in it themselves.’ (HE researcher)
In this latter example, the stakeholder begins to highlight the point that many
researchers employ qualitative methods with little training in these methods and
without a critical awareness of qualitative techniques. Consequently, there was a view
that qualitative research tends to lack rigour in its analysis,
‘Almost everybody does semi structures interviews and then mucks about for
better or worse in a rather unsystematic analysis of what is going on so in the
end, when I read something and I have no idea at the end whether the findings
really related to what the interviewee thought or simply how they have
responded to the researcher’s agenda.’ (HE researcher and RAE panel member),
Another key criticism of educational research was the lack of innovation in qualitative
methods. As one stakeholder questions, what has been the qualitative equivalent of
multi-level modelling in educational research?
‘In volume terms there is a lot of qualitative research going on. In quality terms I
am not sure that there is. I think in a sense one of the problems with this initiative
could be to say that qualitative is okay, quantitative needs developing and needs
linking with qualitative. Whereas, actually there is a quality issue with
qualitative research. What is our equivalent on the qualitative side to Harvey’s
multi level modelling?’ (HE Researcher and TLRP Steering Committee
member).
This stakeholder discussed the lack of qualitative methodological innovation in more
detail, comparing qualitative methods in education with other academic disciplines,
‘So I would say the typical qualitative methodology for a project in educational
research is interviews and there isn’t much else there […] I suppose, that in a
sense, you wonder whether they’re using that method because they think it’s the
best method, which it may be for some things or whether they don’t know any
other methods or they don’t feel comfortable with or experienced enough with
other methods to try using those. I mean, another example would be, you know,
things like, kind of diaries and so on, are quire common place in women studies
in sociology. I don’t see anybody much using that in education […] some people
using focus groups but again you don’t always get the sense that they actually
understand very much about what they are, only that they’re, you know, they’re
not single interview, they’re a group interview. I mean, you know, there’s a
massive amount of stuff written about focus groups but compared to say, some of
the debates I’ve seen in women’s studies around the use of focus groups and
around the issues to do, you know, does it empower the people who are in it,
what are the kind of things that are going on. I don’t see that kind of debate
happening, I must say, in educational research.’ (HE researcher).
The overriding concern was that in education the use of interviews is the primary
analytic tool in most research, and that this is often chosen as a method because it is
seen as easy, and researchers do not have experience in, or training in, other skills –
both qualitative and quantitative. There was also concern that many researchers
23
3. Constraining the capacity for high quality, relevant, research
employ qualitative methods (and perhaps quantitative methods) without rigour and
without a thorough concern for the way such data has been analysed or used.
A related problem with educational research highlighted by many of the stakeholders
was the size of most research projects. The tendency to use qualitative methods in
research has meant that research projects are often relatively small-scale, or viceversa,
‘But a lot of it in education is small-scale qualitative research but individuals
who themselves are often not trained at any particular high level but are doing
research around areas of practice, some of which is good, some of which is
mediocre and some which is awful.’ (HE researcher and TLRP Steering
Committee member)
The presence of small-scale research was not seen as a particular problem in itself.
However, a consequence of a relatively large amount of small-scale research was that
the research tends to be non-cumulative and has led to fragmented knowledge
generation. As this stakeholder argues, small-scale research has a useful role in the
whole research process, but is limited, ultimately, in what it can produce,
‘I think a criticism that was made of educational research, David Hargreaves in
particular, was that a lot of it was very small scale work that didn't build on
previous work and it was small scale. And this image of a lot of individual
researchers doing very small-scale projects around the place that doesn't link up
together. I think there's quite a lot in that. I actually think small-scale research
can be really insightful. When I've finished this large-scale project I'm going to
go off and do some small scale. I'm going to go and hang out in some classrooms
and be beholden to nobody. Great!! I'd love to get back to that. But I also think
large scale projects are really important that can do things and maybe as part of
project can kind of replicate themselves.’ (HE researcher and TLRP Team
Leader).
While the Teaching and Learning Research Programme (TLRP), for example, has
provided the resources for more large-scale teaching and learning research, a shortage
of research resources in the past has meant that research has tended to be small-scale
and non-cumulative. A corollary of this has been the generation of a large number of
researchers whose research skills can, apparently, only be applied to small-scale
studies. This is discussed further in the last section of the report.
In relation to the methodological limitations of educational research many of the
stakeholders began to discuss some of the reasons that may explain the tendency
towards a greater amount of qualitative research rather than quantitative research. For
example, this stakeholder suggested that it is more ‘safe’ to do qualitative research
because it is seen as being more subjective and hence difficult to be criticised,
‘This maybe a bit crude, but it’s easier to show that something is wrong
statistically than through the touchy feely stuff, so you’re reputation is less amiss
if you’re doing the qualitative, I think.’ (HE researcher and UCET Executive
Committee member).
Another stakeholder argued that qualitative research is presented as being easier than
quantitative research, hence giving (a) greater encouragement to new researchers to
24
3. Constraining the capacity for high quality, relevant, research
undertake qualitative research, and (b), a false impression of qualitative research as
being not as rigorous or scientific,
‘I think sometimes qualitative research is presented as being easier than
quantitative researcher. That the concepts in quantitative research require you to
be numerate, they require you to think quantitatively and that to be a qualitative
researcher all you need to do is go out with a tape recorder and get some quotes.
Now we know it isn't as easy as that.’ (HE researcher and TLRP Team Leader).
Although related to the point made by the previous stakeholder, a more common
response was that the choice of research method was possibly related to the creation
of ‘methodological identities’. These identities, some stakeholders argued, were
generated very early on in the development of researchers’ knowledge and skills. As
these stakeholders note, this can severely affect their approach to research training,
‘Its just that they say at an early stage, they say “right I'm going to do an
ethnographic or I'm going to use qualitative interviews, I really don't want to
know how to do anything quantitative.” It's as if they make a very early decision
about what they're going to learn about. I don't know whether it’s a genuine lack
of interest or whether it’s anxiety about their own research that they're just so
preoccupied with getting their own work right that…but I do think that’s a
problem and again I think this is something that Gordon Marshall pointed out,
people coming out with a PhD that isn't a well grounded research training. I think
maybe the one plus three [new ESRC research training programme] is trying to
address this.’ (HE researcher and TLRP Team Leader);
‘My experience with running our existing course is that students sometimes opt
out of coming to the sessions that they don’t think they are going to use and it’s
quite hard to persuade them that really this is part of their education and how are
they going to be able to evaluate what somebody else has written if they don’t
come to the workshops on all the different methodologies.’ (HE Researcher and
research training officer).
This stakeholder highlighted how these ‘methodological identities’, once created, can
last throughout a research career,
‘…and I’ve been supervising and just finished a guy who’s been around for a
long while and that’s just marvellous research and there was no way he was
going to do anything other than interviewing. So problems of identity and where
identities are formed are quite interesting. If people are coming from schools
with their identities shaped that’s already interesting.’ (HE researcher and TLRP
Team Leader)
The creation of these ‘methodological identities’ is complex, as the stakeholder above
suggested. However, there was the belief that many of these identities were being
reproduced now that many of those who have gone through their research careers with
these identities are teaching research methods themselves. And, as this stakeholder
argues, it can be very easy to turn new researchers away from quantitative research
methods,
‘I think it’s partly that qualitative researchers or people who promote it have
actually promoted it in a way that is very, very, appealing to students and so on.
I've sat in on, organised, co-ordinated, been involved in a lot of research methods
25
3. Constraining the capacity for high quality, relevant, research
courses and often, I know I'm caricaturing, it’s often said there are two ways of
seeing the world, a positivist, a scientist in a white coat and you use quantitative
methods and students sort of recoil from that or you can be an open and a
qualitative researcher, you engage with meaning and you know you're a warm
cuddly sort of person and students think “yeah yeah that’s me, I want to do that,
that's me, I don't want to be a scientist in a white coat”.’ (HE researcher and
TLRP Team Leader)
It is not just research training that tends to reinforce the use of particular research
methods. One stakeholder, in relation to the development of research skills amongst
practitioners, argued that it is often more feasible and ‘natural’ to undertake particular
kinds of research within educational settings,
‘If I were to start with students who were doing dissertations as part of their
training to be teachers. They always chose to do the case study. Always. And I
say to myself “why?” and I think they tend to do it because it’s… it isn’t entirely
that they are frightened of numbers, it isn’t that. Because actually what they do
takes a great deal of time and a great deal of concentration and effort. I think that
it is feasible… they see that it is feasible within the length of time that they’ve
got. They also have contact with a few schools so that they can do it. It’s
actually possible for them to go and negotiate this personal relationship for
research. They feel comfortable with that. It coheres with their kind of feeling
about the nature of education being a personal relationship. They feel very
uncomfortable with writing to all schools in the LEA and getting 80% rejection
letters or not returned. They feel they are not in contact with the substance of the
thing that they’re engaged in. I don’t think it’s necessarily a fear of the numbers.
It’s not an emotional thing to do. They don’t feel it’s about teaching. I sense that
also amongst people who are doing research in education, particularly those who
are doing areas where the personal orientation is very important. So there’s that
aspect. There’s a financial aspect. It’s easier to do it and cheaper.’ (HE
researcher and research training officer)
Also, the relatively greater amount of qualitative research means that textbooks and
journals are dominated by qualitative methods, generating a greater profile for this
type of research, and across the entire social sciences,
‘But I suppose that it’s also, that if you look at what’s written on research
methods, there’s an awful lot more written on qualitative methods than
quantitative methods. I mean Sage publish about, you know, ten titles a week, as
far as I can see, and that’s just one publisher on qualitative. They don’t publish
as many on quantitative and they don’t publish as many on mixed methods, so I
kind of think it’s a social science wide phenomenon… it’s certainly true of the
other social sciences that that’s also an issue, so I don’t think it’s specific to
education.’ (HE researcher)
Given these other factors, many stakeholders believed that the lack, or narrow focus,
of research training was still a key factor in the methodological limitations of
educational research. For example, this stakeholder argued that many staff in
education departments have actually never had any research training,
‘One of the problems with educational research in education departments is that
so many of the staff have never had any research training and they don’t know
how to do a factor analysis, they don’t know how to do an open ended interview.
26
3. Constraining the capacity for high quality, relevant, research
They don’t know what the pros and cons of longitudinal studies are, all of these
things.’ (HE researcher and RAE panel member).
One stakeholder asked where such staff have completed doctorates did this constitute
formal research training anyway, because of its narrow focus,
‘… the PhD and does that count as a formal induction and I think in the past it
certainly hasn't because that's also been extremely narrow in its focus.’ (HE
researcher and member of NERF).
Interestingly, there was a notable absence in the stakeholder interviews of discussion
of the role of continuing professional development beyond researchers’ initial
research training. This illustrates how little concern has been given to the way
researchers develop their research skills and knowledge throughout their research
career. Although discussed in greater depth in the final part of this report it is worth
noting some of the reasons that the stakeholders gave for this. For example, the notion
of ‘up-skilling’ or ‘revisiting the classroom’ can be a sensitive one for individuals,
‘actually there is I think a very sensitive issue about… to somebody in my age
group who trained in the seventies – “don’t you think its time you get updated?”
I would relish that now. I haven’t actually had proper updated training. It’s quite
basic that we do give ourselves a chance to actually have that. But that’s a
terrible admission because if that’s your career and to say you can’t do your job
because you’re not properly skilled because things have moved on and left you
behind is a kind of in a sense an admission of failure at one level and people
aren’t likely to do that if they’re seeking promotion. They’re not going to go to
Head of Department and say “actually I can’t do this job so there’s a good reason
to not promote me.” You know what I mean?’ (DfES Officer)
Another significant factor that was considered to deter researchers from developing
their skills and knowledge was the poor quality of staff development provided within
Higher Education Institutions (HEIs),
‘The responsibility of institutions of higher education to provide better staff
development for their staff. That is something I have to say having worked in
three universities I feel is quite poor actually compared with if you like at teacher
development.’ (DfES Officer).
Underlying all the problems of research training discussed above was the belief that
educational researchers are a relatively unique group of people with very different
academic and career backgrounds. This includes at which stage of their life many
individuals become educational researchers and what academic backgrounds they
have,
‘So I think it is partly to do with the point at which people enter education
departments and what they’ve done before, and, of course, once they’re there,
particularly if they come in to do something like teacher training, then they’re
likely not to have much opportunity to pick those skills up as they go along, in
the way that perhaps someone who joined a more conventional social science
department, of course, would still have that, a teaching load but might be
expected to kind of pick up on those kind on things if they have not already
received that training.’ (HE researcher).
27
3. Constraining the capacity for high quality, relevant, research
There was a general belief that the diverse, and perhaps ad hoc, creation of research
careers may have led to the disparate and limited nature of some educational research.
Again, as this stakeholder acknowledges, this may be due to the type and level of
research skills developed,
‘It's the sort of ad hoc nature of what capacity there is and I think that's got
something to do with the ad hoc nature in the way in which peoples skills are
developed and which people enter into educational research... So I think it's
driven accidentally by whatever skills people bring with them.’ (HE researcher
and member of NERF).
The nature and composition of the educational research community featured quite
significantly in many of the stakeholder interviews, and this is dealt with in the next
section.
3.5 The nature and composition of the education research community
The previous discussion outlined how many of the methodological limitations of
educational research were tied to issues relating to research training. In particular, this
began to identify the rather unsystematic approaches to training within the educational
community. This section examines some of these thoughts in more detail.
A factor that the stakeholders believed made the discipline of education research
distinct to other academic disciplines is the route that many people take to becoming
educational researchers. For example, it was noted that there are few educational
researchers who come through the ‘typical’ route of undergraduate degree, to
postgraduate training, on to postdoctoral research, and then finally to research career
stability,
‘It's the lack of a, historically the lack of a through route from undergraduate to
postgraduate to research career. There's been a problem for education
departments. I mean, we've had not very extensive route system into the
undergraduate world… it's going to be important to ensure that that then draws
young graduates into postgraduate work in education and then into a research
career in education.’ (HE researcher and member of NERF).
As this stakeholder explains, many individuals come in to the educational research
community in mid-career, and, as a result, tend to ‘miss’ a research apprenticeship,
‘Education, as you know, as compared to other social sciences, except perhaps
social work, has a lot of people who come in mid career, who don’t necessarily
come in having done a PhD or having gone through a conventional research
apprenticeship and they may not actually go through that stage, if they come in
as a lecturer, or they come as a teacher trainer, so I think that’s quite a big gap.’
(HE researcher).
It also means that new students of educational research may not have the capacities to
understand and go on to use sophisticated or advanced research methods,
28
3. Constraining the capacity for high quality, relevant, research
‘Educational research in higher education is through the classroom and that
raises some particular problems. I taught on our EdD course here where we have
a lot of teachers and it’s very difficult. A lot of them just don’t have the
background to deal with any kind of sophisticated quantitative techniques and
they haven’t gone through the usual research training.’ (HE researcher and RAE
panel member).
It is also possible that educational research careers end rather prematurely to
researchers in other fields. As a result there may be fewer real ‘experts’.
A number of stakeholders pointed to the requirements of Initial Teacher Training
(ITT) as adding to this problem. Consequently, many education departments,
established to prepare students for teaching, consider research to be of less
importance,
‘There is the particular status of education, particularly in university departments
of education, the tradition of it being of research I think has been of lesser
importance as compared with the business of preparing teachers and the teaching
side, as opposed to the research side and the general standing of education
among disciplines in respect of the research I think is still being regarding as
rather low.’ (OECD researcher).
As a consequence of this, recruitment in to such education departments can often be
driven by the need for classroom and teaching expertise. Therefore, it is believed,
many new appointments are less likely to have a research background,
‘The commitment that most university education departments have to initial
teacher training and increasingly to undergraduate work, because what that then
means is that they have to recruit staff who understand the practicalities of
classrooms and can teach initial trainers… That then means that you are
constantly sucking into university education departments people without a
research background and then you create the problem of how do you give those
people research background.’ (HE researcher and member of NERF).
Some of the stakeholders saw a direct connection between this and the quality of
educational research. Their argument was that in the light of the RAE many of these
ITT staff have been pressurised to undertake research of some form. Since they have
not had any research training, nor have the opportunity to develop their research skills
and knowledge, they are likely to do small-scale research and without being able to
critically review the methods and methodology they employ,
‘But if they don’t register for a PhD but actually come in to do initial teacher
training part of which they are expected to do research then there is a problem.
Many of those people are very lost. I mean they are the people who have been
absolutely sort of badgered out of existence under the RAE. You know told that
they’ve got to produce four research papers and they don’t know how to do it.
Not registered for a PhD, had none of the research training and just left to kind of
sink unless some one was nice enough to set up a support group and get them
going. My guess is that in those circumstances people fit into the things that they
know best, that they feel most comfortable with. So they probably start by
thinking of the methodology before they think what they are actually most
interested in doing in research in, if you see what I mean. “Oh I’ll do a case
study”. They go that way around rather than thinking “I’m most interested in
29
3. Constraining the capacity for high quality, relevant, research
how student teachers learn to control the classroom, how would I best go about
doing it.” They don’t do it that way round they do it the other way around and
then you get it fossilised because you then can’t move out of it.’ (HE researcher
and research training officer).
The education research field may be unique in that a great many researchers come
from a practitioner background. But as discussed above this limits the number of
researchers in the education community with formal research training,
‘We've all come into research through a very similar route which is from a
practitioner background. Now that gives us enormous skills in some areas, but it
also means none of us have been formally inducted into the research process.’
(HE researcher and TLRP Team Leader).
However, these former practitioners do offer a range of different skills that, for
example, are generally missing from social science research training,
‘There are skills that come from practice and knowing your way around schools
and knowing your way around teachers. If you are coming in from some other
disciplines, you do not have those skills and they are valuable skills.’ (HE
researcher and member of NERF).
Although entry into educational research mid-career may not mean that any of the
formal research training will be missed, it does mean that the average length of time
for each individual to be engaged in research over their career could be significantly
lower than in other academic disciplines. The consequences of this may be three-fold.
First, the researcher is less likely to find time to accumulate an extensive knowledge
of a range of research methods. Secondly, the potential for each individual to
undertake cumulative research projects is greatly reduced. The third consequence,
highlighted by nearly all of the HE researcher stakeholders, is the perception of an
ageing education research community, being led by a generation of researchers who
came in to education research without formal research training,
‘There is another explanation for some of us who are at the end of our careers
than the beginning or even the middle and that is that we came through from the
curriculum development movement and Barry McDonald has written extremely
well about […] He said that most of us were taken from the classroom as good
teachers and we were thrown into the curriculum development pot and we,
instead of at the end of the projects swimming back to where we’d spawned and
going back into schools, we actually all went on into the research community. So
there was a whole generation of people who came out of schools without
research training…. and that generation will soon have worked through but
behind it I don’t think we’ve got a generation that themselves has yet been well
trained in both [qualitative and quantitative methods].’ (HE researcher).
But, as this last stakeholder suggests there is just as much concern for the next
generation of education research leaders – a point of ‘crisis’ for another stakeholder,
‘The capacity for high quality quantitative research and the growing of new
people to do that is quite a serious problem. I think there is a more general
problem about the age structure in educational research, whether it be about
quantitative or qualitative or whatever… I mean once us lot have gone, whatever
30
3. Constraining the capacity for high quality, relevant, research
you might think of us, there is not much left and most of us are going in the next
ten years, so I do think there is an overall crisis.’ (HE researcher and head of
department).
3.6 Conclusions
When asked why educational research was in its current state the stakeholders
predominantly highlighted the constraints that have limited the capacity for highquality and useful research. These ranged from structural problems associated with
the education research community and the education system more generally to more
precise methodological ‘gaps’ in educational research. Great emphasis was placed on
the roles of researchers and ‘users’, including policy-makers and practitioners, in the
process of research impact and dissemination. Although this was not perceived to be
the sole responsibility of researchers there was a belief that there was little
understanding of the mechanisms and processes in which research findings can be
transformed into knowledge that can be applied to the daily work of ‘users’. This
constraint is believed to be compounded by the size of the educational system and the
slow pace of change is associated with such large systems. However, many
stakeholders also outline a number of constraints associated with the educational
research culture. Such constraints included problems brought about with a competitive
funding environment and, ironically, the greater production of poor quality of research
apparently as a direct consequence of the Research Assessment Exercise (RAE).
Even though these factors may be out of the immediate control of individual
researchers stakeholders were also keen to stress that there are too many ‘naïve’
educational researchers, with limited research training and hence little knowledge or
expertise in research methods and methodologies. The methodological limitations
expressed by the stakeholders were wide ranging, but the key concerns were the
shortage of quantitative skills, limited attention paid to the issue of causality,
constraints brought about by ‘methodological identities’, the nature of research
apprenticeship in educational research, and the lack of continued professional
development throughout an educational research career.
These constraints begin to identify the factors that may need addressing in building
capacity in educational research. However, it is also clear that addressing these factors
will require the attention of the entire education system and not just the research
community. The discussion now continues by presenting the thoughts and suggestions
of the stakeholders as to how these constraints should be tackled and what the
priorities for building research capacity should be.
31
4. Building educational research capacity
4. Building educational research capacity
4.1 Introduction
The first part of this report suggested that, irrespective of how well or poor the
stakeholders viewed educational research to be, there was an overwhelming concern
that it did not fulfil its potential. In expressing this, the majority of stakeholders
concentrated upon what they perceived to be the greatest weaknesses of educational
research. These focussed around issues of quality, impact and relevance. To what
extent these views have been driven by recent critiques of educational research is
unknown. However, they constitute a growing belief that developing educational
research is a necessary and healthy task.
The second part of this report outlines in detail the many factors that the stakeholders
believe have contributed to these weaknesses in educational research. These include
factors that have prevented educational research from having a greater impact on
policy and practice, constraints on the advancement of educational research and
explanations for the presence of poor quality research.
The last part of this report begins to look forward to the process of developing
educational research. It presents thoughts, suggestions and concerns offered by the
stakeholders in addressing issues of quality and relevance. There was a surprising
degree of agreement among stakeholders on what the current state of educational
research currently is and what they believe has limited its greater advancement. This
part of the discussion is little different. All the stakeholders believed that some form
of capacity building was necessary for the educational research community. Although
they presented a variety of ways as to how best this could be achieved, there was an
overarching belief that success depended upon a multi-layered and multi-faceted
approach.
The ideas and thoughts of the stakeholders are presented in the following way. The
discussion begins by examining how research could be more relevant and have a
greater impact, particularly on policy and practice. This is followed by an outline of
the stakeholders’ thoughts on how to make research generate greater and improved
knowledge, primarily, again, for the policy and practice domains. The discussion then
turns to how the organisation of the educational research community and research
teams could be addressed in order to improve research quality. This focuses, in
particular, upon two key suggestions from many stakeholders: first, the need for
greater interdisciplinary research – drawing upon wider social science research and
methodologies; and second, the proposition that this, and other benefits, would best be
served by emphasising the importance of economies of scale in educational research.
Two further areas of focus are then discussed in terms of the organisation and
composition of educational research: the importance of career researchers to this field
and then the role of practitioner researchers to this process.
These suggestions for building research capacity all address the need to try and create
the best circumstances in which to undertake high quality, relevant research. The final
set of suggestions concentrates more on the role of ‘capacity-building’ for individual
researchers. These include addressing issues relating to the design of research, the
development and encouragement of particular research methods, and how to bridge
32
4. Building educational research capacity
the perceived ‘qualitative’-‘quantitative’ methodological divide. This part of the
report ends by discussing the role and form of research training, including the need for
continued professional development.
However, before outlining the ways in which educational research capacity can be
built the report begins by discussing what ‘capacity-building’ may mean. This also
considers some of the issues surrounding ‘capacity-building’ before outlining the
general ideas for improving educational research that the stakeholders gave.
4.2 Research ‘capacity-building’
The first two parts of this report have, in effect, provided the rationale for considering
the need for improvement in educational research. However, ‘capacity-building’ as an
approach to this is seen by some as being problematic. One stakeholder, for example,
found difficulty with the choice of terms being used. However, they also seem to
associate such an approach with a cynical political desire to perhaps control and
engineer change in the research community,
‘I find the term quite a difficult one to deal with […]It sounds as though there is
a reserve out there that needs to be built which will then be sort of drawn on and
dispersed as necessary. That’s a kind of supply and demand model which I’m
not sure I can quite fit that into what I feel has been the primary model in
educational research. I think one of the problems with the present moment is the
pressure that we have from politicians and I’ve been closely associated with the
Best Practice research scholarships. I’ve had a few teachers do these and they
always want to prove that they are right. They don’t want to actually research
their practice and improve it. They want to show something to the parents and
governors. Show that what they wanted to do was the right thing to do and
somehow that belongs to this… this phrase belongs to that sort of model.’ (HE
researcher and research training officer)
This may be a premature criticism, particularly, as in the case of the TLRP Research
Capacity Building Network (as presented in this report), the definition and approach
to capacity building is based upon extensive consultation with the community itself.
But even this approach to ‘capacity-building’ has its critics, since it tends to lean
towards rhetoric rather than substance,
‘I would have to say in my view, there has not been much debate about it.
Capacity building is a sort of mantra, which is brought out and added on to
everything else.’ (HE researcher and TLRP Steering Committee member).
Interestingly many of the stakeholders came to the interviews with an idea of what
‘capacity-building’ is or should be. For example, this stakeholder differentiates
between the need for greater research capacity in terms of volume and the need for
research or skills of a particular kind,
‘It’s the term research capacity building. It sounds as if the thesis behind a term
like that, a notion like that is that the problem in education is there's not enough
research, that there's a lack of capacity, whereas I'm sure that the ideas going
behind this are not quite as straight forward as that, that its not simply about
volume, there's not enough capacity in the sense of volume that there's not
33
4. Building educational research capacity
enough research of a particular kind or enough skills of a particular kind.’
(OECD researcher).
Another stakeholder argued that the scope of ‘capacity-building’ is more than just
about research skills, preferring to see it as a system-wide undertaking,
‘Its [NERF] definition of research capacity as not just being about a set of skills
that career researchers have, but about more generally how the system functions
in terms of producing research that is in some sense needed and I think that sense
has got to be a very broad sense. And that it's equally about the capacity to do
research, sorry to use research.’ (HE researcher and member of NERF).
If a definition of ‘capacity-building’ emerges as a response to the factors outlined
earlier in this report, that have prevented educational research from realising its
potential or from developing further, then this system-wide approach would seem to
be most appropriate. However, determining what ‘capacity-building’ may be from the
perspective of what has not been realised does perhaps underplay what is positive
about educational research. As one stakeholder has already foreseen,
‘I thinks there's a tactical issue in any capacity building exercise of any sort […]
in which you… which is important not to start with the negatives, its important
to start with the positives.” (HE researcher and TLRP Team Leader).
The same stakeholder is also aware that for developments in educational research to
be made it is important, and perhaps necessary, to say what is wrong, or what is
missing,
‘Probably what we need more than anything is the ability to say there is a lack of
skill, a lack of capacity, because its not just skill, a lack of capacity in this
particular area and we can do something about it, whoever the we maybe.
Currently that isn't possible. The system is driven by what happens to be there,
rather than what, from some other view ought to be there.’ (HE researcher and
TLRP Team Leader).
An alternative view would be for ‘capacity-building’ to be ‘aspirational’, where the
perceived gaps in educational research are only part of two greater objectives: (i) to
maximise the current potential for high-quality research; and (ii) to ensure there is the
greatest opportunity for a research community to meet the new demands of an everchanging society. After all,
‘Education is a social venture, it isn’t an academic field. It’s a social project and
it has objectives and educational research is about attempts to understand this
system and its relationship to its objectives’ (HE researcher and member of
NERF)
The advantage of this definition of ‘capacity-building’ is that it is based upon notions
of ‘potential’ and ‘opportunities’. It is not necessarily based upon ‘control’. As a
number of stakeholders discuss, there is a delicate balance to be struck between any
kind of approach to meet the objectives for ‘capacity-building’ and the freedom of the
researchers,
34
4. Building educational research capacity
‘In terms of where the problems are I think its in the lack of coherence and
there's a really delicate balance to be struck, that on the one hand you want
people to develop the skills they feel they need and then to do the sort of research
they feel is important and that flexibility is where you get the creativity, is where
you get the dissident views and so on and so forth. So, the last thing we want is a
heavily co-ordinated system which fails to do that.’ (HE researcher and member
of NERF);
‘Now the balances between the freedom of the researcher and the imperatives
that come from whatever management system you have or facilitative system
you have is a very very delicate one, but I think that would be part of capacity
building.’ (HE researcher and TLRP Team Leader).
This attempt at defining ‘capacity-building’ deviates from a ‘deficit-model’ in that it
is concerned with what is wrong, or missing, for the educational research system as a
whole. However, the factors deemed by the stakeholders to limit the current capacity
of education research ranged from structural, or system-wide, obstacles, down to
particular methodological limitations of existing research. In effect the constraints
occur at a number of levels, including: the policy-makers; the teaching system; the
funding councils; the education journals; the Higher Education Institutions; the
research training, strategic priorities and compositions of education departments; the
composition of research project teams; and the skills of individual researchers
themselves.
Clearly the constraints on educational research occur at a number of levels. Therefore,
‘capacity-building’ needs to operate at all of these levels and, as identified by one
stakeholder, capacity building activities are beginning to emerge at a variety of levels,
‘There are lots of initiatives bumping into each other all over the place. I think
this capacity building stuff has sort of spread - lots of people are concerned about
it now and so there are lots of people doing things. There is obviously a need to
prioritise which are the important things.’ (Director of research funding agency).
One example of this has been the ESRC’s 1+3 research training programme, with its
emphasis on ensuring that all research students undergo training in a broad range of
methods. As this stakeholder also notes this particular capacity building agenda
attempts to address a wider social science concern,
‘In particular the general research methods capacity building agenda and in
particular quantitative which is now at the forefront at… I think the ESRC's
general agenda predates that a little bit. So I think there is a sort of something
strong pushing in that direction and that emphasis there which I think again is not
something which is specific to education. It's a social science issue.’ (Director of
research funding agency).
The funding bodies could also consider using their resources in the continued
professional development of individual researchers, as this stakeholder suggests,
‘I think maybe what somebody, maybe the ESRC or somebody needs to do some
kind of sort of mid-career training for people who are, you know like some of my
colleagues here already established and has educational researchers and teacher
educators, but could actually have a one year sabbatical, not sabbatical that’s the
35
4. Building educational research capacity
wrong word, a fellowship, so it could be a training fellowship of some kind and
they actually say "OK I really do need to develop my skills in mid-career" and
actually were encouraged to do that.’ (HE researcher and TLRP Team Leader).
Capacity building can also come in a variety of forms. For example, the creation of
the EPPI centre and the resources for a series of systematic reviews across educational
research could contribute to the capacity building agenda,
‘I think the systematic reviews are potentially more long term powerful because I
think that’s a way forward that has been grossly under-exploited for years […] I
mean even the people who are doing the current EPPI reviews, the review
groups, keep coming back and saying how shocked they are by the poor quality
of the literature […] almost all of the literature they had been doing the data
extraction on didn’t have sufficient methodological detail yet they thought if they
put the methodological detail in it would become very boring and unreadable and
I can relate to that.’ (DfES Officer)
It is now widely accepted that practitioners need to engage with research to some
degree. A number of stakeholders were also aware that capacity building needed to
extend beyond just the researchers and into the policy-making arena. In particular they
identify the need for politicians and policy-makers to be able to understand, or
consume, research,
‘It still would be useful to train politicians in this. I’ve learned an enormous
amount from sitting on employment tribunals because of the way we’re trained.
They don’t allow us to come to judgement before we’ve written the facts down.
It’s a wonderful training and it’s beginning to influence me politically. It’s a very
good discipline. So there are certain things I think politicians ought to be trained
in and one of them is how to spot the flaws in research.’ (Chair of a LEA
Education Committee and Local Government Association executive member);
‘I mean I think it is incumbent on policy makers to be able to read frequency
distributions.’ (Head of a research funding agency).
Capacity building may also need to address issues of real capacity, in terms of the
number of new researchers entering the education field,
‘There’s another problem with researchers in education, which is the increasing
age profile of educational researchers. Given the fact that schoolteachers are so
well paid now by comparison it’s actually quite difficult for an education
department to attract school teachers into the UDEs [University Department’s of
Education] […] So you could argue we are in a situation where the salaries
outside are more attractive to those inside and there is a gradual and rather
worrying drift of younger, youngish academics in education back into the
schools. I know at least three myself who have found it better… the conditions of
service which includes salary of course to be much better in schools than in
education, so they’ve gone back to schools […] I just… without the full-time
Masters to bring students on from school into UDE’s and the appropriate salary
structure there isn’t a obvious bridge for people like me to come back from
schools and into education.’ (HE researcher and UCET Executive Committee
member).
36
4. Building educational research capacity
Whatever form ‘capacity-building’ takes it requires the research community to engage
with it. In particular it requires researchers, at whatever stage of their careers, to
identify their own needs,
‘There could be a danger, you could put these things on and nobody comes and
then they say, well you know nobody’s interested, because I think there, there is
a problem about running events for people in education departments… I think
there’s this kind of fear that, you know, that you don’t really want people down
the road to find out what you don’t know and my guess is that may also apply to
research methods, that people don’t necessarily like to admit what they don’t
know. They’re happy to tell you what they do know but they don’t want kind of
say, “well I don’t really know anything about this”.’ (HE researcher).
It is also important to consider the ways in which ‘capacity-building’ can get
individual researchers involved. As discussed above, the definition of ‘capacitybuilding’ being presented here is one that offers ‘opportunities’ rather than ‘controls’.
It may be necessary, as this stakeholder suggests, to use ‘carrots’ as a means of
encouraging researchers to engage with the ‘capacity-building’ agenda,
‘It might be interesting to see whether there was a really a nice big take-up and I
think if one allied that with other ‘carrots’, you know, as opposed to going in and
saying, well look, you know, we done three years in and we can tell you that no
bugger understands so and so, in this country and only two people understand so
and so, so you will go… you know.’ (HE researcher).
If ‘capacity-building’ fails to engage individual researchers then it is in danger of
failing. What this might mean for the research community is difficult to gauge. After
all, research will always be needed. But as this stakeholder argues the greatest loss
may be on the reputation and respect of education research, and for society as a
whole,
‘Educational research is needed. For example, there is nobody in the DfES who
is going run a system without wanting a capacity at the very least to be able to
say, "we know what's going on, we can describe the system, we can gather
certain data and tell you a story about it". Nobody is going to sign up to
developments in policy without at the same time signing up to evaluations. So,
educational research is not going to die, no matter how much we cock everything
up. The question is how proud are we going to be of it. I think there is a much
more serious issue which is not what would happen to educational research if we
didn't pull this off because what will… there is a lot to say about that in many,
many, ways. But there are much more serious issues… is how would we feel
about it. Because it would be shameful, just absolutely shameful to not have
contributed to such an important and expensive social project in a way that does
good.’ (HE researcher and member of NERF).
This final part outlines in more detail the priorities that stakeholders gave to building
research capacity. These cover the various levels identified above that have been seen
to obstruct the advancement of educational research. They generally build upon the
discussions presented in the earlier parts of this report. These suggestions are not
exhaustive. Indeed there are a number of factors identified previously as key obstacles
to educational research that are not discussed in this final part of the report. The
37
4. Building educational research capacity
thoughts and suggestions presented here should be considered as priorities to building
research capacity according to the stakeholders.
4.3 Improving relevance and impact of educational research
As discussed earlier, the relevance of research has been questioned by a number of
sources. The stakeholders notionally supported these criticisms, although they took
the view that these were system-wide issues and not the sole responsibility of
educational researchers. Consequently, the stakeholders, notably the policy-makers
and those researchers already closely engaged with the policy-making process, were
able to identify how each of the relevant parties needed to respond in order that
research becomes more relevant and has greater impact upon policy and practice.
For example, it was suggested that the research community had to acknowledge the
importance of research relevance, and ask itself a number of fundamental questions
before beginning to identify ways of ensuring research could be useful,
‘[The questions that need asking are] What kinds of evidence could usefully
inform the policy process, and whether the research activity is set up to generate
those, whether there is sufficient dialogues, that there is a sensitivity to what
those needs are and whether the research capacity has been set up to address
them, to actually develop that evidence. But I think if the lead comes from the
research community to say "I'm sure we can do things differently and better to
try and address then" then a lot of the concerns about research not being valued
enough would actually be met. So rather than researchers lamenting the fact that
there's all this research going on which nobody seems very interested in and
applying. I think the responsibility… it would be very healthy to at least go half
way towards trying to ensure that the research that is undertaken in education is
actually of a kind that can be used. (OECD researcher).
Of course, the stakeholders were also aware that there would be resistance by
researchers to work closer with, say, the policy-making process. But, as this
stakeholder suggested, that did not mean they should not ‘engage’ with such
processes,
‘Rather than being very concerned that ones' research, the purity or the
intellectual underpinnings of the research will be so distorted by being involved
in what is sometimes a distorted political process, where people are using
findings that just fit arguments or not using findings as they should at all. But
that’s not a reason for not being engaged.’ (OECD researcher).
It was also clear that those responsible for the policy-making process would also have
to engage with the task of improving the relevance and impact of research. This was
made evident by one of the policy-making stakeholders,
‘The trouble is if your [the policy-maker’s] perception is wrong you really then
have to question whether you’ve got the right policies and that’s what politicians
do not want to do. The other problem is that if you change your policies you may
actually be challenging your own people’s [the electorate] prejudices and not
actually doing what the people want you to do and you’ve got to balance that.’
38
4. Building educational research capacity
(Chair of a LEA Education Committee and Local Government Association
executive member).
So, on the one hand researchers need to focus greater attention on the actual needs of
policy-makers and practitioners, while at the same time the policy-makers and
practitioners need to identify their own needs in helping make research more relevant,
‘So I think there's quite a job to be done in understanding what the research
needs for the policy process and practice are in order to look at it from that way
round. To start with the policy process or to look at what teachers, schools and
educational administrators, hard pressed as they are in school districts, and so on,
need as much as looking at the characteristics of educational research community
itself.’ (OECD researcher).
But the emphasis was typically on the researchers to focus greater attention on the
policy-making and implementation processes. As this stakeholder suggests, the
research community needs to understand the whole process much better before it can
improve the relevance and impact of its research,
‘So it is a very complicated process and I think one where the better that the
research community understands it, understand what the process is and… the
more chance it has of playing a positive role in it.’ (OECD researcher).
It should be added that, as mentioned above, the enthusiasm for researchers to play a
greater role in the policy-making process generally came from those already engaged
with the process. Although the comment by the following research-based stakeholder
was unique, it does perhaps indicate a counter-view that researchers need to
concentrate their efforts, primarily, elsewhere,
‘I think the relevance problem is over-egged. I think you don't really know in
advance, you can try and build the user concerns and so forth into your agenda,
but I mean the example I always use is because its in my area is when John
Goldthorpe was playing around with the class categories in 1970 for an academic
study of inter-generational relative mobility chances I mean how non-relevant
can you get. He didn't imagine that 30 years later health researchers would be
wetting themselves because his class categories were actually revealing all kinds
of social consequences in social variations in health that they'd never seen before
because they hadn't actually been thinking the right way or measuring what they
thought they were measuring with the Registrar General’s scales.’ (Head of a
research funding body).
In terms of the impact of educational research, more specifically, the stakeholders –
including, this time, the researchers – recommend a similar approach. As this
stakeholder clearly outlines, ‘capacity-building’ must be undertaken by the
researchers and the users, or implementers, for research to have greater impact,
‘With regard to capacity building, there are two elements, I mean one is to
actually have people in the field able to do research at a high level, and another is
to have people who are able to understand and apply research at a high level.’
(HE researcher and head of department)
39
4. Building educational research capacity
This would also include the policy-makers, as they are often the commissioners of
research, and they need to have a greater understanding of research in order to assess
its validity,
‘They [the policy-makers] may well be commissioning academic research and
the use of academic research for their own purposes and it seems to me that there
what they have to be able to do is to be able to analyse and criticise what has
been out before, whether that be a proposal for research or in the form of a report
from research so that they actually do understand what the extent of the validity
of the evidence, that is being presented to them, given the way it has been
collected, talking about empirical studies, that is what we are talking about.’ (HE
researcher and RAE panel member).
Simply a greater understanding of research by all members of the education system
would generally lead, it is perceived, to research having a greater impact,
‘If people understand about research, then they maybe more open to research, in
terms of what they do.’ (HE researcher)
Again, there was a general belief that for the research community to address issues of
impact they had to understand the process of dissemination better,
‘In terms of how research is disseminated and by using that term I don’t mean by
transmission, because most researchers do have a straightforward transmission
model. Helping them to understand that actually that will not produce impact, so
understanding other models of how research is generated and disseminated and
how that is a two way or multi-way set of relationships, I think that that is quite
important.’ (HE researcher and head of department)
This may involve the assistance of a number of key players in the education system.
Such as LEAs,
‘Well our role is dissemination, we have a role of general dissemination and
good practice. So that’s not dissemination of resource, but if as a result of our
learning and teaching activities we are quite clear that there is good practice here,
then we will find ways of disseminating it.’ (LEA Director of Education);
‘One of the big things which we have had to do is look at dissemination and look
at impact. There are formal and informal networks […] I tell you what there is
more than ever now, is website dissemination. So the DFES has it’s website on
local authorities.’ (Former LEA researcher),
and the classroom practitioners, or equivalent,
‘I mean, we have plenty of examples where we have put particular things in
particular classrooms and actually it has not got beyond that classroom door. But
I do think beyond that, the use of practitioners in disseminating messages to
teachers is particularly powerful. Teachers believe other teachers, more than they
do anyone else.’ (LEA Director of Education).
However, the ‘standard’ incorporation of ‘users’ at the end of the educational research
process is limited in the view of some stakeholders. Instead they prefer the
40
4. Building educational research capacity
‘interactive social science’ model, involving ‘users’, etc, throughout the entire
research process,
‘They [other funding bodies], in general, have a very soft definition of users, so I
characterise it as, your user research interest in this work, is that, you think that it
is so fascinating and has hundreds of policy implications, you find one or two
people among your mates, who come from the policy and practice arena, in
relation to this area and you get their endorsement and you go off and do exactly
what you were planning to do anyway. You write up your results and your mates
may endorse them but everybody else says thank you very much. The hard as
opposed to the soft definition is the interactive one and I think that the key things
there are that, it is not the researcher who sets the agenda… I think if we are
serious about things, about implementation, then I think we have to be serious
about involving pupils in defining what are the key educational issues. At least
for some of the work… I think that interactive social science is very important
but I think it is deeply under-theorised and deeply under explored. Most funding
bodies are nowhere near ready to support it, because actually it takes a lot of time
and energy and investment in working with people, to get to the point when you
can generally say that they are setting the agenda. Tokenism does not work.’
(Director of a research funding body)
Consequently, educational researchers need to examine the design of their research
projects to allow for greater flexibility and interaction with the user community.
Although, as this stakeholder notes, the balance between the researchers’ research
training and the users’ demands is a delicate one.
‘Saying that it is kind of important that you get the main structures in place first
but you should always be able to have enough flexibility in research design so
that you can always go back and have input from the practitioners or other users
whoever they are, into the final execution of the study. But it is a difficult
balance I think.’ (HE researcher and RAE panel member).
It was noted, however, that those commissioning or funding research would have to
make allowances for the greater involvement of the ‘user’ community in the research
design and conduct. Since this may involve changes to the research objectives and/or
delays in the delivery of the research findings.
4.4 Knowledge generation in education research
One of the concerns of the stakeholders was how educational research could be
undertaken, and utilised, to generate better quality and sustained knowledge. This was
generally a reaction to the real, or perceived, limitation of educational research impact
on policy and practice. The general perception was that educational research tends not
to produce the kind of evidence that can be transformed into useful knowledge; the
forms of knowledge helpful in the work of both practitioners and researchers
themselves.
For practitioners, the stakeholders argued, educational research needs to produce more
useful tools or artefacts for them to apply, thereby improving, for example, their
delivery of particular curricula,
41
4. Building educational research capacity
‘It would be more in terms of generating tools that practitioners can use or
helping to generate tools that practitioners can use. That might be packages of
approaches with which I have evaluation studies involved in showing that a
particular approach is likely to be effective, particular circumstances.’ (OECD
researcher).
In some ways, as the stakeholder below proposes, this may require researchers to
generate a commodity that is ‘marketable’. The assumption here is that evidence on
the effectiveness of some artefact or tool is required for it to be successfully
‘marketed’. This, the stakeholder alludes, is something that educational researchers
rarely achieve, as they prefer to stress the complexity of teaching and learning
processes,
‘If you show that this research allows you to develop PC's, mobile phones, etc,
etc, you can sell it, because it makes a difference to peoples lives. TLRP needs a
research agenda that is basic science, plus the engineering that generates the
knowledge artefacts that will make measurable difference. You make measurable
difference and there'll be a queue out the door. And I think it will be very, very,
interesting, for example, if in [research project] they generate some kind of social
engineering artefacts that allow parents and teachers better ways of talking about
attainment, so that kids in their programme project really start showing a
difference in attitude, commitment, attendance, attainment. If you can do that
people will buy it. We'll have made a difference. If on the other hand you end up
writing a report that basically says “ain't social life complicated”, thanks for
nothing!’ (HE researcher and member of the NERF).
Similarly, some stakeholders argued that educational research tends not to address
issues of causality, and hence can undermine the validity of explanations offered by
educational research,
‘There are fundamental questions about causality and intention and the nature of
causal explanations and so forth, but they are very difficult questions. I don't
think there are very many people thinking about them.’ (Head of a research
funding agency).
As this stakeholder continues, good theoretical development within educational
research depends upon the importance given to issues of causality in the creation of
useful research knowledge,
‘My mum's got theories of why kids do badly at school, if hers are as valuable as
anybody else's then why spend money on teaching and learning. I mean it's the
classic problem, either what we're doing is better than common sense or it's not.
If it's only common sense then we may as well shut up shop. Our theories are
only as valuable as any body else's. If its not then there has to be something that
distinguishes the net of really difficult questions that we're all aware of about you
know unravelling social mechanisms, demonstrating causality, linking different
kinds of data, so forth and so forth at least there's a transparency about it, you can
front questions.’ (Head of a research funding agency).
Another way in which educational research could generate greater and improved
knowledge, both practically and theoretically, as proposed by the stakeholder below,
is to encourage greater scholarship within educational research. This requires
researchers to think beyond the individual research project,
42
4. Building educational research capacity
‘That work of scholarship where you are drawing richly on both theoretical
backgrounds and the available research. It’s actually much the hardest stuff to do
and it’s… as I see the least well guarded and the least rewarded in the university
world, producing good work of scholarship is actually more valuable than doing
any single empirical study no matter how good it is because a single in social
sciences… I can’t hardly think of a single case where just one study on its own
should be allowed to change anything, but a good work of scholarship which
pulls together that should change things.’ (HE researcher and Director of a
research organisation).
Clearly, as this stakeholder implies, this requires more resource and time for
educational researchers to reflect upon their own and others’ research. Only then
would the knowledge generated be of use to the policy-makers and practitioners. One
component of this would be to undertake systematic reviews, like those being
undertaken by the EPPI Centre. However, many stakeholders suggested that what was
actually needed was more, presumably, quality, research,
‘I think some of the criticisms you know that were made were justified in the
sense of research not building on previous research. I don't think we're in that
situation which perhaps other sciences are or social sciences where you can say
there is a huge body of work. If you're going to progress you must tap into that
and build on that and use previous knowledge and I don't think educational
research is in that position at the moment.’ (HE researcher and TLRP Team
Leader)
In particular, there is a perceived need for more replication studies, and greater
opportunity for such studies, in educational research,
‘It’s also interesting that there are very few replication studies, a point David
Hargreaves made, which I think is a very important point. There should be
money available to allow that to happen.’ (HE researcher and UCET Executive
Committee member).
This could involve the greater sharing of data generated by educational research and
greater availability of useful data,
‘The other thing is we don't share our data. DFES, we have data, local
government often have data, but it's not easy to share it all. OFSTED produce all
this data, but it’s got a whole host of data that it doesn't produce which would be
very useful to share with people. But there seems to be a reluctance to have a
mechanism with the data in one place. The result is there is a number of
organisations keep collecting information from schools and colleges and often
it’s the same information being collected by different people. Surely that
information could be distributed to people who want it. (Chair of LEA Education
Committee and LGA executive member).
One final factor discussed by stakeholders in relation to the ways in which to improve
knowledge generation in educational research was of the role of the practitioners, and
therefore the policy-makers also. One stakeholder outlined carefully how they viewed
the process of creating knowledge from research findings. They argued that
43
4. Building educational research capacity
‘knowledge’ can only be generated with the help of, say, practitioners after
researchers have transformed their research findings into research evidence,
‘I think that evidence is not the same as knowledge so that evidence, which is
interpreted research findings, is contained by the scope of the research and what
has actually come out. There can very often be some wider kinds of issues that
will help to make sense of and probably change that evidence so that, actually
when it comes to down to it, knowledge really requires inputs from other people
than researchers I think.’ (Director of a research funding body).
Whether this is accepted or not there is clearly a need for researchers to consider more
carefully the way in which findings are transformed into knowledge; the knowledge
needed to change policy or practice and the more theoretical knowledge that helps
researchers understand complex phenomenon.
4.5 The organisation and composition of the educational research community, its
academic departments, and research teams
A key solution, it is felt, to many of the concerns outlined in part three of this report is
to undertake more inter-disciplinary research. The belief is that the range and level of
expertise of research methods and methodologies in education would be greatly
enhanced if it were more interdisciplinary. In particular, the development of
quantitative research in education would benefit from this,
‘So its quite a difficult one I think to untangle but I can't help feeling that where
there is inter-disciplinarity and there is, for example, research that brings in
people that might normally be thought of as sociologists, political scientists,
geographers as well as people that might come from education departments then
that's probably where the concerns about lack of quantitative sophistication or
appropriate quantitative development to be effective… that’s where those
worries are best met.’ (OECD researcher).
It was clear from the interviews with stakeholders that educational research can
already be seen to be multi-disciplinary in some respects, since many of its
researchers have come in to educational research from a variety of academic
backgrounds. However, there was a strong feeling that this did not constitute genuine
inter-disciplinary research. As this stakeholder notes, this would require educational
researchers to work collaboratively with researchers from other academic disciplines,
‘Although there’s a kind of inter-disciplinary research in education, it tends to be
people in education from different disciplines working with each other, it doesn’t
tend to be very much. People in education working with people in other social
sciences… it’s a simplistic way and it obviously doesn’t offer all the answers but
what you really want is encourage people in education to work with people from
other disciplines in social sciences on problems where education is an aspect, it
may not be the whole of it, because I don’t too many studies like that really. I
mean, obviously there are, a few examples, but there are not large numbers of
those, in the way that say, for example, you know, people from sociology and
geography and planning cultural studies as started to work together. You don’t, I
don’t think you see that very much in education.’ (HE researcher).
44
4. Building educational research capacity
Many stakeholders articulated the need for inter-disciplinary research in terms of
social science research, more generally. The implication to their argument was that
social science research, whatever this was, had something to offer educational
research. But as this stakeholder points out social science research alone can be
irrelevant,
‘My view is the quality of research, the highest quality of research comes out
where you have a strong social science element but also an educational element
in it as well, in that sometimes of you get a very specific area of social science
that for some reason comes sends its work into education, it does tend to lack a
bit of the relevance that I was talking about as they are just interested in different
things.’ (HE researcher and RAE panel member).
An alternative concern was expressed by another stakeholder, who suggested that this
argument makes it appear that education research has nothing to offer, particularly
with regard to the rigour and critique of research. Instead, this stakeholder goes on to
say, educational research needs to work towards a way of undertaking research that is
similar to other social sciences rather than just allowing itself to be subsumed into the
social sciences,
‘I suppose my hesitation or my warning about that would be that one might get
pulled into a very… to buy into a view that education is at the bottom end of a
set of disciplines in terms of it merits and that the more that educational research
look like other disciplines where there was far less questioning of the quality of
the research… and that to move towards very much a science model of
organising research with major studies that that’s the way to go.’ (OECD
researcher).
Given these concerns, the importance of inter-disciplinary research is still seen as
playing a key role in developing educational research. Interestingly, the concept of
inter-disciplinary research was not conveyed by stakeholders as problematic.
However, there are numerous interpretations of this approach. For example, there is a
distinction between an individual who conducts inter-disciplinary research and a
community or team of researchers working together who each have a different
disciplinary background. The latter example can be defined more precisely as multidisciplinary research.
In the main, when stakeholders were discussing the role of inter-disciplinary research
the implication they made was that this would actually be multi-disciplinary research.
However, this did not mean that the stakeholders did not regard the need for
individual researchers to undertake inter-disciplinary research as important. Indeed,
the last section in this capacity-building part of the report concentrates on the
individual researcher, particularly in terms of their use of particular research methods.
In terms of the composition of the educational research community many stakeholders
took the view that ‘social science’ researchers needed to be ‘brought in’,
‘I mean, I think, I think, we ought to be, perhaps trying to encourage more
people with social science stuff to go into educational research.’ (HE researcher
and journal editor);
45
4. Building educational research capacity
‘Maybe getting other researchers in from other social sciences.’ (HE researcher).
Changes to the composition of the research community to make research more interor multi-disciplinary can be approached at a number of levels. At one level this may
involve recruiting researchers from different disciplinary backgrounds into education
departments or research units. On another level it may involve using researchers from
different disciplinary backgrounds to work on research projects as and when required.
Although they both attempt to achieve the same objective they do require very
different approaches.
The idea of making education departments or research units more inter-disciplinary
was relatively popular amongst the stakeholders. They tended to relate this approach
with the perceived need to build substantial, in volume, research departments, units or
centres. For example,
‘But there are sort of capacity questions about how easy it is for any individual to
get a grasp on what the different types of research involve and therefore it is
bound to involve team working and therefore, team working is a lot easier in a
place like this, where there is a large critical mass of researchers than in the small
unit that most schools of education are.’ (HE researcher and head of department);
‘There is an issue of the scale of education departments as well, so you create
economies of scales and that makes a lot of sense and it's at that point probably
where you can make meaningful links with other disciplines.’ (HE researcher
and member of the NERF).
However, the latter stakeholder goes on to argue that this is best achieved at the HE
institutional level in putting departments together, thereby encouraging collaboration
rather than directly ‘importing’ researchers in to existing education departments.
Other stakeholders argued that the best economies of scale for greater interdisciplinary research could occur at a more regional level. This would be not too
dissimilar to the creation of ESRC-funded research centres,
‘So, we've got to try and build critical mass in the [ESRC] centres.’ (Head of a
research funding body).
Rather than building critical masses around a range of research skills, this stakeholder
argued that these regional ‘centres of expertise’ should be organised around particular
methods or methodologies,
‘I think that what would transform it most would be knowing that there are
places that are perhaps regional, and this goes back to the earlier point, and I
don’t know how far people would use them, but I think having regional bases for
learning, about different kinds of methodology’s across the social sciences.
Because, I think, that, the more we kind of say this is just for education, the more
we separate education out. So I think trying to develop a kind of regional, and I
don’t want to call them centres of excellence, because I’m not really sure if that’s
quite the right term, but they’re kind of centres of expertise, if you like, in
particular methods or methodology’s and seeing if you can actually encourage,
maybe starting with kind of research students and then kind of gradually working
46
4. Building educational research capacity
up to other people, to see those is a place if you want to know about ‘x’, you go
to this, you can go to this place.’ (HE researcher).
This approach may maximise the potential development in particular methods and
methodologies but it is not clear how this would encourage greater multi-disciplinary
research. On the other hand, with greater and stronger networked research activities
this may act to promote the value of educational research, both in terms of its
relevance to policy-makers and practitioners, but to other research disciplines also. As
this stakeholder identifies, a key obstacle to greater inter-disciplinary research is the
intellectual ‘snobbery’ amongst disciplines,
‘Probably the main, the main area for looking for improvement would probably
be how can there be much more collaborative work to gain economies of scale
[…] But its more I think in how to ensure that you have much more synergy
energy and people learning from each other collectively and engage in far more
networked research activities which broadens the actual value of the research
itself […] It must be about how the research is organised among the communities
of researchers and the other players, which was one of the point we were going
to come on to. Not just the researchers, this is not just a research matter. It's
about dialogues. It's about generating interesting questions and important
questions and so on. But also it is about, it is about disciplines and of course its
very difficult, it’s a very tricky area to overcome a very long standing intellectual
snobberies and habits. I think those snobberies and habits are very real and I
think that addressing them and developing horizontal teams and interdisciplinary teams rather than thinking that its each discipline better done is the
way forward. I do think inter-disciplinarity is the way forward.’ (OECD
researcher).
There is, ultimately, a need to develop the right structures and opportunities to
encourage greater research collaboration. Something that this stakeholder alludes to is
the current employment situation of researchers within HEIs,
‘That’s why the model of developing a load of individual experts isn’t going to
adjust the problems. The structural issue of how researchers are employed by
universities, the degree of stability and with inadequate critical mass where there
is real team working.’ (HE researcher and Director of a non-HE research
organisation).
This is an important point since a straightforward way of ensuring that research teams,
in particular, have a range of research skills and knowledge to undertake greater interdisciplinary research is to employ more researchers on individual research projects.
As one stakeholder notes, this is a benefit of research programmes such as the TLRP
with significant resources to help create a critical mass, both in the individual projects
themselves and also across the Programme,
‘I think the approach that we adopted in Phase 2 in terms of large projects was
also a reflection of the desire to build capacity in terms of having critical masses
bringing large numbers of researchers together in terms of having slightly longer
awards rather than the typical 2 year, 2 RA model of research projects which
typically comes through a responsive mode.’ (TLRP Steering Committee
member).
47
4. Building educational research capacity
But many stakeholders, particularly those from the HEIs, were very aware that there
are some serious issues relating to the employment of researchers employed in HEIs
with external research funds. For example,
‘My personal view is that actually the neglected community is the research
fellows, the research associates and we've got quite a large community of people
working on projects at the moment who I think…there's a case for their
professional development which I think is quite important for the future of
research capacity.’ (TLRP Steering Committee member).
The stakeholder above raises a general point about the professional development of
such contract researchers. More specifically, a number of stakeholders stressed that
there needs to be greater stability in their employment. In particular, this relates to the
opportunities there are in furthering their career and being more central to research
projects,
‘So, if you really wanted to build a system that was capable of doing something
different from the way the current system works you would have to think about
how you create more stability […] OK, we know you but we want to see some
names in there that we don't know and some guarantee that these people will be
brought on board with the work that you're doing and where there might be a
relatively loosely defined programme of work to be done, but there wouldn't
necessarily be a series of end to end specific projects to do.” (HE researcher and
member of NERF).
As a funder of educational research, this stakeholder saw the dilemma of being able to
give newer researchers the opportunity to develop their careers by taking a ‘risk’
against their limited research experience,
‘We’ve got a responsibility, as other funders have, to try to help ensure that
people who have some of the skills but maybe not all of them and not all the
experience yet actually get a bite of the cherry to try and progress through the
system. There are some projects that are a higher risk than others but you don’t
have to give them an 8 year contract and 1.8 million.’ (DfES Officer).
The UK research funding system is perceived to be full of such conflicting pressures,
as this stakeholder points out,
‘I can justify talking to you about that in terms of maybe projects should be bold
enough to take more younger researchers with real potential but at the moment
the system requires you and this to have a track records and named people and
the irony is that named people are often too busy to give real in-depth quality and
commitment.’ (HE researcher and TLRP Team Leader).
Here the problem is that those researchers in a position to obtain research resources to
employ more new researchers have difficulty in finding the time to help develop the
careers of such new researchers.
A more radical way of tackling the problem of instability amongst contract
researchers is to follow the route laid by the National Foundation for Educational
Research (NFER). As the NFER Director explains, they stopped using fixed-term
48
4. Building educational research capacity
contracts amongst their staff to ensure that their staff could fully develop their careers
and build expertise within the organisation,
‘We used to have the exploitative Victorian work habit of universities ourselves
but we gave them up. We decided it’s bad practice. It’s exploitative and its not in
either our or the system’s interest. It was probably more than ten years ago now
since we made the change. We decided we would…. at that stage everybody had
a contact with 3 months notice but expiring when the contract you were currently
working on ended […] So we thought it was very silly. So we put a new system
in place. Everybody has a year’s probation that’s important for a number of
reasons. If they were signed off that and I think all but two people have been
signed off successfully, then a contract to 65 [years old], like universities in the
old days but all the same, you’ve got a contract to 65 and then it’s like a normal
employer in the big wide world. If the bottom falls out of our business… you can
see from what I’m saying, we have to combine the culture of research institution
with also business. We’ve got no fairy godmother there. If the bottom were to
fall out of the business then we’d have redundancies.’ (Director of National
Foundation for Educational Research).
As the last point made by the NFER Director suggests, there is no easy solution to
this. And such radical changes, although potentially extremely beneficial, do start to
move beyond the ‘capacity-building’ approach to developing educational research. A
more obvious way of developing the capacity of contract researchers is to ensure they
have access to the necessary training and career development that will keep them in
educational research. Although straightforward, this is not without its problems, as
this stakeholder outlines,
‘At various times we have wondered about how we are investing in helping
research contract staff have more training opportunities, which has been our
attempt at capacity building if you like. We always come across the problem of
that this is the responsibility of the Universities. This is the responsibility of
employers. Therefore, if you start funding it, they are going to do even less. Now
obviously the reality is that they are doing s*d all at the moment, so from maybe
that point of view, you do it for the greater good. I think our feeling was that, I
have never managed to get it beyond thinking about it, is there someway of
constructing an environment that one can be paid for this, but at the same time
levering in other kinds of support for this sort of development. If you are able to
identify the kinds of individually based capacity building and how one does that,
then that might make it a bit easier to add that requirement to the kind of
employment package, which we might be willing to fund, if you see what I
mean.’ (Director of a research funding body).
This highlights the fact that simply increasing the volume of new career researchers in
educational research is not enough to develop research capacity fully. Bringing new
researchers together from different disciplinary backgrounds on individual projects is
a short-term measure to encourage greater inter-disciplinary research. However, as the
following stakeholder argues, there still needs to be the necessary conditions or
incentives to encourage such new researchers to continue to undertake interdisciplinary research as they themselves become ‘established’ educational researchers,
‘You're going to have a group of people who are going to be the career
researchers who will be addressing fundamental problems in education over the
long term, on the large scale and quite possibly from an inter-disciplinary
49
4. Building educational research capacity
perspective. So, you say how do you create not only those people, but the
conditions under which those people can work, and that's something to do with
freeing people up from teaching for instance, creating inter-disciplinary
collaborations within institutions, thinking about inter-institutional
collaborations, making sure the funding is an incentive in the creation of those
sorts of collaborations.’ (HE researcher and TLRP Team Leader).
The increased involvement of practitioners in the research process is also seen by
some stakeholders as a way of changing the composition of the educational research
community in order to build research capacity,
‘My experience with teachers is that they do make very good researchers.’ (HE
researcher and research training officer)
However, there are concerns that the use of practitioners in research could actually be
damaging rather than building capacity,
‘There are all sorts of things that they want to address, which either I think are
not researchable or alternatively seem to me so trivial we should not do it.’ (HE
researcher and RAE panel member).
As one stakeholder points out practitioners do not have the same mechanisms for
quality assurance as HEI researchers,
‘The quality assurance process at the moment is different of course because a)
they don’t go through the RAE unless they happen to team with academic
researchers and b) by and large they don’t go through the normal journal
publishing route.’ (DfES Officer).
It was also highlighted that practitioner-research has, to date, tended to be small-scale
and highly mono-methodic, going against the need for more large-scale and multimethod research,
‘The research becomes highly applied and tends to be very qualitative. The skills
required for quantitative research are so esoteric that most teachers wouldn’t
have the time to get their minds around it, even using something, as relatively
straightforward as SPSS, teachers don’t have the time for that.’ (HE researcher
and UCET Executive Committee member).
For the majority of stakeholders, however, there was consensus that practitioners
should be involved to some degree in the research process but that a great deal of
caution was necessary. This is perhaps not surprising given that many of the
stakeholders believed that one of the problems of educational research was that it was
undertaken by individuals with limited research training.
The degree to which stakeholders thought that practitioners should be involved in
educational research did vary. On the one hand some stakeholders held the view that it
was the professional duty of practitioners to be in a position that they could at least
use research undertaken by others,
‘I’m absolutely clear that all teachers should have a professional responsibility to
use research and once of all we have provided more and more means for them
50
4. Building educational research capacity
accessing that research and access to systematic reviews so they’ve got a little bit
more trustworthy evidence around the system, there should be no excuse… I
think that’s a part of the professional accountability.’ (DfES Officer).
This requires practitioners to develop skills to ‘consume’ research rather than to be
able to carry out their own research. Others also saw the need for practitioners to be
able to undertake research, using research skills, within the context of their own
practice. However, this would be as far as they should be encouraged, since, as this
stakeholder argues, what makes a good teacher does not make a good researcher,
‘I see a very central place in practitioners lives, sorry desirable place in
practitioners lives, where they undertake limited enquiries in relation to their
own practice and so on. If one is talking about high quality research that is
communicable to other contexts that is potentially in the long-term general sable,
then you have, you do have quite big problems. The first problem that I see is
that it seems to me that the characteristics of a good teacher are really
significantly different to the characteristics of a good researcher.’ (HE researcher
and RAE panel member).
To some extent this point is supported by a stakeholder from the practitioner side of
educational research. Here they argue that some research, particularly action research,
can be very useful but realise the limited nature of such research,
‘As far as we are concerned, action research is a very powerful way of teachers
reflecting on their own practice, so I am not claiming that it is high quality
scientific research, is it? That comes in another form. But is does have an
important role to play, because the perceptions that arise for really good and
reflective practice do need to be set alongside whatever emerges from more
abstract research, more fundamental research.’ (LEA Director of Education)
This stakeholder then outlines how such research can still be rigorous, if not
‘scientific’,
‘We have got a number of teachers working on small scale action research
projects about learning in their classroom and we have two people working
alongside the teacher. We have an advisor and an outside expert. The advisor and
the outside expert provide the frame. The teachers then take the frame and devise
projects within their own classrooms and are in the process of coming back and
the advisor and the expert would try and shape those in order to identify the
general messages that cab be drawn from it. So that’s the model really.’ (LEA
Director of Education)
More specifically their research findings are limited in terms of their transferability
and their generalisability,
‘I don’t mean to say that the action research that people have done has not
greatly benefited them, but it has been a this level of personal enquiry not at this
level that has the potential for greater generalisability.’ (HE researcher and RAE
panel member)
51
4. Building educational research capacity
This stakeholder tends to disagree and, instead, acknowledges that there are some, if
not many, practitioners who should be encouraged to undertake more ‘academic’
research,
‘That’s more complicated and its got to remain multi-layered and multi-faceted
so they can go onto do PhD’s and spend their life straddled between schools and
research. I’d like to see a bit more of that. I think the more we get of those it will
have a good spin off on the research and it has a good spin off into schools. But
there are only so many of those kinds of people about.’ (DfES Officer)
This stakeholder continues by proposing that the way practitioners get involved in
research should follow the Canadian model,
‘It’s a sort of Canadian model where you employ somebody to a service, then
you put them into the LEA, then you put them into the school board or whatever
its called and then you put them into the school. Now and it works well where its
been initiated because they love a better understanding of the different contexts
and there isn’t so much of this ‘us and we’ stuff which you get which is very
unhelpful. But its only going to be a limited market, there’s only going to be so
many people who are going to be that flexible and want to do it.’ (DfES Officer).
It is a perception, though, that there is only so much research practitioners can
undertake, particularly with current pressures on their time,
‘Field-based projects involving teachers are actually much more difficult to
deliver in the UK in the present time, given pressures on practitioners.’ (HE
researcher and head of department).
Another stakeholder agrees that the notion of practitioners’ involvement in research
should not be ‘romanticised’,
‘I think there is a lot of hokum taught about the idea that teachers are going to
spend month after month after month doing research and that the research that
they are in a position to do is going to be vitally important and that they are
going to, I'm sorry, even in the EPPI centre sense, that they're going to go onto
databases and work their way through research findings. I think you've got to be
careful not to romanticise the whole business of teaching. But nonetheless the
idea that in some form, probably in some mediated form, they are going to be
doing that, I think is a sensible one. I'm all in favour of that all inclusive notion.’
(HE researcher and member of the NERF).
The mediated form of practitioner-research is certainly the model projected by the
majority of stakeholders. This would be to involve them in research alongside more
experienced and formally trained researchers,
‘The model I prefer there or the models are the ones that involve them with
researchers in teams like the networks are doing and like the EPPI reviews are
doing actually and so on. Because it grows then in a safe system with people who
you know will nurture them sensibly and not send them off to do something
completely outrageous and then tell the world, you know generalise from it in a
way which is totally unacceptable.’ (DfES Officer).
52
4. Building educational research capacity
In turn this could ensure that research could have as useful and significant an impact
as possible,
‘I think people are trying to reach practitioners to a large extent but I guess that a
lot of them, unless they’re in big projects like in TLRP or actually doing it on a
very small scale, you know, they’re sort of grabbing two teachers in the local
primary school and two from the secondary, local secondary school, and that’s
fine but it doesn’t go any further and in a way, the TTA, certainly in England has
exacerbated that by giving these sort of studentships to people or scholarships to
people, saying you know, get on, do your own little bit, such and such, fine, but
actually that doesn’t address that the bigger thing, which is how do you actually
get practitioners to work with researchers together on something rather in it
being something that happens at the end.’ (HE researcher)
Another way of building research capacity via changes to the organisation and
composition of educational research is to develop the role of the LEAs in research.
Although it is felt that capacity for research in LEAs has increased recently it has
tended to be limited in its scope – responding to pressures from central government in
meeting particular requirements,
‘Then there was the National Curriculum in 1988 and all the things that were in
place like that, you know. There was little capacity on the part of the LEAs to
analyse that data in any sort of sophisticated form and clearly that is an agenda
which successive governments and different people in persuasion have kept faith
with and so probably there is now quite a lot of capacity for a kind of analysis at
a local level in authorities, quite strongly supported from the centre […] What I
worry about is whether there is a genuine research capacity within authorities
now, or whether people are actually being supported to do very particular tasks.’
(Former LEA researcher).
Alternatively, there is the view that LEAs contain little, or no research capacity at all,
‘I would say in general we have almost no research capacity, I would say we do
not have a research capacity… Our research and information manager – research
really means in that case gathering information.’ (LEA Director of Education);
‘LEA research officers - I didn't know we had any. I don't know we do actually.
We hire people to do things.’ (Chair of a LEA Education Committee and Local
Government Association executive member).
There is clearly a great deal of variation in the research capacity of LEAs that could
be used to help build research capacity more widely. It must also be acknowledged
that LEA officers may also lack formal research training,
‘Some authorities, I know two both of whom have school improvement teams.
One has a team within school improvement labelled research and that maybe to
do with the analysis of quantitative data and so on and another local education
authority might also have a school improvement team, but won’t have anyone
who could be identified as a researcher. Sometimes it is just simply how you
describe these people. Are they gathering information or planning, or what have
you? Some authorities still see research as something that they go to universities
for. So they might see research as commissioning questionnaires as market
research, they might go to MORI to look at parental attitudes to secondary
53
4. Building educational research capacity
schools or something like that, or reasons for choice of school, or they might link
up with a university education department, with a particular focus on, for
example, the needs of bilingual children, or so on. Sometimes the very word
research is quite loaded and it doesn’t happen within the LEA. Others will see
that as integral to their function. It is hard to get away from.’ (Former LEA
researcher).
But even if there is limited capacity for LEAs to undertake research it may be
necessary for LEA officers, as pseudo-practitioners, to develop their capacity to
‘consume’ research, with all the knowledge needed to be able to critique methods and
methodologies, and to go on and apply suitable research skills in their work,
‘Ideally, we should have a greater research capacity. Every year we look at all
our results in terms of national curriculum assessments and examination results
and we do a rough and ready analysis of it. We just ask some obvious questions
about it. We do actually have quite a lot of data on pupil achievement going back
over many years. I’m sure it’s a completely under-used source of data, but we
certainly don’t have any capacity for someone to look at all those and really
think about it and ask some interesting questions about it and come up with come
conclusions about it.’ (LEA Director of Education);
‘The need for an authority to take a strategic role to identify schools that are
failing, or are in difficulty, or who need additional support, we can provide that
support. The LEA is a key player there. You need to have a range of skills,
which brings me back to what do we mean by skills, and research. They need to
have a whole range of skills, in those LEA teams. Some of which you might
describe classically as research but others which are very much more practically
focused, which could have a definition in research?’ (Former LEA researcher).
These suggestions have shown how stakeholders would develop educational research
by focussing upon the composition and organisation of the educational research
community. To do this they recognised, in particular, the importance of the
relationship between educational research and social science research, and the need to
foster greater inter-disciplinary and multi-disciplinary research. They also addressed
the issue of economies of scale within, for example, education departments, regional
centres of methodological ‘excellence’, and research teams. Subsequently, three
groups or sources of researchers were seen as having a significant role to play in
extending the range of research skills and backgrounds to individual research projects:
new career researchers, or contract researchers; practitioner researchers; and LEA
researchers. However, nearly all of the stakeholders believed that developing the
research capacity of new career researchers would make the most significant
difference to educational research. It was also recognised that a number of approaches
to building their research capacity were needed. These included: the desire to bring
new career researchers from a variety of different disciplinary backgrounds to work
together; to get such researchers to work in relatively large education departments or
research teams whereby they would be exposed to a greater range of methods,
methodologies, theoretical developments and approaches to research; and that a
number of mechanisms should be put in place to encourage their continued research
development and to foster continued genuine inter-disciplinary research where
necessary.
54
4. Building educational research capacity
Behind many of the stakeholders’ suggestions presented in this section was the
assumption that these would only be effective if individual researchers were better
trained and encouraged to undertake educational research in a more ‘social scientific’
way. The discussion now turns to this role of ‘capacity-building’ for individual
researchers.
4.6 Building individual research capacity
All the previous suggestions for building educational research capacity have tended to
focus upon the system and structure of educational research. These are important, as
many of the obstacles to the development of educational research, identified by
stakeholders, were seen to be heavily embedded in the wider education system.
Consequently, there was a perception that to reduce these constraints could only be
achieved in the medium to long term, and beyond the capability of individual
researchers. However, a significant constraint on improving the quality and relevance
of educational research was that of the research capacity of individual researchers.
There was an overwhelming perception that at the end of the day not only was it
necessary to remove obstacles and provide mechanisms to develop research capacity
there was also a need for the majority of, if not all, educational researchers to consider
the way they approach and undertake educational research. This final section outlines
some of the key features, raised by the stakeholders, for individual capacity-building.
As discussed earlier there is a balance between the freedom of researchers and the
need to promote particular methodological developments. However, it was realised
that ‘capacity-building’ should be based upon notions of ‘potential’ and ‘opportunity’.
Although some stakeholders take the view that some coercion is necessary, this
definition of ‘capacity-building’ gives greater priority to the need to ensure that
researchers have the greatest opportunity to develop their own research capacity. It
may involve the promotion of particular capacity-building activities, but should
primarily use ‘carrots’ rather than ‘sticks’.
The areas that stakeholders raised for individual capacity-building were almost
entirely based around research methods and methodologies, and the need for
continuing development of such skills throughout researchers’ careers. In particular,
they focussed on: issues relating to research design; the need for more quantitative
research; how to overcome a perceived qualitative-quantitative schism; addressing the
relationship between qualitative and quantitative methods, i.e. combining and/or
mixing methods; and suggestions for new innovative approaches and techniques that
can help to build capacity in educational research.
In terms of research design the stakeholders are clear that the basic principles need to
be addressed. For example, many stakeholders made the point that the methodological
approach and choice of methods has to be determined after a research question has
been identified and not to allow research to be ‘method-driven’,
‘My view is that they have to decide what question they are interested in and
then borrow the methodology from wherever in order to address that.’ (HE
researcher and RAE panel member).
55
4. Building educational research capacity
And, as this stakeholder reminds us, this has to be combined with having an
appropriate research question in order to ensure the research is relevant in the first
place,
‘There is a dual process I think of arriving at appropriate questions and then
determining the kinds of answers and methodologies that fit those.’ (OECD
researcher).
The point about appropriate research questions cannot be underestimated. As has been
discussed before, a number of stakeholders argue that this, itself, needs to be
addressed,
‘So the first thing to say is that one has to limit it to questions that are
researchable. I think the problem with the poor quality end of your continuum is
that very often people try to do things that are not possible and it is inevitable
that it is going to be low quality stuff.’ (HE researcher and RAE panel member).
Behind the need to choose methods and methodologies that are appropriate for the
questions asked, i.e. a ‘fit for purpose’ model, is that the stakeholders are concerned
that too many researchers are ‘mono-methodic’. In other words researchers tend to
employ a single method or approach to their research,
‘I think on balance it’s a good thing to be problem driven. What did they say
about single methodology people – give a child a hammer and everything
becomes a nail. They don’t want people seeing the world through the
methodological lenses that you put on in university fifteen years ago. Sounds a
bit hard I know. But if knowledge is determined in parts by the ways in which it
has been generated then you’ve got to have access to different ways of
generating knowledge, at the very very least what I’m calling a passive
competence.’ (Director of a research organisation).
These are very general, and arguably basic, suggestions, but their importance is
evident in the number of stakeholders who raised them. A more specific way that
education research could be developed by focussing upon the design of research
relates back to the previous section that discussed the importance of the organisation
and composition of the research teams,
‘I think what's a fairly constant is person hours on the job. You need a lot of it
but the system out there needs a quick answer and I don't think we've thought
through designs that give people quick answers. Whole range of questions about
research management. So if someone gave me a million quid I don't think I
would see myself appointing one researcher for seven years, I think I might see
myself appointing ten researchers for one year. Because if you did that, you'd
just do everything quicker. You'd have the capacity to bring to bear on a quicker
result.’ (HE researcher and member of the NERF)
This view not only addresses the need to make research more relevant and useful to
the ever-changing policy and practice domains, but it would also ensure that research
teams benefit from economies of scale. The option of having relatively larger research
teams ensures diversity of research skills and knowledge, with the implication that
this would produce better quality and more rigorous research.
56
4. Building educational research capacity
Indeed, many stakeholders discussed the need to incorporate a variety of methods
within the research design. This is discussed in more detail later on. However, the
only ‘type’ of research design mentioned by stakeholders in terms of capacitybuilding was the place of randomised control trials (RCTs) in educational research.
There was a general belief that greater attention is needed to the use and application of
RCTs in educational research,
‘The MRC have done some interesting research in Scotland on sex education
which is using (RCTs) and I think it was very, very interesting research and
that’s been, that must have been around a long time around, I mean, it’s about
five, six years goes back, so I think, you know, there are clearly arenas in which
you can use it. My guess is what we actually need is precisely, is to say in what
areas we think it might work in education and where it doesn’t, where if you get
the sort of, you know, this is a wonderful thing and it’s going to transform
everything we do, it’s not going to transform everything we do, anymore than
anything else is going transform everything we do, like, you know, computers or,
you know, the web.’ (HE researcher).
As identified earlier one of the most significant ‘deficits’ in educational research is in
the use of quantitative methods. It is not surprising, therefore, that nearly every
stakeholder addressed the issue of building capacity in quantitative educational
research. Not only would this help to extend the range and balance of methods in
educational research but some stakeholders could see how this would help develop
better quality research irrespective of the methods used,
‘There is a real problem about the shortage of people who can do quantitative
work and you can use that to get people to think in a more rigorous way about
the logic of social research generally. Because quantitative researchers have to
think formally and much harder about things like measurement, indicators,
reliability, sampling, all the things that all researchers actually ought to think
about, and Bob Burgess, who started this thing in the training, with the training
board, was trying to get, but he couldn't figure out how to do it, because when
people do non-quantitative research they tend not to think nearly as rigorous
about the kind of problems that you talk about in the paper [Gorard, 2001] and
about problems of generalisability and learning experimenter effects and the
nature of intention and blah blah blah.’ (Head of a research funding body).
Many stakeholders believed that building capacity in the use of quantitative methods
is not just about increasing the volume of such research, but should focus, in
particular, on the more basic level of ‘quantitative’ techniques,
‘I think I agree with that general thrust that we need more quantitative
researchers, but I'm not sure it’s the very, very, sophisticated techniques we're
talking about. Its more something that’s a bit more accessible and usable.’ (HE
researcher and TLRP Team Leader).
This also requires a firm grounding in the assumptions and principles behind many
‘quantitative’ methods. This, as many stakeholders are keen to point out, may be
necessary for those who already undertake quantitative research but have been trained
in ‘button-pressing’,
57
4. Building educational research capacity
‘Well I suppose the concern hat I have is that people should first of all
understand what quantitative research is doing and as far as they can get a grip of
how to do it. One of the things that has concerned me over a number of years is
that when people go and learn their statistics, very often, they will then go and
hop off and learn SPSS and type the numbers in and do it without understanding
the assumptions that underpin the statistics and I don’t know how to do it, but
nevertheless it does seem to me that it is very important to get them to
understand what the implications are of not paying attention to the assumptions.’
(HE researcher and RAE panel member).
The consequence of educational researchers building capacity in just the mechanics
and procedures of quantitative techniques is clearly outlined by stakeholders,
‘I think you can get terrible quantitative work, there are people who just think
there's a kind of non problematic general linear model reality out there and you
just tag variables and start with race and that’s it… We don't want a generation
of people who are trained, kind of cloned, trained monocular vision, kind of
stimulus response, there are three variables, must be a log linear analysis, switch
on SPSS, press the buttons with all the defaults on is garbage in garbage out.’
(Head of a research funding body).
However, a number of stakeholders believe that concerns about the quality of ‘buttonpressing’ quantitative research could be misleading. They argue that very little of this
research has been undertaken, let alone more rigorous and better conceptualised
quantitative research,
‘I don’t think we’re getting what some people have been afraid of in the past,
which is things like, that, because you can get SPSS for Windows, people can
press buttons and it will be crap research. No, I don’t think we’re getting that. I
don’t think we’re getting those published anyway and I don’t see too much of
that getting even put in, although I do think there’s some filtering.’ (HE
researcher).
Irrespective of the level of quantitative sophistication applied in research, the
stakeholders all believed that there needs to be a greater capacity across the whole
education research community in being able to ‘consume’ these techniques and to
have the ability to critique the data, methods of collection and methods of analysis,
‘I do think the quality of the work they do in the UK, not just education, but
across the board, would be transformed if more people could actually criticise
the work rigorously and seriously and actually read an article in the ASR
[American Sociological Review] and say not say I don't understand this but say
I've got real reservations about this and I'll tell you why and then give you a
coherent story about problems and the way the data are constructed and possible
sampling error, the way the measurement done, whether you know you can use
that technique on these kinds of data, that kind of thing.’ (Head of a research
funding body).
This also applies to the more sophisticated quantitative techniques being developed
and utilised in educational research,
‘You may not be able to use the methods, I don’t expect everybody to do multilevel modelling but I do expect them to understand what it is all about, why it is
58
4. Building educational research capacity
important as an advance that has been made in the last fifteen years.’ (HE
researcher and RAE panel member).
However, this should not detract from the need to address the deficit in quantitative
research directly. And, as this stakeholder argues, this must be achieved without
neglecting the issues of appropriateness and relevance discussed above,
‘But I suppose the thing that worries me at the centre of all this, is that, there’s a
kind of sense in which, yes it’s, it’s a deficit model but it’s also a kind of
argument that says, well unless it’s quantitative, it’s no good and what we
presumably want is for people to do good quantitative work when that’s
appropriate and not to do it when it’s inappropriate.’ (HE researcher).
One of the key obstacles, identified earlier, to building the capacity for quantitative
research is the difficulty in overcoming issues of ‘identity’ between different
methodological approaches. The few quantitative researchers amongst the
stakeholders are all too aware of this and how ‘being’ a quantitative researcher has its
problems,
‘There’s actually an anti-quantitative not just kind of ignorance. In some quarters
there are people… people like me… we get labelled as positivists. This is a term
of abuse. That doesn’t help and it can be… I think there was a period in my own
institution and this is highly selective but there was a period five or six years ago
here with I know there were people going around criticising any kind of
quantification. They were criticising work that I and colleagues were doing as
being positivist in a kind of derogatory sense […] Which is not to say that there
aren’t some people who are already abusing quantitative techniques and that
deserves criticism but you can’t be against quantification in general because
there are few examples of its abuse.’ (HE researcher and RAE panel member);
‘It's all nonsense, all of it. I'm fed up being labelled a positivist by people who
wouldn't know positivism if they had it in their soup.’ (Head of a research
funding body).
These comments illustrate the hurdles that need to be overcome in order to build the
capacity in educational research to ‘consume’ and use quantitative methods and
methodologies. In some respects the stakeholders believe that research capacitybuilding needs to address these issues of methodological identities just as much as it
needs to train researchers in quantitative methods,
‘If I’m looking at the future then I think, at least we need to explore how
identities are formed and if we’ve got people in the team who have the capacity
for dual identity then we need to support them in developing a dual identity by
managing the things and the project in ways that will actually support them in
that.’ (HE researcher and TLRP Team Leader).
As the stakeholder above notes this requires research project leaders to encourage new
researchers to develop their skills in both quantitative and qualitative research
methods. Another stakeholder states that there should be no resistance to using either
quantitative or qualitative methods, arguing that ‘fit for purpose’ is the approach that
all educational researchers must take,
59
4. Building educational research capacity
‘I actually think there are two ways and for me I just think those images that go
with the different images is a complete travesty. I think they are just different
ways of doing research. Sometimes the project’s appropriate to use qualitative,
sometimes its appropriate to use quantitative methods and sometimes you want
to use both. Just like sometimes you want to drink red wine and sometimes you
want to drink white wine. That’s it. Sometimes you want to drink everything.’
(HE researcher and TLRP Team Leader)
However, how and when to use both qualitative and quantitative methods in research
is not straightforward. As this stakeholder points out there are few examples of how it
can be achieved,
‘I also think there’s a thing about kind of using mixed methods and people not
always knowing how to use mixed methods and there isn’t, I mean, there is some
mixed methods research but there’s not a huge amount of it actually.’ (HE
researcher).
Even developing the capacity of new researchers in both qualitative and quantitative
methods, as the ESRC’s new training guidelines attempt to do, is not without its
problems. This stakeholder is very aware that just as it is important to understand the
appropriateness, rationale and limitations of particular methods so to is it necessary to
understand the appropriateness, rationale and limitation of mixing and/or combining
qualitative and quantitative methods,
‘You shouldn't necessarily be trying to produce, you know, ‘identikit’
researchers off a production line.’ (Head of a research funding body).
Although the quality and rigour of qualitative research was questioned earlier in this
report, very few stakeholders discuss the need to build capacity in qualitative
research. The only exception is this stakeholder who argues that being able to do
rigorous qualitative research is just as hard as developing more numerate skills,
‘We’ve assumed that numerical is the terrifying… the statistical is the terrifying
when in fact that’s not entirely the case. Certainly people I’ve dealt with have
been very frightened of raising categories from the [qualitative] data by
constantly re-reading and allowing the categories to arise from the data and then
trying them and testing them and doing that sort of analysis on conversational
data or other data. They’ve been just as frightened of doing that.’ (HE researcher
and research training officer).
Many of the stakeholders are keener to suggest that individual researchers need to
consider building the capacity to develop and use relatively new and innovative
research methods and methodologies. These included the greater use of ICT, and, in
particular, the techniques and technologies to analyse video footage,
‘One should try to, sort of, it’s not just quantitative methods, it’s things like ICTbased stuff. I have a, an interest, for example, in using ICT. I’ve long thought
that, for example, there are forms of video investigation that we’re just not using
and it’s sort of thing when you say, you say Christ, you know, it’s Star Trek or
something.’ (HE researcher)
60
4. Building educational research capacity
Similarly, this stakeholder suggests that there is a need to develop the skills and
knowledge on data collection methods,
‘But it’s not just quantitative data analysis, it is about data gathering and that is
one of the key things about education, it’s so messy, real life is so messy, that
you need unobtrusive, you know, ethically questionable [laughs], data gathering
methods, you know, and triangulation at that low level and so forth, and then you
need to get to grips with it with all sorts of analysis, whether it’s, you know,
qualitative naturalistic human head, sort of stuff, or right through to structures
and comprehend complexity type of thing and so forth. So I think there are those
things, there are those things like, no, like the sort of data gathering level, as
opposed to the analysis level, which are easily forgettable but we really should,
so…’ (HE researcher).
In particular, the stakeholder argues there is a need to consider how research can
approach and investigate more complex processes and phenomenon. This, the
stakeholder outlines, includes what data is needed, how it can be collected, and how it
can be analysed,
‘You know, it’s the, you know, generic things and broad measures, even with,
with, sort of, quite complex multi-variate stats, certain, certain sort of techniques,
they’re not going to tell you very much because they don’t get at the dynamics,
so things like complexity theory, dynamics and all that sort of stuff, I think
sooner or later we’re got to get, that’s, if anything, we’re going to get into that.
Meanwhile, it’s things like, the human access to that, through what you might
call ethnographic, sort of stuff, but, as far as I can see, we’re not yet getting
towards, the sort of data collection, which is going back to the data collection
stuff, sort of data collection that could some how be fed into, you know, complex
but more specific types of analysis of the, of the sort of, types I was mentioning,
like, say, networks, neural networks being put to work on this. I mean, you
know, neural networks to pick up the regularities in classrooms, for example, or
modelling, non-linear dynamic type modelling, but what do you put into it, sort
of stuff, and where do you get the data.’ (HE researcher).
The suggestions presented in this section for ‘capacity-building’ have all focussed
upon the individual researcher and the application of particular research methods. In
particular, there is a near consensus over the importance of increasing the volume and
appreciation for ‘quantitative’ techniques in educational research. In this section the
stakeholders also started to differentiate between the need for skills to use and to
‘consume’ such methods. The next section discusses this differentiation further with
respect to research training and the need for continued professional development for
new and existing educational researchers.
4.7 Research training and continued professional development
This final section outlines some of the thoughts and suggestions that stakeholders
have in relation to research training. In particular, it covers the form that research
training needs to take and which educational researchers it is primarily for. The
discussion concludes by stressing the need for continued professional development
and for capacity building activities to have a non-formal role in research training.
61
4. Building educational research capacity
As discussed earlier in this report, one set of factors that are perceived to limit the
development of educational research are the non-traditional and varied career paths
into this research. For the majority of academic disciplines the PhD or doctoral
research is the most significant formal training a researcher receives. Clearly one way
to increase research capacity in education could be to increase the number of doctoral
students. However, very few stakeholders saw this as a principal way of building
capacity in education,
‘The fact is there are plenty of them. There might be fewer, but there are plenty
of them. I personally train two a year for example, I've put couple a year through
for years and the fact is they then disappear into a system. They do their
doctorate and they go and do something else like school teaching or something.
Many of the ones I've trained are already schoolteachers and they do perfectly
good doctorates and then they go off and become headteachers or what have you.
So, you could double that, you could treble it. But if they are all going to walk
through and walk off, enter stage left leave stage right, I actually don't see the
point […] I mean it's hardly got anything to do with quantity […] To just churn
out more doctorates is a… you hear it right through out: here [the UK] we don't
have enough engineers, we actually train twice as many engineers per capita than
the Germans and so why do they do better in engineering than we do? Because
they use their engineers better.’ (HE researcher and member of the NERF)
One stakeholder actually considered this approach in a previous attempt to build
capacity, and concluded also that PhDs are costly,
‘Because basically they’re so costly, they would have eaten, this was, this was a
six figure sum, but they would have eaten into this very, very fast, if you’d done,
you know, full-time studentships over three years. And so we didn’t go, we
didn’t go for that, we went for other things including a series of more or less
annual conferences bringing more experts in and so forth.’ (HE researcher)
So if increasing the number of doctoral students may be an inefficient way of using
resources to build capacity is there some way that doctoral students can become better
trained? It has already been made clear that some stakeholders have reservations over
the quality of doctoral research training. As this stakeholder argues, with the example
of EdD training, there is a tendency for new researchers to undertake small-scale
qualitative research because of research constraints and limited training,
‘The obvious way, the natural way to get around it is to have the two skills in the
one person and that’s what the training is supposed to provide. Whether the EdD
training will provide that is another matter. I haven’t seen any evidence of it in
EdD’s that I am an external for. They are very qualitative and again you can
understand it because if an individual EdD student, when they come to their
thesis, they are not going to be involved with large scale sampling because they
haven’t got the money or the time.’ (HE researcher and UCET Executive
Committee member).
Similar reservations about social science research more generally have emerged from
the ESRC, leading them to formulate their 1+3 scheme with a greater emphasis on
acquiring a range of research skills,
62
4. Building educational research capacity
‘They can see that there is actually nothing wrong with saying to social scientists
who've had seven years of public money or aid, if they were in Scotland, that
should be able to come out after eight years of study with more than one
technique that they can use.’ (Head of a research funding body);
‘As you probably know ESRC have required people trying to have their courses
recognised for ESRC recognition to build in research training into those research
programmes and I think the training is absolutely excellent for what I’ve seen.
Not on the ground I mean but in theory. That someone who went through that
training would have a very wide range of research skills. But people like me
haven’t been through that training. We’ve picked up what we need on a need to
know basis virtually and as you get older I guess or lazier the tendency is to go
where you know.’ (HE researcher and journal editor).
As this last stakeholder points out, however, this attempt to build capacity may only
work in theory. What actually happens in reality is yet unclear. There are also two
further limitations of this approach to educational research. First, it does not address
the needs, or capacity, of existing researchers, and second, this may further compound
the problem of whether the new scheme can be delivered since existing researchers
constitute the providers and trainers of this scheme. As the stakeholder continues there
is probably a need to undertake some quality assurance of doctoral research training
so as not to intensify the existing problems with educational research for the future,
‘I think it’s, so, I think that you’ve probably got to assess and you’ve probably
got to have a closer look when you do your inspection. I mean the ESRC don’t
really inspect, they believe what you say, sort of thing, and then they, they look
at the qualifications and the plausibility of, of the people you put in but you
probably need some sort of…’ (HE researcher and journal editor)
An alternative model for formal research training in education may be the IKON – a
collaboration between the Institute of Education, King’s College London, Oxford
University and the National Foundation for Educational Research (NFER). This not
only uses greater economies of scale to ensure that the relevant expertise is utilised in
the training, it also involves a one-year placement on a relatively large-scale research
project. This may mean that the student has access and the opportunity to use a
number of research methods and to be exposed to a variety of approaches to research
from working in a research team – opportunities that are not normally available to a
traditional three-year PhD student.
As many of the stakeholders argue ‘capacity-building’ must also target existing
researchers. As discussed earlier this means that educational researchers must identify
their own limitations and needs. They then must be given encouragement to build
their individual capacities,
‘If that article was right that most of us are in our early 50’s or early to late 50’s I
think the boat’s been missed as it were. We’ve got to work from the situation
we’re in now. So there has to be some ‘carrot’ to allow researchers to put their
reputation on line and say “I actually don’t know what regression analysis is or
whatever, I don’t know how it can be applied” […] The other way around is to
encourage researchers of my age to go for the subject… the skills that they
haven’t got so that they create teams which have those skills in them, which are,
if necessary, cross-departmental or cross-institutional. In my own case that’s
63
4. Building educational research capacity
what I’ve done, I’ve picked up them up across the institution, because they don’t
exist in the department.’ (HE researcher and UCET Executive Committee
member)
As the stakeholder above suggests this can involve ensuring the educational
researcher has the specific skills around them, in their research team, or in their
department. Alternatively, they should be encouraged to participate in further research
training. The model proposed by stakeholders is no dissimilar to traditional continued
professional development (CPD). From some of the comments presented earlier in
this report this appears to be a considerable weakness of the research community and
HEIs more generally. An example of how CPD could work for researchers can be
found in the National Foundation for Educational Research (NFER). As one of the
stakeholders qualified,
‘If you look at NFER, for example. They’re always getting research right, the
staff development there is quite an impressive model and we recruit quite
actively from NFER for research analysts in here because they are properly
trained. They don’t give us dumb people who say they can do quantitative data
analysis but then when you give them some they can’t do it.’ (DfES Officer).
As the Director of the NFER outlined, their CPD begins with an appraisal of an
individual’s needs. These needs then form the basis for their upcoming years’
priorities for staff development. The NFER then ensure that they can offer the
required professional development activities. All throughout this process there is
dialogue and agreement between the individual researcher and an appraiser,
‘Its part of the culture even if when people are very busy the assumption is that
you should be doing some professional development. Everyone has an annual
appraisal which is focussed on staff development and needs. There’s a written
document at the end of that outlining what you’re going to do in the coming year.
Looks at what you’ve done the previous year, outlining what you want to do in
the coming year and how you will be facilitated by us in doing that. That has to
be signed up by the appraiser and the appraisee. Now to give that a bit of more
substance: We have a range of professional development activities here which
we have developed ourselves. There are modules on things like interviewing
studies, case studies, developing questionnaires. We provide a range of courses
on the statistical front from remedial stats, “what is a mean?”, through to multilevel modelling. And those are organised on an ad hoc basis as people come in.
Whenever there is a significant new IT application we run a training event.’
(Director of the NFER).
But the success of this still depends upon the extent to which individual researchers
wish to develop their methodological skills and to stay abreast of developments in
their substantive areas,
‘For existing staff there is an expectation that you’re going to be updating your
knowledge both substantively if you’re an expert in given areas, they are
expected to keep abreast of those areas over and above the updating that will
come from being engaged in projects because if you’re working on current
policies issues, which a lot of people would be, you’ve got to keep up-to-date
there. But more particularly on the methodological areas and things like writing
workshops.’ (Director of the NFER).
64
4. Building educational research capacity
The need for greater CPD for educational researchers should not detract from other,
more informal, means of acquiring new, or develop existing, research skills. Many
researchers would argue that they already do this on an ad hoc and intuitive basis.
‘Capacity-building’ activities, therefore, must be based and designed around the
formal CPD model and the more informal intuitive model to developing research
skills and knowledge.
Throughout this report stakeholders have repeatedly talked about the need to develop
skills to understand and critique research as well as to develop skills to undertake
research. This distinction is useful in terms of the ‘capacity-building’ needs of the
entire educational research system, since many stakeholders have called for the
‘users’ of research, including practitioners and policy-makers, to be able to ‘consume’
existing and future research. The distinction is also useful within the research
community itself. As this stakeholder argues, even if the composition of a research
team is organised around a division of labour, where each member has expertise in a
relevant area, there is still a need for all members of the research team to be able to
understand their colleagues,
‘It’s our experience and belief, or maybe it should be the other way around
because we believe based on the experience, that we are more likely to be able to
do that well more often if we have this division of labour. I suppose there are two
kinds of animal here. One are the people who are an expert either in a substantive
field or in research methodology, or there are people with a broad range of skills,
at best with an active competence if I can make that distinction across the board,
but certainly at least having a passive competence in the ones… I don’t want a
qualitative researcher who can’t have an intelligent conversation with the
statistician and won’t understand when the statistician says “no”, that they will
understand why they are saying no.’ (Director of an educational research
organisation).
It is also necessary to take this approach to building capacity just to ensure that
individual researchers have a basic level of ‘competence’ in new methodological
developments, even if they or their research team will never actually use these
methods,
‘I think we’re at a point where in fact there is a huge amount of new ways of
looking at things. I mean within qualitative field the effect of using data analysis
packages like NUDIST and so on. The effect of constructivist methodologies and
PCP and all these other things that people use and the concern that in order to be
a literate researcher you need to know about these. You need to know what they
do. You need to be able to speak reasonably knowledgeably about them and then
you need also to be able to speak knowledgeably and to be able read about
statistical surveys and all sorts of other quantitative analyses of data and at the
moment I don’t think any of us can really do it. I certainly can’t do it as well as
I’d want to be able to do it. And I think the next generation of researchers has got
to be able to do this.’ (HE researcher and research training officer).
However, alongside this particular model of building capacity to ‘consume’ methods
and methodologies should be a desire to show, with examples, how such methods can
be used. There is, as the following stakeholder indicates, a danger of ‘under-training’
65
4. Building educational research capacity
researchers such that they falsely apply methods or pay lip-service to new
developments,
‘The real problem is I think that if all you ever do is teach people the basic stuff,
they don’t know about… they don’t know when they need to use it and they go
to a specialist. So what you have to do, while you can’t teach them all the
technicalities, what you can do is give them some feel for what is involved in
doing something more sophisticated, more complex nature and that’s very
difficult. I suppose it’s easy to say but what you should be doing is providing
examples of where this makes a difference. There’s nothing like a kind of few
high profile cases to encourage people – “oh yeah that really worked in that case,
I think I ought to know about that.” It’s actually worth paying some serious
thought to whether one can come up with any of these sorts of things. They do
tend to be very persuasive.’ (HE researcher and RAE panel member).
This section has begun to outline how ‘capacity-building’ activities could be delivered
and in what form they could take. In particular it stressed the need for such activities
to address the needs of existing research staff as well as ensuring that new researchers
have been appropriately trained in a range of research methods and methodologies. It
also made a distinction between the needs of researchers to ‘consume’ methods and
methodologies and the need to use such tools.
It was clear from this discussion that individual researchers must identify their own
needs and limitations. This entire report has tended to reflect the needs and limitations
of the entire research community. Although these have significant importance in the
current debate on the quality and relevance of educational research generally it is
important to note that for sustainable research development a culture of continued
professional development needs developing amongst educational researchers.
4.8 Conclusions
The thoughts and suggestions presented here for building educational research
capacity have begun to outline the areas for improvement. Although in the previous
section the stakeholders identified many structural constraints to developing
educational research they tended to suggest ways in which individual researchers
could contribute to this process. As discussed at the beginning of this section these
suggestions are not definitive but they do offer guidance on the areas of priority for
building educational research capacity. They also help define what ‘capacity-building’
is, since the approach of the RCBN has been to use its consultation exercise to set the
objectives.
The stakeholder interviews have helped to set out five areas of attention for building
educational research capacity. These are:
(i)
Improving relevance and impact – with the need to develop an ‘interactive
social science’ model of research, engaging fully with ‘users’ throughout the
majority of the research process, and the need to begin to understand the
process of knowledge generation more fully; this leads to the second area of
attention,
66
4. Building educational research capacity
(ii)
(iii)
(iv)
(v)
The need to generate better quality and more sustained knowledge –
particularly in developing marketable commodities or tools and artefacts,
informed by rigorous research, that can be applied in practice on the basis of
its qualified effectiveness, and the need to focus more on causal links in
education, which directly leads to greater theoretical development;
The organisation and composition of the educational research community –
with the need to undertake greater inter-disciplinary research, to work in
research groups with a breadth of research skills and approaches, and to assist
in the development of new educational researchers’ careers;
The need to enhance the methodological skills and knowledge of individual
researchers – from the design of research projects, through to using new or
combining qualitative and quantitative research methods, and in doing so
overcome ‘methodological identities’ that stifle debate and critique; and
finally
The need to improve research training, for new and existing researchers –
including the need for researchers to identify their own individual needs, both
to use and consume particular research skills, and then to provide the means to
meet those needs through formal models of continued professional
development and more informal intuitive models of developing research skills
and knowledge.
In effect these five areas of concern have defined what ‘capacity-building’ means to
the stakeholders interviewed. However, these suggestions should be interpreted and
addressed alongside the basic principles of ‘capacity-building’ discussed at the
beginning of this section. Those principles being based upon notions of ‘potential’ and
‘opportunities’ – in order to maximise the current potential for high-quality research
and to ensure that there is the greatest opportunity for the research community to meet
the demands of the educational research community.
67
5. Summary and conclusions
5. Summary and conclusions
‘It’s an interesting time to be a researcher.’ (HE researcher and UCET Executive
Committee member)
This report has presented the findings of interviews with twenty-five key
stakeholders, each representing the major constituencies of the UK education research
system. The interviews addressed three key issues in relation to ‘capacity-building’:
(i) What the state of educational research is currently;
(ii) Why it is like this; and
(iii) What can be done to further develop educational research
Given that many of the stakeholders tended to agree, at least in principle, with some
of the criticisms of educational research made by Hargreaves, Woodhead, Hillage et
al., and Tooley and Darby, it is perhaps not surprising that they focussed upon
problems with the current state of educational research. In particular they emphasised
the general poor quality of educational research, although many would agree that the
overall quality is mixed, but is improving. They also argued that the impact of
research, as opposed to the more vague notion of relevance, was a key area of concern
for the whole education system. However, all the stakeholders would agree that there
is some excellent and decent research being produced. But the perspective offered by
the stakeholders was, without being pessimistic, to argue that educational research in
the UK needed to and, incidentally, could be enhanced.
When asked why educational research was in this state the stakeholders
predominantly highlighted the constraints that have limited the capacity for highquality and useful research. These constraints ranged from structural problems
associated with the education research community, and the education system more
generally, to more precise methodological ‘gaps’ in educational research. Great
emphasis was placed on the roles of researchers and ‘users’, including policy-makers
and practitioners, in the process of research impact and dissemination. Although this
was not perceived to be the sole responsibility of researchers there was a belief that
there was little understanding of the mechanisms and processes in which research
findings can be transformed into knowledge that can be applied to the daily work of
‘users’. In turn ‘users’ must be able to ‘consume’ research methods in order to engage
critically with research.
The limited impact of educational research was believed to be compounded by the
size of the educational system and the slow pace of change associated with such large
systems. However, many stakeholders also outlined a number of constraints
associated with the culture of educational research. Such constraints included
problems brought about by a competitive funding environment and, ironically, the
greater production of poor quality of research as a direct consequence of the Research
Assessment Exercise (RAE). Although these factors may be out of the immediate
control of individual researchers stakeholders were also keen to stress that there are
too many ‘naïve’ educational researchers, with limited research training and hence
little knowledge or expertise in research methods and methodologies.
68
5. Summary and conclusions
The methodological limitations expressed by the stakeholders were wide ranging, but
the key concerns were:
§
§
§
§
§
§
§
§
the shortage of quantitative skills – both to use such skills and to ‘consume’
research that employs quantitative tools or methods;
the propensity for non-cumulative, small-scale and/or ‘flawed’ research;
the limited attention paid to causal links and the effectiveness of teaching and
learning artefacts;
concerns over the lack of rigour and lack of innovation in qualitative research
constraints caused by the friction between qualitative and quantitative
‘methodological identities’;
the limited or absent research ‘apprenticeship’ for many educational researchers;
the particular demands of education departments to train teachers at the expense
of educational research; and
the lack of continued professional development throughout a the typical
educational researcher’s career.
These constraints begin to identify the factors that may need addressing in building
capacity in educational research. However, it is also clear that in order to address
these factors requires the attention of the entire education system and not just the
research community. The discussion now continues by presenting the thoughts and
suggestions of the stakeholders as to how these constraints should be tackled and what
the priorities should be for building research capacity.
Although the stakeholders identified many structural constraints to developing
educational research they tended to suggest ways in which individual researchers
could contribute to this process. These suggestions are not definitive but they do offer
guidance on the areas of priority for building educational research capacity. They also
help define what ‘capacity-building’ is, since the approach of the RCBN has been to
use its consultation exercise to set the objectives.
The stakeholder interviews have helped to set out five areas of attention for building
educational research capacity. These are:
(i)
(ii)
(iii)
(iv)
The organisation and composition of the educational research community –
with the need to undertake greater inter-disciplinary research, to work in
research groups with a breadth of research skills and approaches, and to assist
in the development of new educational researchers’ careers;
The need to enhance the methodological skills and knowledge of individual
researchers – from the design of research projects, through to using new or
combining qualitative and quantitative research methods, and in doing so
overcome ‘methodological identities’ that stifle debate and critique;
The need to improve research training, for new and existing researchers –
including the need for researchers to identify their own individual needs, both
to use and consume particular research skills, and then to provide the means to
meet those needs through formal models of continued professional
development and more informal intuitive models of developing research skills
and knowledge;
Improving relevance and impact – with the need to fully understand the
relationship between research, policy and practice, and to identify ways of
69
5. Summary and conclusions
(v)
improving the use and impact of educational research; and this leads to the
final area of attention; and
The need to generate better quality and more sustained knowledge –
particularly in developing marketable commodities or tools and artefacts,
informed by rigorous research, that can be applied in practice on the basis of
its qualified effectiveness, and the need to focus more on causal links in
education, which directly leads to greater theoretical development.
In effect these five areas of concern have defined what ‘capacity-building’ means to
the stakeholders interviewed. However, these suggestions should be interpreted and
addressed alongside the basic principles of ‘capacity-building’ discussed at the
beginning of Part 4 of this report. Those principles being based upon notions of
‘potential’ and ‘opportunities’, in order to maximise the current potential for highquality research and to ensure that there is the greatest opportunity for the research
community to meet the demands of the educational system. Similarly they should be
modified in light of findings from other elements of the RCBN consultation exercise.
70
6. References
6. References
Gorard, S. (2001) A changing climate for educational research? The role of research
capacity building, Occasional Paper Series Paper 45, Cardiff: Cardiff
University School of Social Sciences
Hargreaves, D. (1997) In Defence of Research for Evidence-based Teaching: A
Rejoinder to Martyn Hammersley, British Educational Research Journal, 23,
4, 405-420
Hargreaves, D. (1999) Revitalising educational research: lessons from the past and
proposals for the future, Cambridge Journal of Education, 29, 2, 239-249
Hillage, J., Pearson, R., Anderson, A. and Tamkin, P. (1998) Excellence on research
in schools, Sudbury: DfEE
Hodkinson, P, (2001) NERF strategy proposals: a major threat to academic freedom,
Research Intelligence, 74, 20-22
Major, L. E. (2001) Don't count on us, Guardian p.9
Marshall, G (2001) Social Sciences, ESRC
Tooley, J. and Darby, D. (1998) Educational research: a critique, London: OFSTED
Woodhead, C. (1998) Academia gone to seed, New Statesman, 26 March 1998, pp.
51-52
71