Working Paper number 8.1
Research - Policy
Interaction
A study of a series of DfES
convened seminars
Lin Norman
December 2004
Executive summary
Background
This report makes a small contribution to the growing debate about how
research evidence can link more effectively with policy and practice in
education. It looks specifically at one area for potential impact - the interaction
between researchers and policymakers in the presentation and discussion of
research evidence. It is based on observations from a small NERF study of
five seminars convened by the then Research Coordination team at the DfES
between March and September 2004. The seminars provided opportunities
for researchers and policy makers to explore research evidence and to
discuss how it might be used in the policymaking process.
The report covers seminar purpose and design, pedagogic factors,
communications and potential impact. It identifies factors associated with
successful interaction at such seminars and suggests two areas for further
investigation which are likely to improve the process of interaction.
Success factors
Observations of the seminars suggest a number of factors for success:

a seminar leader who knows and understands the worlds of both research
and policy-making

a clearly stated purpose which relates to the planned outcomes of the
seminar. The purpose and desired outcomes then determine the design
and format of the seminar

the identification of the key policy issues around which the seminar is to be
built and research evidence selected. Defining the scope of the policy
area enables relevant research evidence to be presented

collaborative design of the seminar involving researchers and policy
makers. This fosters a balanced, problem-solving approach. Contributions
from both groups should be planned

papers sent out in advance to minimises unnecessary information transfer
at the seminar. This provides maximum time for exploration and
discussion of policy areas and evidence

identification and early introductions of participants to ease communication
and generate a collaborative approach

a room layout which creates an informal atmosphere and facilitates small
group discussion as well as presentations

presentations from both researchers and policy makers to highlight the
main policy issues and the key evidence
2

a seminar design which achieves more than the transfer of information.
Use of a range of activities in addition to presentations enables all
participants to be involved in debate and discussion. A workshop approach
helps to achieve active participation

gathering feedback from participants as a means of reviewing and
improving the interactive process
Challenges
During the observations of communication between researchers and
policymakers it became increasingly evident that this sort of interaction
presents challenges. Different working language, styles, time scales and
imperatives lead to different understandings of how to communicate about
research evidence in the context of the policy-making process. Appendix 2
lists examples of these challenges.
A wider context for the seminars
The effectiveness of the seminars would be greater if they were to form part of
a series of interactions at different stages of the research cycle, not solely the
final ones, and also corresponded to relevant stages of the policymaking
cycle.
3
Introduction
How can effective interaction between researchers and policy makers help
link research evidence with the policy making process? To further
understanding of both the question and possible responses, the National
Education Research Forum (NERF) undertook an exploratory study of a
sample of seminars for researchers and policy makers convened by the DfES
between March and September 2004.
The seminar series - background
In March 2004 the then Research Coordination Team at DfES launched a
series of seminars at which research evidence is presented to policy
developers/makers. The seminar series is an innovative development and is
part of an emerging strategy to strengthen the links between research and
policymaking. Although by themselves the seminars cannot ensure that
research evidence is used by policymakers, they make an important
contribution to raising awareness of research outcomes and developing
evidence based policy.
Seminar topics cover research evidence presented from a range of reviews
from the TLRP (Teaching and Learning Research Programme), the EPPI
(Evidence for Policy and Practice Information and Co-ordinating) Centre and
from other individual projects.
The DfES team responsible is using this study and its own reviews to develop
and improve the effectiveness of the seminars. As the series progresses
changes have been incorporated and additional strategies for successful
interaction are being put into place.
The study
The aim of the study was to identify features of effective interaction between
researchers and policymakers at the seminars. The working assumption was
that effective interaction could facilitate the process of feeding evidence in to
the policy making process.
A sample of five seminars was observed using a framework designed to
capture processes of communication and interaction. A sample of individuals
was interviewed before and after the seminar.
4
The seminars – observations
(In this section key features for effective interaction are italicised for quick
identification)
Purpose
The overall purpose of the seminars, currently, is to stimulate dialogue and
interaction between researchers and policymakers. They also provide an
opportunity for cross-fertilization across different policy areas at DfES.
For each of the five seminars observed the stated aims reflected the
developmental nature of the series. For example, in the early seminars aims
such as
‘to share knowledge and ideas’
‘an opportunity for policy people to engage with researchers…’
reflected the intention, at this stage, to develop the communication processes.
However other aims in later seminars suggest they have a wider purpose:
‘to explore implications of research findings’
‘to inform further research’
The stated purpose of a seminar informs the design and interaction methods
so the words chosen for aims and objectives are important. The design of the
seminars observed proved more effective for information transfer than for
discussion of the implications for policy. The latter could be an area for
development in future seminars.
Design and preparation
Design
A common format was used for each of the seminars. A DfES chair led with
an introduction referring to key policy areas and an outline of the major policy
issues. This set the context for the research evidence, and brought together
policy issues and evidence throughout the seminar. Observations highlighted
that having a chair who knows and understands the worlds of the different
groups of participants is a key success factor for effective interaction.
A lead researcher provided the broad overview of research activity, followed
by a series of presentations by researchers (typically three or four with a
maximum of six). After each presentation, time was scheduled for the
audience to ask questions and for discussion. The chair summarised briefly at
the end.
5
This design enabled large quantities of information to be transferred by the
researchers and many participants welcomed the chance to hear about the
evidence and ask questions. The seminars were successful in presenting
research outcomes. Information gathered from papers and presentations was
clearly interesting and potentially useful. Relatively little time was available for
group discussion of the evidence on key policy issues – important for linking
the two.
The design could be broadened to include a workshop approach with
opportunities for evidence and policy implications to be discussed in a
planned way. As an example, in the seminar on School Effectiveness and
Improvement a small group discussion took place at the end of the session
enabling everyone to raise issues, put a point of view, input information and
begin to tease out the nature of the evidence presented and possible
implications for policy.
Preparation
DfES led the preparation for each seminar, working in conjunction with lead
researchers. Research presenters were briefed in advance on key policy
issues, papers to support the presentations were prepared by the researchers
for distribution at the seminar and a broad outline of the seminar programme
was sent out in advance. Invited participants were encouraged to submit
questions for the presenters to address. It is not known whether the
participants read the research papers after the seminar, or what was the
nature and extent of the questions submitted beforehand by policy makers
since they were not directly referred to during the seminars.
Further development of collaborative preparation (already underway in some
areas) could ensure that the key policy issues are clearly identified in advance
and form the main content around which the seminar is built. Papers could
then directly address policy issues and key topics/questions for discussion
could be agreed.
Papers sent out in advance would help to ensure people arrived briefed about
the issues and emerging evidence and to be more fully prepared to discuss
the policy implications. Only the key research findings need then be presented
on the day.
Participants
The seminars were primarily organised for researchers and DfES policy
people. The latter included those with a team leadership role, a team
membership role and those with a research or analytical role. People from
Schools Directorate attended – post-16 teams were not involved. Perhaps
they should be? The groups consisted of between 20-35 invited participants
from DFES teams, plus researchers and representatives from national bodies,
for example TTA, QCA, Ofsted, GTCe, BECTA, Nuffield Foundation, and
6
NFER. Although some people at each seminar knew each other having a
means of individual identification helps to ease introductions and facilitate
communication.
Pedagogic issues
Room layout (often beyond the control of the organiser) was either conference
or boardroom style. The latter was the better of the two but “cabaret” style
seating round small tables would foster a more informal, collaborative
atmosphere and is organisationally more effective for open or small group
discussions.
Policy people were the majority in all but one of the seminars observed,
although they had little formal input except through the chair. At each seminar
research issues dominated with approximately fifty percent of time taken by
research presentations. Follow up question and answer slots also tended to
have research issues as the topic of exchanges between the presenter and
audience.
Presentations by both researchers and policy people which included both
policy areas and evidence could be explored and a collaborative approach
encouraged. It would also help to avoid domination by either group. One
seminar on School Effectiveness and Improvement came close to this by
including key policy issues in presentations by researchers.
While the seminars were informative and seemed well received by
participants, they were information heavy. In a two or three hour seminar,
information overload and a predominately didactic style risk losing the
potential impact of the evidence for policy and occasionally losing the
attention of the audience. Where important policy issues were raised they
merited longer and more in-depth discussion to tease out how the evidence
might be used. Again, a workshop design would provide more flexibility and
variety for interaction and engage all participants more fully, for example by
including short presentations, group tasks possibly using prepared material or
problem solving discussions on policy issues with feedback from groups
Participants were not asked for written feedback on the seminar, although
encouraged to voice opinions about the experience. Follow up from the
seminars could include feedback from participants to provide information from
which to refine and adjust the format and continuously improve effectiveness
Communication
Observation of the communication flows at the seminars revealed these to be
predominantly of two kinds:

one-way transfer of information from researchers to audience, through the
presentations. As a format, presentations are useful for conveying large
amounts of information, but have limited value for open discussion of
7
issues. They rarely include all participants and risk being dominated by
research issues at the expense of policy issues.

two-way communication between the presenter and individuals in the
audience in the follow up questions and answer slots (with occasional
three way discussion). There was more two-way discussion between
researchers than between researchers and policy people. Possible
reasons why are included in Appendix 2 ‘Challenges to the process of
interaction’.
Group discussions, or focussed tasks, with researchers and policy makers
working together would provide opportunities for more in depth work on how
to make use of evidence and encourage multi-directional communication.
The distinctness of the groups in the rooms was evident. Different working
environments, skills, language, imperatives, working time scales and cultures
all influence communication flow. Easing the flow between the participants,
and increasing understanding of the each other’s world, would be likely to
facilitate communications. An example of action that could be taken would be
for the main groups or teams present to be introduced by the chair at the
beginning of the seminar.
Seminar impact
Follow-up activities are already taking place, for example through:

regular dialogue between TLRP researchers and DfES policy teams

involvement of researchers in writing guidance for schools

disseminating a pamphlet series carrying issues and outcomes from
the seminars
From a small number of follow-up interviews with organisers and policy
makers shortly after the seminars, there was no direct evidence that the
seminars have had an immediate impact. It is probably too soon to establish
what this might be. The seminars are, after all, a new experience for many.
However, people said they had enjoyed them and would probably think about
some of the information they had received and possibly review their work.
However, the timescales they were working to did not necessarily fit the timing
of the emergence of research evidence. Accessing and using evidence in
policy development is a large and complex issue. A more comprehensive
strategy, of which seminars might be a part, would probably be needed to
raise the influence of research on busy working lives.
Conclusions
The study set out to identify the factors for effective interaction in researchpolicy seminars, which might then lead to research evidence linking with the
policy making process more effectively. The series of seminars has begun
well and provided valuable experience for the participants. It has enabled us
8
to identify the factors which relate directly to the success of a seminar. These
have been highlighted in the report and are summarised in Appendix 1. They
could be used as a checklist when designing or developing such interactive
seminars
There are also two wider conclusions from the study:

challenges for the process of interaction between researchers and
policy makers became evident as the observations progressed. These
seemed sufficiently important to summarise separately in Appendix 2.
They indicate areas of further work which would be useful in
developing the interaction process more fully

further improvements in the overall process of linking research
evidence to policymaking could be made through interactions at
different stages of the research/implementation cycle (briefly outlined in
Appendix 3). Although beyond the scope of this study this would be a
useful area to explore further.
9
Appendix 1
Summary of success factors in research and policy
interaction seminars
Observation suggests the following success factors:
Success factor
a clearly stated purpose which
relates to the planned
outcomes
Example
‘to agree ways of taking relevant evidence forward into
the policy making process’
‘to develop a policy document’
‘to develop advice for ministers’
identification of key policy
issues around which the
seminar will be built and
research evidence selected
collaborative design by
researchers and policy makers
Lead researcher and policymaker work together to
agree the purpose and outcomes and design the
seminar
a chair/leader who knows and
understands the worlds of both
research and policy making
a variety of methods for giving
and exchanging information
and exploring issues/evidence
and ways forward
short presentations, opportunities for small group
discussion and problem-solving tasks with
feedback/proposals/ solutions captured. A workshop
method ensures all participate
papers with research evidence
and policy issues circulated in
advance
Papers with key policy issues and questions together
with research project/evidence summaries
room layout which fosters an
informal atmosphere and
flexible working methods
Café style tables layout
inputs from both researched
and policy makers
The lead presentation outlines the policy area and
issues, research evidence then presented and both
drawn together in an overall presentation which raises
questions and sets the discussion topics
identification of individuals and
early introductions of groups
participating
Name badges; chair/leader says which groups are
represented during the introduction
10
Appendix 2
Challenges for the process of interaction
During the observations of communication between researchers and
policymakers it became increasingly evident that there are a number of
challenges in handling this sort of interaction. For example:

researchers may not be well informed about the policymaking process.
They may be unsure of procedures, timescales and how and when
evidence can feed into the process

expectations of interaction may well differ - for researchers nothing is
absolutely certain, whereas for policy makers answers are needed to
difficult questions

researchers are used to presenting information about methodology as
well as evidence whereas policymakers are looking for an emphasis
evidence

the research culture encourages questioning, often in great detail, of
the work of other researchers. This can dominate or and lead to the
focus of discussion at seminars being between researchers at the cost
of losing interaction with policymakers

researchers see the merit of further research – there can be a
tendency to see the outcomes from a project as much in terms of
questions for further research as in terms of useful evidence

researchers are rarely trained in ‘business style presentations’ to nonprofessional researchers. While some researchers have developed this
skill this, training is needed for others

different working cycles, timescales and imperatives dictate the pace of
activity in the work of researchers and policy-makers. Understanding
these and working with them is a challenge for both groups
11
Appendix 3
Interaction in a wider context
The seminars have proved a strong vehicle for developing communication
links and providing information. However they are, currently, confined to
communication between researchers and policy makers at a stage when
findings are emerging or after the research has been completed.
In a typical research /implementation cycle 1 there are at least four stages with
opportunities for interaction and each would have a different purpose.
drawing implications
interpreting findings
agenda setting
research design
The stage of an interaction within the research cycle would determine the
purpose of the interaction as well as the design and the leadership. The
purpose of this series of seminars would therefore fall under either
‘interpreting findings’ or ‘drawing implications’ for policy.
For these two stages interactive workshops could provide excellent
opportunities to tease out how emerging evidence could feed into policy and
for issue led debate about evidence, with both researchers and policy makers
providing inputs. Workshops at all four stages would increase the likelihood
of evidence feeding into policy, particularly if researchers and policy makers
jointly identified the purpose of a workshop.
A further consideration is the stage in the policy cycle at which interaction
takes place. One researcher commented in his paper for the seminar:
‘the results of this study do not bear directly on recent Government initiatives
to increase the levels and training of support staff in schools. This is because
the rapid pace of change means that results will not necessarily reflect recent
changes and improvements in provision’ 2
While the seminars provided policy makers with opportunities to hear about
research evidence, researchers would welcome the opportunity to deepen
for fuller details of interaction at stages in the research cycle see ‘Collaborative Research in
Practice’ A.Morris and L.Norman LSDA 2004
2 from a paper by researchers at the Institute of Education, University of London for the
seminar on Paid Adult Support
1
http://eppi.ioe.ac.uk/EPPIWeb/home.aspx?page=/reel/review_groups/inclusion/review_two.htm)
12
their knowledge and understanding of the policy development process and
time scale. Shared understanding of processes would be beneficial in
ensuring that evidence could feed into policy decisions in a timely and
appropriate way. A key question to pose is ‘how can researchers and policy
makers interact effectively at different stages in the research cycle and the
policy development process?’
13