POST PRINT VERSION.
Accepted by International Journal of Educational Research on 27 August 2012.
*Note – this is a copy of the final draft version submitted on 4 July 2012 after peer review.
International Journal of Educational Research 56 (2012) 23–34
http://dx.doi.org/10.1016/j.ijer.2012.08.001
Author details:
Corresponding author: Adrian Cherney
*Dr Adrian Cherney
School of Social Science
The University of Queensland
Brisbane, St Lucia 4072
ph + 61 7 3365 3236
fax + 61 7 3365 1544
email: a.cherney@uq.edu.au
Dr Jenny Povey
Professor Brian Head
Professor Paul Boreham
Michele Ferguson
Institute for Social Science Research
The University of Queensland
Brisbane, St Lucia.
Acknowledgements: This project is supported through ARC Linkage project: LP100100380.
1
What influences the utilisation of educational research by policy-makers
and practitioners? – The perspectives of academic educational researchers
Abstract
In the field of education much has been made of the need for academics to engage more closely with
policy-makers and practitioners in the process of knowledge production and research uptake. This
paper reports results from a survey of academic educational researchers in Australia on their
experience of research uptake and engagement with policy-makers and practitioners. We examine a
range of variables to understand factors influencing the use of educational research. The results
indicate that while research uptake is enhanced through mechanisms that improve the intensity of
interactions between academics and end-users, the dynamics of research collaborations have a
significant bearing on research use. Our findings provide insights into the challenges that can be
confronted when academics engage in research aimed at influencing policy or practice.
Keywords: research utilisation, research collaborations, education, knowledge translation, policy
impact, academic research
2
1: Introduction
In the field of education the observation has been made that academic research rarely has a
policy impact and often fails to meet the needs of policy-makers and practitioners (Coburn &
Talbert, 2006; Hemsley-Brown & Sharp, 2003; Hess, 2008; Hillage et al 1998; Levin &
Edelstein, 2010; Oancea, 2005). This disjunction is partly seen as originating in
communication problems between policymakers, practitioners and academic researchers,
drawing on the argument that they live in different worlds with differing languages, values
and professional rewards (Bell et al., 2010; Kirst, 2000; Levin, 2011; Orland, 2009;
Vanderlinde & van Braak, 2010). There is some validity to this argument, with studies in the
field of education indicating that educational researchers, bureaucrats and teachers often have
different priorities and perceptions about what constitutes useful and valid research, the role
of theory, data quality and research methods, project outcomes, brevity of results and the
practicality of research recommendations (Bell et al., 2010; Coburn & Talbert, 2006; Cousins
& Leithwood, 1993; Honig & Coburn, 2008; Levin & Edelstein, 2010; Saha, Biddle, &
Anderson, 1995; Vanderlinde & van Braak, 2010; Wilkins, 1988; Zeulie, 1994).
Greater collaboration between academic research producers and users, or consumers
of research, is seen as one way of addressing the dissonance between knowledge production
and its transfer or translation to policy and practitioner contexts. There is some evidence that
when academic researchers and policy-makers or practitioners work closely in the
formulation and execution of research projects, the research is more likely to have an
influence on policy or practice (Cordingley, 2008; Cousins et al., 1996; Cousins & Simon,
1996; Huberman, 1990; Nutley et al., 2007). However, closer collaboration alone is
insufficient to ensure that research has a policy or practice impact, with studies demonstrating
that a range of variables influence the uptake and use of academic social research by nonacademic end-users (Bell et al 2010; Bogenschneider & Corbett, 2010; Cherney & McGee,
3
2011; Huberman, 1990; Landry et al., 2001a, 2001b; Weiss & Bucuvalas, 1980). Moreover,
research collaborations can be inherently problematic with participants’ involvement
influenced by individual and institutional constraints and contingencies (Bell et al 2010;
Bogenschneider & Corbett, 2010; Coburn & Talbert, 2006; Cousins & Simon, 1996; Edwards
et al 2007).
The issue of closer synergies between educational researchers and non-academic endusers raise important issues concerning the role that educational research should play in
relation to policy and practice. As Hammersley (2007) points out, ones position on this issue
is influenced by judgments on whether academic educational research should be integral to
practice or ought be judged as a value in its own right. While more nuanced positions are
often adopted that recognise the multi-dimensional value of educational research (e.g. see
Cooper, Levin and Campbell 2009) this is a highly contested issue (Burkhardt & Schoenfeld,
2003; Lingard, 2011). The intensity of debate about the value of academic research has also
been increased through University Research Assessment Exercises, such as the Research
Excellence Framework in the UK, which has required academics to demonstrate the impact
of their research. Academic researchers in Australia are subject to similar pressures (e.g.
through the Excellence in Research Australia initiative and at the time of writing the
Excellence in Innovation for Australia trial).
While our aim here is not to elaborate upon all these issues, our paper does provide
new analyses relevant to the debate around research impact and evidence-based policy in the
field of education. Using survey data from academic university researchers in Australia who
engage in research collaborations with external partners, the paper principally aims to
examine factors that influence the uptake of social research, as interpreted through the
experience of knowledge producers in the field of education. Using the scale of research
4
utilisation (Knott & Wildavsky, 1980) we examine factors that appear to influence reported
levels of research impact.
The paper is organised as follows. Firstly, the explanatory model underpinning this
study (i.e. the scale or ladder of utilisation) will be discussed. Secondly, the paper outlines the
data collection methods used for the survey administered to Australian academics. Thirdly,
key results from the sample of educational researchers are provided, focusing on reported
levels of research utilisation and variables that appear to influence knowledge transfer and
application. Finally, the paper discusses the results (and some data limitations) and concludes
with broader observations about the study of research uptake in the field of education.
2: Literature review
2.1: Measuring Research Use
When it comes to measuring research utilisation, no single conceptual model has been
unanimously adopted (Belkhodja et al., 2007; Lester, 1993). One reason for this is the
methodological problem associated with specifying the dependant variable of research use
given it can be defined either as a process or an outcome. Furthermore social research can
provide answers to technical questions, such as “did this program work?” to helping policymakers or practitioners interpret problems in ways that change their understanding about
issues or choices (Biesta 2007; Nutley et al., 2007). Hence the complexity surrounding the
use of academic social research can make it difficult to measure, particularly when attempting
to quantify its interruptive function (Beista 2007). Scales of research use can be particularly
helpful in understanding that the breadth of social research usages ranging from practices that
encompass transmission through to actual application (Cherney & McGee, 2011). Such scales
can be valuable because they can be used to identify how utilisation is related to various
decision-making processes, particularly concerning the actions of the producers and
consumers of social research (Landry et al., 2001b).
5
Methodologically this study replicated a modified version of the Knott & Wildavsky
(1980) research use (RU) scale, , similar to that adopted in the study by Landry et al (2001a).
This scale was adopted because it has been frequently cited in the literature, has been used to
measure research use among government officials and academics, and has been shown to be
reliable (Landry et al., 2001a; Lester and Wilds 1990; Lester 1993)1. Conceptually the
research use scale can be referred to as the “ladder of utilisation” and table 1 provides the
descriptions for each stage of research use (or rung of the ladder), as presented in our
questionnaire to Australian social scientists. The benefit of this scale is that it operationalises
research use as a cumulative process that progresses through a number of stages:
transmission, cognition, reference, effort, influence and application. The scale is cumulative
in the sense that cognition builds on transmission, reference builds on cognition, effort on
reference, influence on effort, and application on influence. The RU scale has been criticised
as perpetuating a linear understanding of research utilization (Davies & Nutley, 2008).
However it does recognise the fact that the research utilisation process varies between a range
of activities that involve knowledge transfer and uptake (Cherney & McGee, 2011; Knott &
Wildavsky, 1980; Lester, 1993).
<Insert Table 1>
2.2: Independent Variables Influencing Research Use
Just as there is no agreed conceptual model relating to research utilisation, there is no
definitive list of variables developed to help predict knowledge use (Lester, 1993). Most
studies have categorised variables under broad groups relating to supply-side and demandpull factors, as well as dissemination and interaction variables. Supply-side factors include
research outputs and the context in which the researcher works. These can include the types
1
Another key reason this scale was chosen was to allow for international comparison with studies such as
Landry, Amara and Lamari (2001a) who also adopted the research use scale. We have not provided such
comparative results because it falls outside the scope of this paper.
6
of research outputs produced by academics (e.g. qualitative or quantitative studies2), whether
research is focused on non-academic users, the importance of internal or external funding
sources, and the institutional drivers that influence the initiation of collaborations with
external partners and end-users (Bogenschneider & Corbett 2010; Cherney et al., 2011).
Demand-pull factors concern whether end-users consider research to be pertinent, whether it
coincides with end-users’ needs, whether users accord it credibility, and whether it reaches
users at the right time to influence decision-making. Added to this are organisational
processes such as level of skills to apply research knowledge, that could inhibit uptake of
research and thus influence the overall demand for academic research within end-user
organisations (Belkhodja et al., 2007; Coburn & Talbert, 2006; Ouimet et al 2009).
Dissemination variables relate to efforts to adapt and tailor research products (e.g. reports) for
end-users and to develop strategies focused on the communication of research (Huberman,
1990). The assumption is that the more researchers invest in adaptation and dissemination,
the more likely research-based knowledge will be adopted. Adaptation includes efforts to
make reports more readable and easier to understand, efforts to make conclusions and recommendations more specific or more operational, efforts to focus on variables amenable to
interventions by users, and efforts to make reports appealing (Cherney & McGee, 2011).
Dissemination efforts include strategies aimed at communicating research to targeted endusers, such as when researchers use different social media to communicate their research
messages, hold meetings to discuss the scope and results of their projects with specific users
or partners, and target particular forums, e.g. reporting on their research to government
committees. Finally, interaction variables focus on the intensity of the relationships between
knowledge producers and potential users. The types of factors considered relevant include
2
Both qualitative and quantitative approaches can serve different purposes, relating to the technical and
interruptive functions of social research (see section 2.1). We have used this distinction between qualitative
and quantitative approaches in order to be able to measure and identify whether chosen research
methodology mediates the impact of social research.
7
informal personal contacts, participation in committees or experience with research
partnerships, e.g. the number of research partnerships an academic has engaged in
(Huberman, 1990; Landry et al., 2001a; Lomas, 2000).
3: Current Study
The data used in this research were drawn from a broader study examining evidence-based
policy and practice (Cherney et al., 2011). The study has four phases and the data reported
here was obtained from Phase 1, which used a purposive sampling technique to target
academic social scientists in Australian Universities3. The final sample recruited was 693,
which constitutes an overall response rate of 32 per cent. For the purpose of this analysis,
only data pertaining to academics who identified their primary research discipline as
education have been used (n=156). The academic survey was partially based on existing
items or scales (Bogenschneider & Corbett, 2010; Landry et al., 2001a, 2001b) but with
additional items included to gauge the dynamics of research partnerships.
3.1: Dependent variable
Knowledge utilisation was measured using a validated version of the Knott and Wildavsky
(1980) research use scale. As indicated the scale is based on six stages namely: transmission,
cognition, reference, effort, influence, and application. For each of these six stages
respondents were asked to estimate what had become of their research using a 5-point scale
ranging from 1 (never), 2 (rarely), 3 (sometimes), 4 (usually), to 5 (always).
3
This involved targeting Fellows of the Academy of the Social Sciences in Australia (ASSA) as well as
Australian academics who had secured at least one Australian Research Council (ARC) grant between 2001 and
2010 within the field of social and behavioural science. Academics who met either of these criteria were sent an
electronic survey to complete. ASSA fellows are recognised for their outstanding contributions to the social
sciences in Australia and abroad and dominantly comprise senior academics. Australian Research Council
(ARC) grants are national competitive grants and funds a significant proportion of research activity in
Australian Universities. The reason this form of sampling was adopted was because studies have shown that
seniority and the number of external competitive research grants are key determinants of engagement with nonacademic end-users (Cherney & McGee 2011; Landry, Amara, and Lamari 2001a, 2001b).
8
Previous researchers (Cherney & McGee, 2011; Landry et al., 2001a) have used this
scale cumulatively (with each stage building upon the next) and assigned a value of 1 when
respondents replied always, usually, or sometimes, and with all other responses assigned the
value of 0 or a fail. There are two ways that this cumulative approach can be analysed. The
first is to run a separate logistic regression for each stage of research utilisation as Landry et
al (2001a) did in their study. Hence respondents who pass all six stages would be represented
in each stage or regression model (see Figure 1). This is particularly problematic with our
sample, because the majority (75%) of the sample reported they passed all six stages. Hence
the question arises whether such a method would really be determining what predicts
movement from one stage to the next, or whether progression across each stage is masked by
the dominant group. In order to address this criticism, a second approach would be to create
an ordinal variable with seven levels, including in each level only those individuals who
passed that level. Thus respondents in each level would be unique. Table 2 presents the
number of respondents categorised in each level/echelon according to such progression
criteria. For instance, two percent of the sample passed the transmission stage but did not
progress further. However, an ordinal logistic regression analysis is not possible due to our
small sample size and the number of cases in each level.
The next possible option would be to examine whether these stages are in fact
exclusive. Does failure in one stage preclude academic researchers from progressing to other
stages? Or should these stages comprise an index? Descriptive statistics, as presented in
Figure 2, illustrate that failure in one stage does not preclude academic researchers from
passing subsequent stages. This is an important consideration because the process of research
utilisation has been argued to be non-linear, with data in Figure 2 indicating that one does not
necessarily have to traverse in sequence each rung of the research utilisation ladder to reach
the ultimate stage - i.e. application. A factor analysis of the items (or stages), revealed a 1-
9
factor solution and a Cronbach’s alpha coefficient of 0.77 (Table 3). Thus, the results indicate
that these items are measuring one construct and that the index seems to be reliable. It was
thus decided to use the items as an index to measure research use. A mean index score was
calculated for all 6 six stages. The mean score for the research utilisation index is presented
in Table 3.
<Insert Figure 1>
<Insert Table 2>
<Insert Figure 2>
3.2: Independent variables
A number of indices were created and included in our model as independent variables. The
items used in each index were determined by factor analyses, with each index comprising a 1factor solution.
The Cronbach’s alpha coefficients for these independent variables are
presented in Table 3 and detailed descriptions of index compositions are presented in
Appendix 1.
<Insert Table 3>
Descriptive statistics for each independent variable are presented in Table 4. Academic
researchers from the discipline of education indicated that academic funding (i.e. national
competitive grants such as Australian Research Council grants, and internal University funds)
were more important than funding from government and non-government agencies in
ensuring their research is conducted. Academic researchers indicated that the ‘relevance’ of
the research is given a higher priority by end-users, compared to other features such as the
quality or feasibility of the research. A higher level of importance was attributed to tailoring
research to meet the needs of end-users and meetings to discuss findings with end-users.
Table 4 also illustrates a high level of agreement among academic educational researchers
concerning the fact that they encounter barriers in the transfer and uptake of their research. A
10
very high level of importance is accorded by academic educational researchers to the use of
refereed publications as a method through which to disseminate their research. The number of
research partners with whom researchers engaged ranged between 0 and 35, with an average
of 6 research partners per researcher. The number of grants received by these academic
educational researchers varies between 0 and 44, with the average researcher having received
8 grants. In general, our sample has a high level of experience in engaging in research
partnerships and securing research grants.
<Insert Table 4>
3.3: Data analysis
Given that our dependent variable is approximately continuous, an Ordinary Least Squares
(OLS) regression model was used to estimate the associations between research utilization
(our dependent variable) and a number of explanatory variables such as benefits and barriers
associated with engaging in research with policy-makers and practitioners. As a preliminary
check, we examined the correlations between all variables in the model. They ranged
between .002 and .68, suggesting that multicollinearity was unlikely to be a problem (the
correlation matrix was too large to depict in the Appendix). This was confirmed by a
relatively low value of the mean Variance Inflation Factor (VIF) of 1.66, with the individual
variable’s VIFs ranging from 1.18 to 2.48. The four highest correlations were between
problems relating to the orientation of research partnerships and ‘consequences’ of investing
in research partnerships (0.68); ‘consequences’ of investing in research partnerships and
barriers academics experience in the transfer and uptake of research by end-users (0.57);
importance of using contacts, seminars and reports to present research to policy-makers and
practitioners and importance of meetings and dissemination activities with end-users (0.55);
and importance of meetings and dissemination activities with end-users and importance of
tailoring research when end-users are the focus (0.55). All four correlations were statistically
11
significant.
3.4: Regression Results
The regression results are presented in Table 5. The results indicate that eight variables were
significantly related to the utilisation of educational research. The more academic researchers
perceived collaboration with external partners as beneficial, the more likely they report
utilisation. As the number of grants increases so does the likelihood of research utilisation.
The more negative the perceived consequences for academic educational researchers when
engaging in research partnerships, the less likely it is reported that their research will be
utilised by policy-makers or practitioners. The importance of tailoring the research for endusers is positively and significantly associated with reported levels of research use. Academic
educational researchers also reported that when end-users felt that research was relevant it
was more likely to lead to utilisation. Finally academic researchers indicated that the more
policy-makers or practitioners prioritise the ‘feasibility’ of research (i.e. policy-makers or
practitioners place greater emphasis on research being economically and politically feasible)
the less likely were academics to perceive that end-users would use academic social research.
When academic researchers perceived there to be problems associated with research
partnerships they were less likely to report research uptake by external agencies.
<Insert Table 5>
4: Discussion
Before proceeding to a discussion of possible practical and theoretical explanations for this
pattern of results, the methodological limitations of the present study should be noted. Firstly,
the survey data is based on self-reports, which can be subject to social desirability biases. In
the context of academic researchers reporting levels of research use this can be a salient
factor in influencing responses to survey items (Davies & Nutley, 2008; Levin, 2011).
Furthermore it is possible there is a self-referential process occurring in that respondent’s
12
reported level of research utilisation is reinforced by how relevant they perceive their
research to be for policy-makers or practitioners and vice versa. We have not examined the
reference point (e.g. specific project contexts) establishing why respondents believed their
research had an impact or why problems were encountered. One additional way to measure
judgments about research utilisation among academics is to adopt a two-step approach using
a yes-no question, first about a particular research use stage (e.g. my research reports have
been read and understood by end-users) and then asking respondents to indicate how
frequently they think it occurs. This can help increase measurement accuracy. There is little
doubt though that tracing the impact of research evidence and collaborations needs to be
complemented by more detailed qualitative work. The survey data only gives insight into
broad patterns of utilisation as reported by our academic sample – hence providing one
possible perspective and potentially reflecting particular biases. This does not undermine the
validity of the results reported here, because a full understanding of the process of research
transfer and uptake requires a close examination of the experiences and perspectives of both
knowledge producers and users, and an exploration of broad general patterns as well as
specific instances of research uptake.
The results reported above indicate that our sample of educational researchers
recognise that, in order for their research to have an impact, there is a requirement to directly
engage with end-users through meetings and dissemination processes and that it is important
to tailor research projects and findings to end-user needs. Despite reports that academic
researchers do not understand the needs of policy-makers or practitioners (Burkhardt &
Schoenfeld, 2003), our sample were aware that non-academic end-users do have different
priorities and perceptions when it comes to judging the relevance and use of research
evidence. This awareness is perhaps linked to the fact that our sample was experienced in
engaging with multiple research partners, making them mindful of these contextual issues.
13
Reported levels of research use was also shown to be heavily influenced by access to
funding particularly national competitive funding and internal University funds, compared to
others sources available from government and non-government agencies. Intuitively one
would assume that this finding would be the other way around, given the applied focus of
many government and non-government funding schemes. One explanation for this result is
that a high reliance on academic funding sources is possibly related to the fact that
government and non-government schools and educational departments face significant
funding constraints and hence do not have large amounts of discretionary funds to direct
towards research. Hence educational researchers in Australia need to rely heavily on national
competitive academic research funds (such as the Australian Research Council) and internal
sources within Universities (Holbrook et al., 2000). Such funding issues are not unique to the
Australian context (Burkhardt & Schoenfeld, 2003).
When it comes to the levels of research utilisation reported by our respondents, the
perceived costs and benefits of engaging in research partnerships had a significantly strong
relationship to the reported transfer and uptake of academic research. While much is made of
the need for academics to work more closely in partnership with external agencies, the
potential consequences and costs of doing so cannot be overlooked, especially in relation to
problems that may arise in the context of managing research partnerships and the priorities of
partners. Given the differing “world views” about research and the institutional cultures
driving academics and policy-makers or practitioners, tensions in collaborations are
inevitable (Bogenschneider & Corbett, 2010; Edwards et al., 2007). A major challenge for
educational researchers is that developing the necessary know-how to manage collaborations
more effectively is not part of mainstream academic training and is typically a skill developed
over time through experience and mentoring.
14
Two noteworthy results that arose from the regression analysis are the findings about
the reported importance of the ‘feasibility’ of research and the industry-orientation of
partnerships in relation to perceived levels of research use. The ‘feasibility’ variable
comprised three items: research recommendations are seen as economically and politically
feasible and research findings support a current position or practice (see Appendix 1). When
feasibility is seen as a strong priority, our respondents reported that academic research is less
likely to be used. This finding points to a key lesson that the quality of research evidence is
not the only priority potentially driving research use and nor is it the single most important
factor in determining uptake. Even if one accepts the realities of end-user preferences and
tries to work within the policy and practice space offered by collaborations with external endusers, the ‘orientation’ of research partnerships can further influence/undermine levels of
uptake. This relates to the finding that partnership orientation can create problems for
research impact. This ‘orientation’ variable comprised five items (see Appendix 1) relating to
the fact that partnerships can become dominated by what industry funding partners desire,
which can challenge the integrity and rigor of the academic knowledge production process. It
is little wonder some of our sample were weary of engaging in research collaborations. The
findings discussed above do not mean that educational researchers should avoid engagement
in research collaborations with government or non-government partners. On the contrary, the
implication is that educational researchers need to be mindful of the priorities driving
external agencies when they seek out research and engage in collaboration.
What are the implications of our findings for the impact of educational research on
policy and practice? One is that educational researchers need to become adept at
understanding the priorities of policy-makers and practitioners and craft knowledge
translation activities, such as research presentations, so they more effectively address the
knowledge needs of non-academic end-users. This does though raise the risk of academic
15
educational research simply being co-opted and directed by pragmatic concerns, which has
implications for knowledge production by potentially deflecting attention away from areas of
research that prove uncomfortable for policy-makers or practitioners. Translating educational
research is a difficult balancing act between ensuring research is relevant to non-academic
end-users, but in a way that does not compromise its scholarly integrity. The problem though
is that knowledge translation activities are not well understood in the academic social
sciences (Levin, 2011).
The work involved in collaboration does bring benefits to academic researchers in the
context of providing opportunities to influence policy or practice within educational settings
such as schools. However, academics must learn to tolerate the costs of such engagement.
The willingness to endure such costs can be heavily influenced by the institutional incentive
systems within academia including whether collaboration is seen as contributing to valued
forms of scholarship. Our data also indicate that when educational academic researchers seek
to undertake ’research for policy‘, their work must inevitably contend and compete with a
mix of political and economic priorities and values that influence policy and practitioner
decision-making (Head, 2008; Lingard, 2011).
5: Conclusion
The analysis presented in this paper provides insights into the types of challenges academics
face in generating impact through research collaborations. It provides empirical backing to
the broader debate and commentary within the field of education as to why there appears to
be dissonance between the evidence produced by educational researchers, and its transfer and
uptake within policy and practitioner contexts. Our data indicates how this research/policy
gap can be reduced, but it also signals that doing so through partnership synergies between
academic researchers and external agencies is not a straightforward solution. Understanding
the dynamics of different research collaborations with government and non-government
16
agencies is essential in ensuring that they are a success, and these dynamics require further
study. While our data only provide insights into the perspectives of academics, investigating
their experiences of collaborations and perceptions of research impact is essential for
identifying how these processes can be improved.
17
Appendix I: Independent variables measures
Research Approach
Quantitative studies
The quantitative research approach is a single item variable that
reflects how often researchers use a quantitative approach such as
surveys research, statistical analysis, and GIS in their research.
The results reported are the percentage of respondents who
indicated always or usually. These responses were recorded as 1,
while all other responses were recorded as 0.
Qualitative studies
The qualitative research approach is a single item variable that
reflects how often researchers use a qualitative approach such as
interviews, focus groups, ethnography, and observation in their
research. The results reported are the percentage of respondents
who indicated always or usually. These responses were recorded
as 1, while all other responses were recorded as 0.
Researchers’ Context
Research targeted to user This Index measures whether the majority of research conducted
by academics is directed at policy-makers and practitioners. This
index is comprised of four dimensions that range on a 4-point
scale, ranging 1 (never) to 4 (always). The four dimensions are:
(1) policy makers within government; (2) practitioners/managers
within the public sector; (3) practitioners/managers within the
community sector; (4) practitioners/managers within the private
sector.
Importance of academic
funding
This Index measures how important academic type funding is in
ensuring research is conducted. This index is comprised of two
dimensions that range on a 6-point scale, ranging from 0 (does not
apply), 1 (very unimportant) to 5 (very important). The two
dimensions are: (1) my university’s internal research funds; (2)
funding organisations such as ARC, NHMRC, CRC.
Importance of other
funding
This Index measures how important other funding sources are in
ensuring research is conducted. This index is comprised of five
dimensions that range on a 6-point scale, ranging from 0 (does not
apply), 1 (very unimportant) to 5 (very important). The five
dimensions are: (1) not for profit organisations; (2) federal
government agencies; (3) state government agencies; (4) local
government agencies; and (5) private sector organisations.
18
Benefits of collaborative
research
This Index is based on academic perceptions of the benefits of
carrying out research in collaboration with government, industry
or community sector partners. This index is comprised of ten
dimensions that range on a 6-point scale, ranging from 0 (not
applicable), 1 (strongly disagree) to 5 (strongly agree). The ten
dimensions are: (1) I have been able to use data that would
otherwise be difficult to access; (2) Research partnerships have
provided me with opportunities for my research to have an impact
on policy and practice; (3) Research partnerships have helped to
increase my industry contacts; (4) My industry contacts have
helped with developing future research projects; (5) Research
partnerships enable me to generate extra income for my work
unit; (6) Such projects have provided me opportunities to
commercialise research outcomes; (7) Research partnerships have
helped me with career advancement; (8) Such projects have
required me to be pragmatic and realistic in relation to research
outcomes for industry partners (9) Research partnerships have
enabled me to publish in a broad range of publication outlets (10)
I find projects with external partners more satisfying than
fundamental “blue sky” research.
Barriers academics
experience in transfer &
uptake of research
This index is based on the barriers academics experience in the
transfer and uptake of their research. This index is comprised of
five barriers that range on a 6-point scale, ranging from 0 (not
applicable), 1 (strongly disagree) to 5 (strongly agree). The five
dimensions are: (1) There are high costs (e.g. time and resources)
in translating the results of research for policy-makers and
practitioners; (2) There are insufficient forums and networks
available for bringing together researchers and non-academic endusers of research; (3) Academic reward systems do not adequately
recognise dissemination work to non-academic end-users; (4) The
academic requirement to publish primarily in peer-reviewed
journals inhibits a focus on policy and practitioner audiences; (5)
Networks and partnerships that might support research uptake are
often undermined by turnover of contact staff in public agencies.
Consequences of
investing into research
partnerships
This index is based on problems relating to investing time and
resources and accommodating partnership work that academic
researchers encounter when carrying out research with partners
from government, industry or the community sector. This index is
comprised of ten items that range on a 6-point scale, ranging from
0 (not applicable), 1 (strongly disagree) to 5 (strongly agree). The
ten dimensions are: (1) There are inadequate university resources
19
to support research partnerships with end-users; (2) I find there
are different research orientations between academics and
external partners; (3) You need to invest a lot of time in
coordinating the work between different partners; (4)
Confidentiality requirements often restrict what you can report
and publish; (5) You can lose ownership of intellectual property;
(6) You are subject to delays that impede your ability to publish
results in a timely manner; (7) I am under pressure from my work
unit to undertake contract research to meet budget requirements
(8) External partners do not appreciate the full costs of research;
(9) The ethics process can be time consuming and cumbersome;
(10) The complexity of contractual arrangements can lead to
delays in commencing research.
Problems relating to
orientation of research
partnership
This index is based on problems relating to the priorities and
expectations of the partners when carrying out research with
partners from government, industry or the community sector. This
index is comprised of five items that range on a 6-point scale,
ranging from 0 (not applicable), 1 (strongly disagree) to 5
(strongly agree). The five dimensions are: (1) I find there is
pressure to produce favourable results for partners; (2) I believe
such projects overemphasise applied outcomes; (3) I do not feel
comfortable working on projects carried out in collaboration with
industry or government agencies; (4) I feel that industry partners
place too much emphasis on specific deliverables; (5) I feel that
there is too much pressure to meet deadlines.
Research Time
This is a dummy variable created from the question asking
academics to indicate the nature of their position, either research
and teaching or research only. The research only was used as the
reference group.
Number of external
grants
This index is the sum of all the research grants (i.e. ARC
discovery, ARC linkage, other external competitive grants)
academics have received.
Users’ Context
End-users prioritise high
quality research
This index is based on academic researcher’s perceptions of what
research characteristics end-users prioritise when using
academically produced social science research. This index is
comprised of seven dimensions that range on a 5-point scale,
ranging 1 (not a priority) to 5 (high priority). The seven
dimensions are: (1) high quality research; (2) unbiased findings;
(3) adds to theoretical knowledge; (4) statistical analysis is high
20
quality; (5) findings can be generalised; (6) offers a new way of
thinking; and (7) reputation of researcher.
End-users prioritise the
useability of the research
This index is based on academic researcher’s perceptions of what
research characteristics end-users prioritise when using
academically produced social science research. This index is
comprised of four dimensions that range on a 5-point scale,
ranging 1 (not a priority) to 5 (high priority). The four dimensions
are: (1) findings available when decisions need to be made; (2)
findings have direct implications for policy & practice; (3)
findings written in a clear style; and (4) report has brief summary
of findings.
End-users prioritise the
feasibility of the
research
This index is based on academic researcher’s perceptions of what
research characteristics end-users prioritise when using
academically produced social science research. This index is
comprised of three dimensions that range on a 5-point scale,
ranging 1 (not a priority) to 5 (high priority). The three
dimensions are: (1) recommendations are economically feasible;
(2) findings support a current position & practice; and (3)
recommendations are politically feasible.
Policy-makers have
found my policy
focussed research to be
relevant
This index is based on the experience of researchers who have
had a policy focus in their research and whether policy-makers
found their research to be relevant. This index is comprised of
three dimensions that range on a 6-point scale, ranging from 0
(don’t know), 1 (strongly disagree) to 5 (strongly agree). The
three dimensions are: (1) relevant to their needs and expectations;
(2) valid and reliable; (3) trustworthy.
Barriers policy-makers
experience in transfer &
uptake of research
This index is based on the barriers policy-makers are perceived to
experience in the transfer and uptake of academic research. This
index is comprised of two barriers that range on a 6-point scale,
ranging from 0 (does not apply), 1 (very unimportant) to 5 (very
important). The five dimensions are: (1) Policy-makers and
practitioners lack expertise in how to interpret or understand the
findings of research; (2) Policy-makers and practitioners lack
expertise in how to apply the results of research to policy
problems.
Dissemination
Importance of tailoring
research when end-users
are the focus
This index is based on the importance attributed to various
aspects of tailoring research when the focus is on end-users. This
index is comprised of seven dimensions that range on a 6-point
scale of adaption, ranging from 0 (does not apply), 1 (very
21
unimportant) to 5 (very important). The seven dimensions are: (1)
readability and use of comprehension of my reports and research
articles; (2) specific, operational nature of conclusions or
recommendations; (3) provision of data that can be analyses by
end-users; (4) sensitivity to end-users’ expectations; (5)
presentation of reports (graphics, colour, packaging); (6) on-time
presentation of research findings to end-users; (7) attention to
‘deliverables’.
Importance of meetings
& dissemination
activities with end-users
This index is based on the importance attributed to organising
meetings and dissemination activities for end-users when
carrying-out research. This index is comprised of four dimensions
that range on a 6-point scale, ranging from 0 (does not apply), 1
(very unimportant) to 5 (very important). The four dimensions
are: (1) preparing and conducting meetings in order to plan the
subject and scope of projects with end users; (2) regular formal
meetings to report on a study’s progress with end-users; (3)
formal meetings to discuss findings with end-users; (4) preparing
and implementing research dissemination activities for end-users.
Importance of using
presentations and reports
to present research to
parliamentary
committees
This index is based on the importance attributed to using methods
such as presentations and reports for presenting research to
parliamentary committees. This index is based on two items
measured on a 6-point scale, ranging from 0 (does not apply), 1
(very unimportant) to 5 (very important). The two items are: (1)
presentations to parliamentary committees and (2) sending reports
to parliamentary committees.
Importance of using the
media to present
research
This index is based on the importance of using media to present
research conducted by academics. This index is based on three
items measured on a 6-point scale, ranging from 0 (does not
apply), 1 (very unimportant) to 5 (very important). The three
items are: (1) participation in radio and/or television programs;
(2) publication of articles in non-academic outlets; (3) publication
in electronic media, e.g. blogs and other social media.
Importance of using
refereed publications to
present research
The Journal variable is comprised of one item, publication of
articles in refereed journals and is another method employed by
academics to present or discuss their research. The importance of
this method is measured on a 6-point scale, ranging from 0 (does
not apply), 1 (very unimportant) to 5 (very important).
Interactions
Importance of using
contacts, seminar and
This index is based on the importance attributed to using methods
such as informal contacts, seminars and reports for presenting
22
reports to present
research to policymakers and practitioners
research to policy-makers and public practitioners. This index is
based on six items measured on a 6-point scale, ranging from 0
(does not apply), 1 (very unimportant) to 5 (very important). The
six items are: (1) informal contacts with policy personnel of
government agencies; (2) informal contacts with public or
community sector practitioners; (3) participation in seminars and
workshops organised by government policy agencies; (4)
participation in seminars and workshops organised by
practitioners within public or community sectors; (5) sending
reports to government policy agencies; (6) sending reports to
practitioners within public or community sectors.
Importance of using
contacts, seminar and
reports to present
research to private sector
organizations
This index is based on the importance attributed to using methods
such as informal contacts, seminars and reports for presenting
research to private sector organizations. This index is based on
three items measured on a 6-point scale, ranging from 0 (does not
apply), 1 (very unimportant) to 5 (very important). The three
items are: (1) informal contacts with personnel of private sector
organisations (2) participation in seminars and workshops
organised by private sector organisations (3) sending reports to
private sector organisations.
Number of external
partnerships
This index is the sum of all the partnerships with external
organisations that academic researchers have engaged in.
Role of the funding source: This project was finically supported through the Australian
Research Council Linkage project LP100100380. This project has received cash and in-kind
support from the following industry partners: Australian Productivity Commission;
Australian Bureau of Statistics; Queensland Health; Queensland Dept of Communities;
Queensland Dept of Employment; Queensland Dept of Premier and Cabinet; Victorian Dept
of Planning and Community Development; Victorian Dept of Education & Early Childhood
and the Victorian Dept of Human Services. The analysis and interpretation of the data
presented in this paper was undertaken solely by the authors. The Industry Partners did not
have any involvement in the study’s design or the survey data collected from academic
researchers in Australian Universities.
References
Bell, M., Cordingley, P., Isham., C. & Davis., R. (2010) Report of Professional Practitioner Use of
Research Review: Practitioner engagement in and/or with research. Coventry: CUREE, GTCE, LSIS
& NTRP. Available at: http://www.curee--‐ paccts.com/node/2303.
23
Belkhodja, O., Amara, N., Landry, R., & Ouimet, M. (2007). The extent and organizational
determinants of research utilization in Canadian health services organizations. Science
Communication, 28(3), 377-417.
Biesta, G. (2007). Why 'what works' won't work. Evidence-based practice and the democratic deficit
of educational research. Educational Theory 57(1), 1-22.
Bogenschneider, K., & Corbett, T. J. (2010). Evidence-Based Policy Making: Insights from Policy
Minded Researchers and Research-Minded Policymakers. New York: Routledge.
Burkhardt, H., & Schoenfeld, A. H. (2003). Improving educational research: Toward a more useful,
more influential, and better funded enterprise. Educational Researcher, 32 (9), 3-14.
Cherney, A., & McGee, T. R. (2011). Utilization of social science research: Results of a pilot study
among Australian sociologists and criminologists. Journal of Sociology, 47(2), 144-162.
Cherney, A. Head, B. Boreham, P. Povey, J. & Ferguson, M. (2011) The utilisation of social science
research in policy development and program review. Preliminary report: Phase 1 Results.
Institute of Social Science Research, University of Queensland
Coburn, C., & Talbert, J. (2006). Conceptions of Evidence Use in School Districts: Mapping the
Terrain. American Journal of Education, 112, 469-495.
Cooper, A., Levin, B. and Campbell, C. (2009) The growing (but still limited) importance of evidence
in education policy and practice. Journal of Educational Change, 10: 159-171.
Cordingley, P. (2008). Research and evidence-informed practice: focusing on practice and
practitioners. Cambridge Journal of Education, 38(1), 37-52.
Cousins, J. B., & Leithwood, K. A. (1993). Enhancing knowledge utilisation as a strategy for school
improvement. Knowledge: Creation, Diffusion, Utilisation, 14 (3), 305-333.
Cousins, J. B., & Simon, M. (1996). The nature and impact of policy-induced partnerships between
research and practice communities. Education Evaluation and Policy Analysis, 18(3), 199218.
Davies, H. T. O., & Nutley, S. M. (2008). Learning more about how research-based knowledge gets
used: Guidance in the development of new empirical research, working paper. New York:
William T. Grant Foundation.
Edwards, A., Sebba, J., & Rickinson, M. (2007). Working with users: some implications for
educational research. British Educational Research Journal, 33 (5), 647-661.
Fullan, M. (1992, June). Teachers as critical consumers of research. Washington, DC: Paper
commissioned by OECD for Seminar on Producers and Consumers of Research.
Gibbins, M., Limoges, C., Nowotny, H., Schwatsman, S., Scott, P., & Trow, M. (1994). The new
production of knowledge. London: Sage.
Head, B. W. (2008). Three lenses of evidence-based policy. Australian Journal of Public
Administration, 67(1), 1-11.
Hammersley, M. (2007) Introduction. In: M. Hammersley (Ed) Educational Research and Evidence‐
based Practice. London; SAGE.
Hemsley-Brown, J., & Sharp, C. (2003). The use of research to improve professional practice: a
systematic review of the literature. Oxford Review of Education, 29 (4), 449-471.
Hess, F. M. (Ed.). (2008). When research matters: How scholarship influences education policy.
Cambridge, MA: Harvard Education Press.
Hillage, J., Pearson, R., Anderson, A., & Tamkin, P. (1998). Excellence in research on schools (Report
no. 74). London: Department for Education and Employment.
Holbrook, A., Ainley, J., Bourke, S., Owen, J., McKenzie, P., Misson, S., & Johnson, T. (2000). Mapping
educational research and its impact on Australian schools In Department of Education
Training & Youth Affairs (Ed.), The Impact of Educational Research, (pp. 15-278). Canberra:
Commonwealth of Australia,.
Honig, M. I., & Coburn, C. (2008). Evidence-based decision making in school district central offices
toward a policy and research agenda. Educational Policy, 22(4), 578-608.
24
Huberman, M. (1990). Linkages between researchers and practitioners: A qualitative study.
American Educational Research Journal, 27(2), 363-391.
Kirst, W. M. (2000). Bridging education research and wducation policymaking Oxford Review of
Education, 26 (3-4), 379-391.
Knott, J., & Wildavsky, A. (1980). If dissemination is the solution, what is the problem? . Knowledge,
Creation, Diffusion, Utilization, 1(4), 537-578.
Landry, R., Amara, N., & Lamari, M. (2001a). Climbing the ladder of research utilization - Evidence
from social science research. Science Communication, 22(4), 396-422.
Landry, R., Amara, N., & Lamari, M. (2001b). Utilization of social science research knowledge in
Canada. Research Policy, 30(2), 333-349.
Landry, R., Lamari, M., & Amara, N. (2003). The extent and determinants of the utilization of
university research in government agencies. Public Administration Review, 63(2), 192-205.
Lester, J. P. (1993). The utilization of policy analysis by state agency officials. Knowledge: Creating,
Diffusion, Utilization, 14(3), 267-290.
Lester, J.P., Wilds, L.J., 1990. The utilization of public policy analysis: a conceptual framework.
Evaluation and Program Planning 13, 313–319.
Levin, B. (2011). Mobilising research knowledge in education. London Review of Education, 9 (1), 1526.
Levin, B., & Edelstein, H. (2010). Research, policy and practice in education. Education Canada, 50(2),
29-30.
Lingard, B. (2011). The impact of research on education policy in an era of evidence-based policy.
Public Lecture, The University of New South Wales, 18 November, 2011.
Lomas, J. (2000). Using linkage and exchange to move research into policy at a Canadian foundation.
Health Affairs, 19, 236-240.
Nutley, S., Walter, I., & Davies, H. (2007). Using evidence: How research can inform public services.
Bristol: Policy Press.
Oancea, A. (2005). Criticisms of educational research: key topics and levels of analysis. British
Educational Research Journal, 31(2), 157-183.
Orland, M. (2009). Separate orbits: The distinctive worlds of educational research and policymaking.
In G. Sykes, B. Schneider, D. Plank, & T. Ford (Eds.), Handbook of education policy research
(pp. 113-128). New York: Routledge.
Ouimet, M., Landry, R., Ziam, S., & Bedard, P. (2009). The absorption of research knowledge by
public civil servants. Evidence and Policy, 5(4), 331-350.
Rickinson, M., Sebba, J., & Edwards, A. (2011). Improving research through user engagement.
London: Routledge.
Saha, L. J., Biddle, B. J., & Anderson, D. S. (1995). Attitudes towards educational research knowledge
and policy-making among American and Australian school principals. International Journal of
Educational Research, 23(2), 113–126.
Vanderlinde, R., & van Braak, J. (2010). The gap between educational research and practice: Views of
teachers, school leaders, intermediaries and researchers. British Educational Research
Journal, 36(2), 299-316.
Weiss, C. H., & Bucuvalas, M. (1980). Social Science Research and Decision-Making. New York:
Columbia University Press.
Wilkins, R. (1988). Research and policy in teachers’ organizations. Journal of Education Policy, 3(2),
89-103.
Zeulie, J. (1994). How do teachers understand research when they read it? Teaching and Teacher
Education, 10(1), 39–55.
25
Table 1. Research Utilisation Scale
Variable
Transmission
Cognition
Reference
Effort
Influence
Application
I transmit my research results to end-users
My research reports have been read and understood by end-users
My work has been cited in reports and strategies by end-users
Efforts were made to adopt the results of my research by end-users
My research results have influenced the choices and decisions of end-users
My research has been applied by end-users
Table 2. Proportion of education respondents at each stage of the research utilisation
scale
n
%
No echelon
6
3.9
Transmission
3
1.9
Cognition
16
10.3
Reference
11
7.1
Effort
2
1.3
Influence
1
0.6
Application
117
75.0
156
100.0
26
Table 3. Internal reliability coefficients (Cronbach’s alpha) for variables – Education
research
Name of variable
RU Index
Researchers’ Context
Research targeted to user
Importance of academic funding
Importance of other funding
Benefits of collaborative research
Barriers academics experience in the transfer & uptake of
their research
Consequences of investing in research partnerships
Problems relating to the orientation of research
partnerships
User’s Context
End-users prioritise high quality research
End-users prioritise the useability of the research
End-users prioritise the feasibility of the research
Policy-makers have found my policy focussed research to
be relevant
Barriers policy-makers experience in transfer & uptake of
research
Dissemination
Importance of tailoring research when end-users are the
focus
Importance of meetings & dissemination activities with
end-users
Importance of using presentations and reports to present
research to parliamentary committees
Importance of using the media to present research
Interactions
Importance of using contacts, seminar and reports to
present research to policy-makers and practitioners
Importance of using contacts, seminar and reports to
present research to private sector organizations
Number
of cases
Number Cronbach
of items
alpha
in a scale
156
6
0.77
156
156
156
156
156
4
2
5
10
5
0.60
0.34
0.78
0.85
0.74
156
156
10
5
0.85
0.81
156
156
156
156
7
4
3
3
0.72
0.72
0.68
0.97
156
2
0.95
156
7
0.69
156
4
0.88
156
2
0.89
156
3
0.62
156
6
0.79
156
3
0.87
27
Table 4. Means and standard deviationsa Education research
Range
Research Utilization Ladder
Research Approach
Quantitative studies
Qualitative studies
Researchers’ Context
Research targeted to user
Importance of academic funding
Importance of other funding
Benefits of collaborative research
Barriers academics experience in transfer & uptake of research
Consequences of investing in research partnerships
Problems relating to the orientation of research partnerships
Research Time (teaching and research or research only
positions)
Number of external partnerships
User’s Context
End-users prioritise high quality research
End-users prioritise the useability of the research
End-users prioritise the feasibility of the research
Policy-makers have found my policy focussed research to be
relevant
Barriers policy-makers experience in transfer & uptake of
research
Dissemination
Importance of tailoring research when end-users are the focus
Importance of meetings & dissemination activities with endusers
Importance of using presentations and reports to present
research to parliamentary committees
Importance of using the media to present research
Importance of using refereed publications to present research
Interactions
Importance of using contacts, seminar and reports to present
research to policy-makers and practitioners
Importance of using contacts, seminar and reports to present
research to private sector organizations
Number of external grants
a.
Min
Max
Mean
1
5
3.64
Std.
Err.
0.05
0
0
1
1
0.40
0.83
0.04
0.03
1
0
0
0
0
0
0
4
5
5
5
5
5
5
2.37
4.03
3.49
3.36
3.90
3.68
2.67
0.04
0.06
0.07
0.07
0.06
0.06
0.07
0
1
0.81
0
35
6.22
0.49
1
1
1
5
5
5
3.91
4.64
3.90
0.05
0.03
0.06
0
5
3.54
0.12
0
5
3.23
0.08
0
5
4.24
0.04
0
5
4.21
0.05
0
5
2.41
0.12
0
0
5
5
3.20
4.71
0.07
0.05
0
5
3.77
0.06
0
5
2.91
0.10
0
44
7.76
0.56
Standard deviations only reported for continuous measures.
28
Table 5. Regression equations predicting utilisation of education research
M
Research Approach
Quantitative studies
0.14
Qualitative studies
-0.11
Researchers’ Context
Research targeted to user
-0.02
Importance of academic funding
-0.06
Importance of other funding
-0.05
Benefits of collaborative research
0.19***
Barriers academics experience in transfer & uptake of research
-0.06
Consequences of investing in research partnerships
-0.20**
Problems relating to the orientation of research partnerships
0.15*
Teaching & research position
0.05
Number of external partnerships
-0.01
User’s Context
End-users prioritise high quality research
-0.05
End-users prioritise the useability of the research
0.25*
End-users prioritise the feasibility of the research
-0.14*
Policy-makers have found my policy focussed research to be
0.07*
relevant
Barriers policy-makers experience in transfer & uptake of research
0.03
Dissemination
Importance of tailoring research when end-users are the focus
0.34**
Importance of meetings & dissemination activities with end-users
0.03
Importance of using presentations and reports to present research to
-0.00
parliamentary committees
Importance of using the media to present research
-0.01
Importance of using refereed publications to present research
-0.06
Interactions
Importance of using contacts, seminar and reports to present
0.13
research to policy-makers and practitioners
Importance of using contacts, seminar and reports to present
0.02
research to private sector organisations
Number of external grants
0.02***
Constant
1.28
Observations
155
Adjusted R2
0.28
SD
(0.10)
(0.14)
(0.10)
(0.06)
(0.06)
(0.07)
(0.08)
(0.10)
(0.09)
(0.13)
(0.01)
(0.10)
(0.13)
(0.07)
(0.04)
(0.05)
(0.14)
(0.11)
(0.04)
(0.06)
(0.08)
(0.08)
(0.04)
(0.01)
(0.79)
Standard errors in parentheses
*
p < 0.10, ** p < 0.05, *** p < 0.01
29
Figure 1. Number of education researchers climbing the echelons of the ladder of
knowledge utilisation – progression is subject to passing previous echelons
160
147
150
Number of researchers
140
131
120
118
120
117
100
80
Pass
Fail
60
40
20
16
6
11
3
2
0
Transmission Cognition
Reference
Effort
Influence
Echelons of the ladder of knowledge utilisation
1
Application
Figure 2. Number of education researchers passing each stage of research utilisation –
failure in one stage does not preclude passing subsequent stages
160
148
150
140
135
133
128
134
Number of researchers
120
100
80
Pass
Fail
60
40
28
21
20
6
23
22
8
0
Transmission Cognition
Reference
Effort
Influence
Stages of research utilisation
Application
30