Evaluating the impact of research projects in tertiary learning
and teaching: exploring the geography of change
Plenary Address, HERDSA Conference, Auckland
1:45 –2:30 pm, Tuesday 2nd July 2013
Kirsty Weir & Peter Coolbear (Ako Aotearoa)
Tilly Hinton (Office of Learning and Teaching & University of the Sunshine Coast
Plan of our presentation
• Why? – why are we doing this?
• Context – the impact landscape
• Practice - activities undertaken in Australia and New
Zealand to better understand the impacts of funded
learning and teaching research
• Reflections - what this means for researchers,
institutions and funding agencies
Why Measure impact of research into teaching and
learning?
•
Moral purpose
•
Addressing a value for money agenda - part of the
funding landscape (PBRF, ERA, etc):
•
Identifying ways to gain additional leverage from
work already done
• Improving funding body processes
• Understanding the benefits that result from learning and teaching research
• Surfacing stories from project teams that could otherwise go unheard
Funding overview
• Ako Aotearoa funds evidence-based change projects
with a high potential to benefit learners. There is only
one, primary objective of Ako Aotearoa funding – to
enhance educational outcomes for learners.
• The OLT funds “academics and professional staff to
investigate, develop and implement innovations in
learning and teaching. Grants facilitate scholarship
and research into learning and teaching, and promote
systemic change in the sector.”
What do we mean by impact?
Outcomes
Outputs
Deliverables
Learner benefit
What do we mean by impact?
“None of these terms is
entirely satisfactory, but we
lack a better vocabulary.”
Levin (2004, p. 2)
Louise Carter
Year 2 Massey University Design student
•
“…impact occurs when research, in any of its multiple forms, makes a
difference to subsequent actions that people take or refrain from taking.”
(Levin 2004)
•
impact is determined when there is a “clear demonstration of learner
benefit.” (Alkema 2012)
•
“an effect on, change or benefit to the economy, society, culture, public
policy or services, health, the environment or quality of life, beyond
academia” (UK Research Excellence Framework definition)
•
“impact is about people not publications” (Stilgoe 2011)
So we need to recognise and manage the risks here
• Adopting an overly transactional model
• Over simplification of the change process
• Assumptions about causality
• Assumptions about generalisability
The Ako Aotearoa Impact Evaluation Framework (IEF)
APPLIED IN NEW ZEALAND:
• Conversations with project teams at
6, 12 and 24 months post project
completion. Focusing on:
• outputs
• impact on practice
• impact on learners
• impact on project teams
APPLIED IN AUSTRALIA:
• A pilot of the Ako approach all aspects
but for a random sample of 18 projects
•
Conversations with project
leaders/teams once only, focussing
on:
–
–
–
–
–
Changes in practice
Student benefits
Impact on the project team
Future plans
Deliverables
•
Evaluative conversations placed in
the context of the original project
rationale and goals
•
Applied to all projects Ako funds
(including retrospectively)
•
Applied to a random sample of
projects across all grant types
•
Influence Stories written for 15
projects
•
Influence Summaries written for 18
projects
Why post-project conversations?
• Conversations surface unrecognised benefits
• Change processes take time, almost always beyond the final report
date
• Relationships between projects and funding body are valuable
• Student learning is central
• Evidence-based culture
• Meaning is made collaboratively
• Opportunities for further benefits can be unearthed through the
conversation
• An opportunity to reflect and be heard
Why stories or case studies?
•
Case studies address the causality problem by
providing context and identifying:
 Enablers
 Barriers
 Other change drivers
• As case studies develop, they tell us about the
directions of travel and differing time-frames of change
• Projects are taken on their own terms, rather than
applying metrics or indicators
“stories”
“narratives”
“case studies”
“summaries”
What this means for researchers
• A research plan which moves to implementation of
findings
• Understand climate of readiness for change,
engagement, and transfer
• Willingness to tackle tough change agendas
• Move from requesting to dollars to selling an
opportunity for change/benefit
• Recognition from funders that change takes time and
is ongoing
What has funding made possible?
•
•
•
•
•
•
•
•
Online graduate attributes system deeply embedded across an entire
school
Education students supported to develop resilience, a factor in retaining
staff in the teaching profession
Report cited as key evidence for a wage increase across the profession
Emergent academics have ongoing access to mentorship and peer
support
Almost three quarters of a million dollars of industry and philanthropic
funds secured to further the work started by the grant
Special issues of journals that further explore themes of the grant
Publications from a grant cited fifty-eight times in other publications
Work integrated learning experience changed for all students in the
discipline after project negotiated a change in accreditation standards
What does this mean for learners?
8
15 Major NZ projects:
benefits to learners
7
6
5
Actual impact (evidence)
4
Observed impact
3
2
1
0
Academic enhancement Learning environment and
resources
Relationships
Personal development
Ako Aotearoa National Projects – the running score
sheet:
14 of 15 projects have had identifiable impact on practice
and / or learners. In total these have had an:
• impact on the practice of 2,400 staff
• 50,000 students have been offered enhanced learning
opportunities
What this means for institutions
• Processes for tracking and enabling impact
• Understand, lever and strategically request funding
from funders
• A longer term investment, projects only notionally
“finish” on submission of the final report
• Build a culture focussed on impact
• Aspects could be scaled into institutional funding
models
What this means for funding agencies
•
•
•
Ako Aotearoa
Government endorsement
The Impact Evaluation
Conversations are
interventions in their own right
Further leverage by Ako
Aotearoa and project teams to
enhance impact
Office for Learning and
Teaching
The pilot during 2012-13 has
been well-received in the
sector.
The model is not scalable
directly into the Australian
context.
Work is underway to translate
the strengths of the approach
into a scalable set of strategies
Ako Aotearoa has established rubrics for the assessment
of future proposals
Rating
Project integrity
Achieving sustainable
change
Excellent
For a project to be considered
‘excellent’’, all aspects must be met
• The project design and method
has no significant gaps or areas
of weakness
• How learner outcomes will be
achieved is well reasoned
• The project team has mana
relevant to the proposed work
• Capability building is embedded
in the project
• The focus of the project is highly
valued in New Zealand and
internationally
For a project to be considered
‘excellent’’, all aspects must be met
• The organisation develops a clear
commitment to act on the
outcomes of the work
• Relationships are established with
key stakeholders
• The dissemination plan is well
developed
• The project team is actively
engaged in achieving sustained
change for learners beyond the
scope of the project and
participating organisations
Mapping the complexity of impact
“Necessarily then the
academic community
ought move towards
a critical map of
impact networks via a
cartography of
impact(s).”
Massey University engineering design students
(Watermeyer, 2012)
Peter Coolbear
Tilly Hinton
P.Coolbear@massey.ac.nz
tilly.hinton@innovation.gov.au
tilly.hinton@usc.edu.au
@matilda_marie
www.akoaotearoa.ac.nz
http://sayat.me/TillyHinton
http://olt.gov.au/secondmentprojects