Effective dissemination and evaluation
Workshop on Teaching and Learning Grants
and Carrick Schemes, 2008
Merrilyn Goos and Clair Hughes
m.goos@uq.edu.au; clair.hughes@uq.edu.au
Session overview
• A project scenario
• Dissemination and evaluation defined
• Purposes of dissemination and
evaluation
• Small group development of
dissemination and evaluation strategies
• Resources
A project scenario
• “Introduction to University Teaching and Learning” program:
aims, objectives, participants, teaching team
• New course structure and content from 2008
• How to disseminate this innovation - not just to inform
potential participants and other stake-holders but to embed
good practice?
• How to evaluate the course in its first offering in order to
monitor and improve its implementation?
Dissemination defined
Dissemination is more than distribution of
information or making it available in some
way. Dissemination also requires that some
action has been taken to embed and upscale
the innovation within its own context and to
replicate or transform the innovation in
a new context and to embed the innovation in
the new context.
Carrick Institute Dissemination Framework
Evaluation defined
Evaluation is the process of determining the merit
and/or worth of things for a range of purposes.
Consider funding agency requirements for evaluation:
• Ensure that evaluation is undertaken at multiple points
throughout the project and is understood and reported within an
evaluation framework.
Carrick Competitive Grants program: Guidelines and Supporting Information – 2008 (p.23)
• Must provide a strategy for the evaluation of the project with
information on outcomes for students, whether there has been a
noticeable improvement or not in the expected outcomes for
students and how the project outcomes will be disseminated to
others within the discipline.
UQ Teaching and Learning Small and Large Grants Scheme Guidelines
Purposes of project dissemination
Dissemination for
awareness
(scattering the seed to
the wind)
Information provision
Distributing information through reports,
publishing papers, conference
presentations, sending emails, developing
websites.
One way information process.
Dissemination for
understanding
(sowing the seed in
prepared soil)
Information provision
Distributing information in a more
purposeful manner, e.g. workshops, visits.
Requires interaction with others who are
involved in the project.
Dissemination for action
(propagating, breeding,
growing, grafting)
Engaged dissemination
Purposeful, directed, systematic, proactive
engagement, evidence based, involves
adaptation and implementation, resulting
in changes in practice.
Purposes of project evaluation
Formative
(to inform ongoing project
decision-making)
•Monitor project progress
•Revise/enhance project processes and outcomes
•Engage key stakeholders early and during project to:
–demonstrate respect
–intensify participation
–collaborate in identifying key indicators of success
–progressively expose to new perspectives, or information
•Make interim reports
Summative
(to determine/support
project findings and
outcomes)
•Report on success/results/value
•Demonstrate impact
•Account to sponsors
•Sustain project outcomes
•Support related submissions (eg grants for funding for further
related projects; promotion; teaching and learning awards)
Enlightenment
(to learn)
•Provide opportunities for project team to maximise the learning
developed through participation in the project
Adapted from “The Learning Partnership” 2007 and W. K. Kellogg Foundation
Discussion
• In groups of 3 or 4:
– address the following dissemination and evaluation
questions in the context of the project scenario presented
earlier (new “Introduction to University Teaching and
Learning” program)
– Consider how you would address dissemination and
evaluation for a project in your own educational context
– Share your responses with the larger group
Dissemination questions
Who are the users of and other
stakeholders in this program?
What do we want to disseminate (process,
ideas, products)?
How could we make users and other
stakeholders aware and involved?
How could we engage users and other
stakeholders so they use and/or support
the program?
How could we obtain feedback from users
during the project?
How will we know there has been change in
practice?
Evaluation questions
Who is our project audience
(stakeholders)?
Select two stakeholders/groups and list
one or more questions each may have in
relation to this project.
How could we collect data to answer
these questions? (methods)
From whom or where (stakeholders and
others) could data be collected?
(sources)
What will count as evidence? (consider
both qualitative and quantitative)
The evaluation framework
EXAMPLE
Evaluation
questions
Sources/Methods
Institutional
contacts
What is Generic
Graduate Attribute
(GGA) policy in
Australian
universities?
Telephone interview
(to clarify, address
omissions or current
activity)
What are current
barriers to effective
GGA implementation?
Focus groups to elicit
responses to issues
paper based on
literature review
Expert reference Existing records/
group
artefacts
Web search of
university policy
documents
Written responses to
draft issues paper
Literature review
(research into
implementation
experience)
Analysis of AUQA
reports
Models of planning for evaluation
Implementation
Evaluation
Implementation
Evaluation
Dissemination
Dissemination
Implementation
Evaluation
Dissemination
Implementation
Evaluation
Dissemination
References
•
•
•
•
•
•
•
•
Carrick Institute for Learning and Teaching in Higher Education. (2006). Carrick Institute Dissemination
Framework Retrieved from
http://www.carrickinstitute.edu.au/carrick/webdav/site/carricksite/users/siteadmin/public/disseminati
on_carrickframework_2006.pdf
Carrick Institute for Learning and Teaching in Higher Education. (2008). Competitive Grants program:
Guidelines and Supporting Information – 2008. Retrieved from
http://www.carrickinstitute.edu.au/carrick/webdav/site/carricksite/users/siteadmin/public/grants_com
petitive_guidelines2008_july07.pdf
Evaluating projects resource A resource to assist applicants planning the evaluation framework
component of their Grants Scheme application has been prepared by Murdoch University
<http://www.murdoch.edu.au/teach/carrick_evaluation/index.html>
Felton, P., Kalish, A., Pingree, A., & Plank, K. (2007). Towards a scholarship of teaching and learning in
educational development. In D. Robertson & L. Nilson (Eds.), To Improve the Academy: Resources for
Faculty, Instructional, and Organisational Development (pp. 93-108). Boston: Anker.
The Learning Partnership. (2007). Material distributed during Leading and Managing Projects, a workshop
sponsored by the Carrick Institute for Learning and Teaching in Higher Education. Melbourne, 17 – 18
September.
TEDI Evaluation Services http://www.tedi.uq.edu.au/evaluations/index.html
The University of Queensland. (2007). UQ Teaching and Learning Small and Large Grants Scheme
Guidelines. Retrieved 29 October, 2007, from http://www.uq.edu.au/teachinglearning/download/TLStrategicGrantsGuidelines07.pdf
W. K. Kellogg Foundation. Executive summary: Introducing Program Logic Models. Material distributed
during The Professional Development Program for Medical and Healthcare Educators offered jointly by
the University of Queensland and Harvard Medical International. 21-25 August, 27-29 November, 2006.