Principles of Effective
Evaluation
Prof Keithia Wilson
Academic Leader Student Success
& Retention
Importance of Evaluation





Results in evidence-based practice
Helps us to identify the most effective &
strategic interventions (make a difference
or add value)
Leads to sustainable & enjoyable
practice
Provides accountability to School & Group
leadership
Contributes to scholarship of the FYE
Kirkpatrick (1994) Model
of Evaluation - Overview
4 levels of outcome evaluation –
1.
2.
3.
4.
Level
Level
Level
Level
1
2
3
4
–
–
–
–
Reaction
Learning
Behaviour
Results
Level 1 - Reaction
Measures participant’s reactions to the training
program, including:

reactions to the overall program (outcomes) e.g.,
* “To what extent did you find the Orientation
program useful?”

reactions to specific components of the program
(processes) e.g.,
* What aspects of Orientation did you MOST
APPRECIATE and found USEFUL?
* What aspects of Orientation did you LEAST
APPRECIATE and feel is MOST IN NEED OF
IMPROVEMENT?
Level 1 - Reaction

Important to include closed-ended items
including rating scales e.g.,
How effective was the Orientation program overall in helping you to
make a transition to university?
not at all effective
very effective
1
2
3
4
5
6
7
How useful was the initial information session on the predictors of
success in first year?
not at all useful
very useful
1
2
3
4
5
6
7

As well as open-ended items
What else would you have liked included in the Orientation
program today?
Level 1 - Reaction
Some FYE examples –
 Starting@Griffith Survey questions on
Orientation (retrospective student survey)
 School specific Orientation Program
evaluation at the end of O-Day & the
Common Time (ongoing Orientation
Program) – student surveys, focus groups
 Peer Mentoring Program final evaluation by
mentors & mentees
Level 2 - Learning
Measures what participants have learned from
involvement in the program
 What is measured needs to relate to what was
covered in the program viz. learning objectives
 Typically covers knowledge, skills, or attitudes
 Needs to include both rating scales & open-ended
questions
 Can include self-report & tests of actual knowledge
Level 2 - Learning
Some examples –
1.
Student self-report
* I feel confident that I know what to do to be successful academically
this semester
not at all confident
very confident
1
2
3
4
5
6
7
* What have you learned about the transitions you will need to make
as
a commencing student in order to be successful?
2. Student knowledge
* I know the name of my First Year Advisor
yes
no (circle one)
* I know the difference between a lecture and a tutorial (circle one)
Strongly Disagree | Disagree | Neutral | Agree | Strongly Agree
Level 2 - Learning
Some FYE examples –
 Starting@Griffith Survey data evaluates learning
outcomes in terms of knowledge & attitudes (e.g.,
knowing where to go to get help; perceptions of
good teaching)
 Outcome evaluation for Peer Mentees (What are
your 3 most important learnings from your
participation in the Peer Mentoring Program?)
 First-Assessment-First-Feedback amber alert
intervention (What have you learned as a result of
your participation?)
Level 3 - Behaviour
Measures the transfer of knowledge, skills & attitudes
from the training context to in vivo or real-life
contexts (the classroom & study).
 Evaluate both before & after the program if
practical
 Use survey, focus groups, interviews with students,
mentors, staff (convenors, sessionals)
 Starting@Griffith Survey (Did you attend
Orientation? How much time are you spending on
study?)
Level 3 - Behaviour
Some FYE examples –
 Starting@Griffith survey data
 Attendance at E-Day, O-Day &
Common Time/ASP Program
 Attendance at lectures & tutes
 Assessment submission rates in
courses
Level 4 - Results
Measures “return-on-investment”, or the extent to
which the program has produced results.
Some FYE examples include –
 Hard student outcomes such as * pass rates in courses for assessment items
* GPA in semester 1, 2 & FY overall
* student retention into year 2
 Soft student outcomes such as –
* student satisfaction
Evaluation
Key question – What difference do our FYO&E
interventions (e.g., Orientation, Peer Mentoring,
Attendance, First Assessment Intervention etc.)
make overall?
Program/Intervention1 Reactions2 Learning 3
Behaviour 4 Overall Productivity/Result
Program/InterventionFeelKnowDowhat is the
result or difference overall?