Principles of Effective Evaluation Prof Keithia Wilson Academic Leader Student Success & Retention Importance of Evaluation Results in evidence-based practice Helps us to identify the most effective & strategic interventions (make a difference or add value) Leads to sustainable & enjoyable practice Provides accountability to School & Group leadership Contributes to scholarship of the FYE Kirkpatrick (1994) Model of Evaluation - Overview 4 levels of outcome evaluation – 1. 2. 3. 4. Level Level Level Level 1 2 3 4 – – – – Reaction Learning Behaviour Results Level 1 - Reaction Measures participant’s reactions to the training program, including: reactions to the overall program (outcomes) e.g., * “To what extent did you find the Orientation program useful?” reactions to specific components of the program (processes) e.g., * What aspects of Orientation did you MOST APPRECIATE and found USEFUL? * What aspects of Orientation did you LEAST APPRECIATE and feel is MOST IN NEED OF IMPROVEMENT? Level 1 - Reaction Important to include closed-ended items including rating scales e.g., How effective was the Orientation program overall in helping you to make a transition to university? not at all effective very effective 1 2 3 4 5 6 7 How useful was the initial information session on the predictors of success in first year? not at all useful very useful 1 2 3 4 5 6 7 As well as open-ended items What else would you have liked included in the Orientation program today? Level 1 - Reaction Some FYE examples – [email protected] Survey questions on Orientation (retrospective student survey) School specific Orientation Program evaluation at the end of O-Day & the Common Time (ongoing Orientation Program) – student surveys, focus groups Peer Mentoring Program final evaluation by mentors & mentees Level 2 - Learning Measures what participants have learned from involvement in the program What is measured needs to relate to what was covered in the program viz. learning objectives Typically covers knowledge, skills, or attitudes Needs to include both rating scales & open-ended questions Can include self-report & tests of actual knowledge Level 2 - Learning Some examples – 1. Student self-report * I feel confident that I know what to do to be successful academically this semester not at all confident very confident 1 2 3 4 5 6 7 * What have you learned about the transitions you will need to make as a commencing student in order to be successful? 2. Student knowledge * I know the name of my First Year Advisor yes no (circle one) * I know the difference between a lecture and a tutorial (circle one) Strongly Disagree | Disagree | Neutral | Agree | Strongly Agree Level 2 - Learning Some FYE examples – [email protected] Survey data evaluates learning outcomes in terms of knowledge & attitudes (e.g., knowing where to go to get help; perceptions of good teaching) Outcome evaluation for Peer Mentees (What are your 3 most important learnings from your participation in the Peer Mentoring Program?) First-Assessment-First-Feedback amber alert intervention (What have you learned as a result of your participation?) Level 3 - Behaviour Measures the transfer of knowledge, skills & attitudes from the training context to in vivo or real-life contexts (the classroom & study). Evaluate both before & after the program if practical Use survey, focus groups, interviews with students, mentors, staff (convenors, sessionals) [email protected] Survey (Did you attend Orientation? How much time are you spending on study?) Level 3 - Behaviour Some FYE examples – [email protected] survey data Attendance at E-Day, O-Day & Common Time/ASP Program Attendance at lectures & tutes Assessment submission rates in courses Level 4 - Results Measures “return-on-investment”, or the extent to which the program has produced results. Some FYE examples include – Hard student outcomes such as * pass rates in courses for assessment items * GPA in semester 1, 2 & FY overall * student retention into year 2 Soft student outcomes such as – * student satisfaction Evaluation Key question – What difference do our FYO&E interventions (e.g., Orientation, Peer Mentoring, Attendance, First Assessment Intervention etc.) make overall? Program/Intervention1 Reactions2 Learning 3 Behaviour 4 Overall Productivity/Result Program/InterventionFeelKnowDowhat is the result or difference overall?