Enhancing formative feedback in large cohort modules: a case study from CARBS by Cemil Selcuk Dept. of Economics Cardiff Business School Outline 1. Rankings: Feedback is a major issue. 2. This project: a controlled experiment 3. Survey results and some analysis Bristol, Cardiff, Lancaster Teaching is OK Feedback is an issue..! Source: NSS – http://unistats.direct.gov.uk/ Large Modules • Econ modules (Micro, Macro, Econometrics, Money and Banking etc.) are taken by other business majors. • Providing feedback in large modules is a challenge. No magic solution..! • Most modules have tutorials, with less than 20 students in each session. • Idea: Involve TAs in the feedback process. BS2550: Microeconomic Theory • Module size: >180 students • 14 Tutorials groups, 13-15 students each. 3 TAs • Students advised to solve questions before tutorials. Structure of Tutorials Week 3 Groups A1 – G1 Question Set 1 Week 4 Groups A2 – G2 Question Set 1 Week 5 Groups A1 – G1 Question Set 2 Week 6 Groups A2 – G2 Question Set 2 Feedback so far • • • • Oral feedback in lectures and tutorials. Answering questions via email. Answers for past examinations. Whole-class general feedback sheets (after Jan exams) A controlled experiment • Treatment – 25% of students • One TA committed to this: “Submit your own solution before the tutorial and I’ll return them in two weeks with corrections.” • Participation was voluntary; no actual grading. • Control – 75% of students • Remaining TAs did not practice this (did not have to). From Students’ Perspective • Voluntary participation; not graded. • Real effort by students. No incentive to cheat, so, monitoring is not an issue. • Self selection: students who care about feedback participate (Reduces hateful outliers in surveys, will come back to this). From TA’s perspective Not a lot of extra work because: • A TA sees at most 50 students and not every student participates. • Spread over two weeks. • Nobody fights back for extra points. • Corrections can be brief if time is short. Survey • Online survey at the end of the semester. • About 25% of all students were “treated”. • Question: Does the treatment make any difference on key feedback questions? - “I have received helpful feedback on my work.” - “Feedback (generic or individual) has helped me clarify things I did not understand.” Results “I have received helpful feedback on my work.” “Feedback has helped me clarify things I did not understand.” Treated students 82% 79% Regular students (excluding the treated) 53% 60% Cardiff Econ in NSS Survey 65% 37% Seems to be working...! Hateful Outliers “I have received helpful feedback on my work.” “Feedback has helped me clarify things I did not understand.” Treatment Group Av = 8.2, Min = 6 Av = 7.9, Min = 6 Control Group Av = 5.3, Min = 1 (Ten 1s, five 2s, ...) Av = 6.0, Min =1 (Eight 1s, three 2s,...) Cardiff Econ in NSS 65% ? 37% A lot of 1s. • 1 = “I hate this course.” Usually accompanied by other 1s. • Occurs if “service received” falls below a certain threshold. • Treatment improved the score by reducing/eliminating outliers Distribution of Scores Treated • In the uncontrolled group there is a significant number of “angry customers”. Control 0.5 0.45 0.4 0.35 Angry Customers 34% in total • They’re the main reason why the average is low. 0.3 0.25 0.2 • Upper tails are alike = little difference in the number of “happy customers”. 0.15 0.1 0.05 0 1 2 3 4 5 6 7 8 9 10 Distribution of Scores Treated Control 0.5 0.45 • With the treatment the mass in the lower tail is eliminated and spread over. 0.4 0.35 0.3 0.25 0.2 • Improving the average significantly..! 0.15 0.1 0.05 0 1 2 3 4 5 6 7 8 9 10 Other Questions Treatment Control How would you rate your attendance to lectures? (max =5, min = 1) 4.51 4.58 How many tutorials did you attend? (max = 4, min = 1) 3.53 3.58 Before each tutorial: 4- I solved all the questions on my own. 3 - I solved some questions on my own. 2 - I skimmed through the questions without attempting to solve them. 1 - I did not even look at the questions. * Av = 3.41 * Av = 2.78 * Min =3 * 26% chose 1 or 2. • No difference in attendance to lectures or tutorials. • Every student in the treatment group attempted to solve the questions. • In the control group 26% did not make an attempt or, worse, did not even look at the questions. Cardiff’s New Feedback Policy` • In line with the basic principles. • A good example for “feed-forward”. Suggestions if rolled out • Applicable only if there are sufficiently many TAs. Compensation? • TAs may need training. The objective is providing feedback; not marking. • Should not have any weight in grading, – Monitoring becomes a serious issue. – Kills the incentive to put in own effort. The End