Enhancing formative feedback in
large cohort modules: a case study
from CARBS
by
Cemil Selcuk
Dept. of Economics
Cardiff Business School
Outline
1. Rankings: Feedback is a major issue.
2. This project: a controlled experiment
3. Survey results and some analysis
Bristol, Cardiff, Lancaster
Teaching is OK
Feedback is
an issue..!
Source: NSS – http://unistats.direct.gov.uk/
Large Modules
• Econ modules (Micro, Macro, Econometrics,
Money and Banking etc.) are taken by other
business majors.
• Providing feedback in large modules is a
challenge. No magic solution..!
• Most modules have tutorials, with less than 20
students in each session.
• Idea: Involve TAs in the feedback process.
BS2550: Microeconomic Theory
• Module size: >180 students
• 14 Tutorials groups, 13-15 students each. 3 TAs
• Students advised to solve questions before tutorials.
Structure of Tutorials
Week 3
Groups A1 – G1
Question Set 1
Week 4
Groups A2 – G2
Question Set 1
Week 5
Groups A1 – G1
Question Set 2
Week 6
Groups A2 – G2
Question Set 2
Feedback so far
•
•
•
•
Oral feedback in lectures and tutorials.
Answering questions via email.
Answers for past examinations.
Whole-class general feedback sheets (after
Jan exams)
A controlled experiment
• Treatment – 25% of students
• One TA committed to this: “Submit your own solution before
the tutorial and I’ll return them in two weeks with
corrections.”
• Participation was voluntary; no actual grading.
• Control – 75% of students
• Remaining TAs did not practice this (did not have to).
From Students’ Perspective
• Voluntary participation; not graded.
• Real effort by students. No incentive to cheat, so,
monitoring is not an issue.
• Self selection: students who care about feedback
participate (Reduces hateful outliers in surveys, will come back to this).
From TA’s perspective
Not a lot of extra work because:
• A TA sees at most 50 students and not every
student participates.
• Spread over two weeks.
• Nobody fights back for extra points.
• Corrections can be brief if time is short.
Survey
• Online survey at the end of the semester.
• About 25% of all students were “treated”.
• Question: Does the treatment make any
difference on key feedback questions?
- “I have received helpful feedback on my work.”
- “Feedback (generic or individual) has helped me clarify
things I did not understand.”
Results
“I have received helpful
feedback on my work.”
“Feedback has helped me clarify
things I did not understand.”
Treated students
82%
79%
Regular students
(excluding the treated)
53%
60%
Cardiff Econ in
NSS Survey
65%
37%
Seems to be working...!
Hateful Outliers
“I have received helpful feedback
on my work.”
“Feedback has helped me clarify
things I did not understand.”
Treatment Group
Av = 8.2, Min = 6
Av = 7.9, Min = 6
Control Group
Av = 5.3, Min = 1
(Ten 1s, five 2s, ...)
Av = 6.0, Min =1
(Eight 1s, three 2s,...)
Cardiff Econ in NSS
65%
?
37%
A lot of 1s.
• 1 = “I hate this course.” Usually accompanied by other 1s.
• Occurs if “service received” falls below a certain threshold.
• Treatment improved the score by reducing/eliminating outliers
Distribution of Scores
Treated
• In the uncontrolled group
there is a significant number
of “angry customers”.
Control
0.5
0.45
0.4
0.35
Angry Customers
34% in total
• They’re the main reason why
the average is low.
0.3
0.25
0.2
• Upper tails are alike = little
difference in the number of
“happy customers”.
0.15
0.1
0.05
0
1
2
3
4
5
6
7
8
9
10
Distribution of Scores
Treated
Control
0.5
0.45
• With the treatment the
mass in the lower tail is
eliminated and spread over.
0.4
0.35
0.3
0.25
0.2
• Improving the average
significantly..!
0.15
0.1
0.05
0
1
2
3
4
5
6
7
8
9
10
Other Questions
Treatment
Control
How would you rate your attendance to lectures? (max =5, min = 1)
4.51
4.58
How many tutorials did you attend? (max = 4, min = 1)
3.53
3.58
Before each tutorial:
4- I solved all the questions on my own.
3 - I solved some questions on my own.
2 - I skimmed through the questions without attempting to solve them.
1 - I did not even look at the questions.
* Av = 3.41 * Av = 2.78
* Min =3
* 26%
chose 1 or
2.
• No difference in attendance to lectures or tutorials.
• Every student in the treatment group attempted to solve the questions.
• In the control group 26% did not make an attempt or, worse, did not
even look at the questions.
Cardiff’s New Feedback Policy`
• In line with the basic principles.
• A good example for “feed-forward”.
Suggestions if rolled out
• Applicable only if there are sufficiently
many TAs. Compensation?
• TAs may need training. The objective is
providing feedback; not marking.
• Should not have any weight in grading,
– Monitoring becomes a serious issue.
– Kills the incentive to put in own effort.
The End