Evaluating and improving
student achievement based on
pre-set criteria
Brian Whalley
(b.whalley@sheffield.ac.uk)
(For more information and resources see:
http://brianbox.net/pedagogies/criteria/criteria.html)
Various things in the notes for each slide plus a
few references at the end
We’re going to look at
• Assessment
• Feedback
• Supporting students
and, by the way,
Talk about a variety of aspects of education
- whatever you want!
(But we shall need to use a bit of ‘jargon’)
What is it like to be a student?
(with apologies to Chiroptera and Thomas Nagel)
Please discuss with a
neighbour
What you mean by:
1. Feedback
2. Feedforward
What do (you think) students mean (ie think
we mean) by these terms?
Outcomes
(Constructive) alignment
Alignment thus …
Learners construct meaning from what they do to learn.
[This concept derives from cognitive psychology and
constructivist theory, and recognizes] the importance of
linking new material to concepts and experiences in the
learner's memory, and extrapolation to possible future
scenarios via the abstraction of basic principles through
reflection.
The teacher makes a deliberate alignment between the
planned learning activities and the learning outcomes.
This is a conscious effort to provide the learner with a
clearly specified objective, a well designed learning
activity or activities that are appropriate for the task,
and well designed assessment criteria for giving
feedback to the learner.
Feedback
• Should be incorporated into the
assessment
• Students learn most by ‘feedback’
– Geoff Petty
So we need to bring feedback into the
alignment process
when to give feedback
When can you?
When should you?
When do you?
Do others in your department agree on
when you do?
Do others in your department give
‘proper’ feedback anyway (module
scheme)?
The Ten Principles of Assessment
and Feedback
•
1. Help clarify what good performance is
•
•
•
•
•
via aims, criteria, standards
2. Encourage ‘time and effort’ on challenging
learning tasks.
3. Deliver high quality feedback information that
helps learners self-correct.
4. Encourage positive motivational beliefs and
self-esteem.
5. Encourage interaction and dialogue around
learning (peer and teacher student)
(Nicol 2007)
10 principles continued
• 6. Facilitate the development of self-assessment
and reflection in learning.
• 7. Give learners choice in assessment – content
and processes.
• 8. Involve students in decision-making about
assessment policy and practice.
• 9. Support the development of learning
communities.
• 10. Help teachers adapt teaching to student
needs.
How do we use exams (and coursework)
to deliver these?
It’s all about assessment …
’Students can escape bad teaching but they cannot escape bad assessment’’
(David Boud, 1995)
And thus about
and a bit about
Feedback
Feedforward
So we’ll need to clarify things a bit and explain
something about jargon
and so we need a plan ….
Hmmm…..
I think we do this too often
Constructive Alignment
is the underpinning concept
behind the requirements for
programme specification,
declarations of Intended
Learning Outcomes (ILOs)
and assessment criteria, and
the use of criterion based
assessment.
Towards a better learning system
(‘Abolish feedback’)
It depends what you mean by……
• Information as ‘thing’ or ‘concept’
• Feedback as ‘thing’ or ‘concept’
– We are not explicit in this, students do not often/always
know what we mean?
– And do we know….?
Avoid:
Positive feedback = good
Negative feedback = bad
Feedforward = information in advance
Feedback
in a mechanical control system
The centrifugal governor
Is used to keep shaft speeds constant - but, for
students - we want to increase the ‘rate of
learning/education’, call this ‘attainment’
Feedback and Feedforward
in a control system sense
Do ‘exams’ provide a reference input (or even output?)
A MCQ might do this. Criterion Referencing might too.
Educational ‘ramp up’
To raise the level of ‘attainment’
We need controlled input, ramping up.
– Output, achievement, results
Ee
Ee
Ese
T
attainment
Ee
Marks, no remarks for
Exam events
Delayed marks and remarks for
Essay event
Task set with Criterion
Referenced assessment
Can I be more constructive?
Bringing feedback into modules
start
Ee
‘Attainment’
Ee
Activity
T
MR
Ee
How do we
increase the
gradient?
Delayed marks and remarks for Exam event, Ee
E
e
Task
end
Marks
A
M
Remarks
T
MR
MR
Two Activity
c/work
A
MR
T
A
Two Activity
c/work + exam
R
A
Gradient =
‘learning rate’
The ‘connoisseur
approach’ to
assessment
They want ‘the highest
standards’ - achievements
How do competitors know
they have achieved?
feedback
What do you (think you) mean by:
• Standards
• Criteria
• Rubric
Remarks: ‘You did quite well …’ (to an
American) = ‘You did very well’
‘Marks’ and ‘Remarks’ (my usage)
Moral: students and tutors need to make sure that the
terms they use are understood by everyone
concerned; this includes terms like ‘feedback’
‘Marks’ and ‘Remarks’
• Marks - are just that, for a piece of work
– (This begs the Q of what the work is….)
• Remarks - the comments that go with it
• The trick is to get appropriate remarks to go with the
marks, as quickly as possible
[Preflights and After Action Reviews help
– Increases the frequency of control ‘shoves’
– Helps avoid ‘sticking points’ (Troublesome knowledge etc)
– Increases student-tutor contact]
A quick look at ‘Standards’
• UPPER SECOND
• Students are able to demonstrate the
following, with respect to the criteria relevant
to their discipline:
o a thorough understanding of key concepts and
knowledge, and of a range of supporting evidence;
o informed consideration of key issues and interpretation of
evidence;
• ability to integrate material from a variety of sources, and
to deploy established techniques of analysis and enquiry,
accurately and effectively;
etc
University of Sheffield, 29_Assessment.doc
Also ‘Grade Descriptors’
• These are from the tutor’s point of view.
• How do students know how to achieve
designated grades?
• How does ‘feedback’ help with this
process?
• How are standards maintained? Eg by
various markers?
‘I know a 2:1 when I see it; understanding criteria for
degree classification in franchised university
programmes’ Ecclestone, K. (2001) Journal of Further and
Higher Education 25: 301-313
Assessment criteria
are descriptions of what the learner has
to do in order to demonstrate that the
learning outcomes have been achieved.
[threshold standard; minimum
expectation to achieve credit]
Dumbing down?
Actually, you are coaching
(mentoring, providing scaffolding etc)
• By providing exemplars of how to achieve
the standards
• By showing students their attainments (or
not)
• To show how students can better achieve
and
• Gain confidence in their work (and
achievements)
‘Everyone’ needs trainers
Umm, sometimes it works ……
Teamwork
Marshall Goldsmith
Feed forward
It is not judgmental.
The negative connotations of past failures are banished
There is no such thing as failure just Feedforward.
It is much easier to deliver.
People are less defensive when discussing future performance
Feedforward is taken less personally, provokes less resistance.
Charting
• DIDET project
• Allison Littlejohn
– Flexible learning
Undergraduate-postgraduatePhD and the wider community
charting
Supporting
employers
and
employment
using
another 3Cs
Slides from:
Littlejohn, JISC
Presentation, 2008
Using criterion referenced
assessment
•
•
•
•
•
•
•
Avoid the ‘connoisseur’ approach
Show students what is needed to accomplish tasks
Provide remarks with the marks
Marks and remarks go hand in hand
Give a sense of achievement
Thus, criteria need to be assignment specific and so
Link all this back to JITT (Just in Time
Teaching/Preflights) and AAR (After Action
Reviews)
– Providing proper feedback and feedforward
Examples etc:
In general:
1. Determine the Task and set the Activity
2. Set and refine your Objectives and Learning Outcomes
3. Set the Assessment Criteria (4 - 7 are usually enough)
4. Note potential problems, ‘troublesome knowledge’ etc and provide
assistance with this (eg ‘preflights*’ before the main activity)
5. Students do the activity, write it up, paying attention to the criteria
6. You mark it according to the criteria (if you have helpers this makes it
easier for them as the criteria are easily defined and adhered to)
7. Record marks for each criterion in a spreadsheet row (and add a few
comments in a cell), use these to calculate the task’s mark profile (more
feedback)
8. E-mail out the row of marks and remarks, separately, to each student, in
one go using a nifty-tifty bit of php scripting§.
Learning outcomes
At the end of the module students will be expected to be able
to:
 Plan and structure a major piece of work
 Use referencing appropriate to a published paper
Assessment Task
• In week 9 you are required to submit a plan for the
dissertation. The plan should include the following:
 a summary of the main argument of the dissertation (200
words);
 an indication of the structure, section headings and outline of
the content of each section (500 words);
 indication of, and full references for, texts consulted and
texts to be discussed in the dissertation.
Assessment Criteria
Writing your criteria
1. Consider the learning outcomes being tested.
2. Consider the assignment you wish to set.
3. Brainstorm requirements for, or attributes of,
successful performance in the assignment.
4. If necessary, specify the level of difficulty and the
particular context for the assignment.
5. Focus on what is essential and categorise the
requirements into clear criteria.
6. Check that criteria are assessable (valid and
reliable) and that they are clearly worded and
unambiguous (ask students to explain what they
think you mean).
(based on an example in Gosling and Moon, 2002)
Marking and criteria
(an example from the dissertation preparation case)
Evidence of study/reading in preparation of the
plan (20%)
Articulation of a clear idea/theme to be
discussed in the dissertation(30%)
Design of a workable and defensible structure
for the dissertation (30%)
Identification of texts to which the dissertation
will refer using the Harvard referencing
system (20%)
Although the criteria here are quite broad it at least
shows how much effort should be expended on each
section, e.g. don’t overload it with a literature review.
Practical (poster) assessment
Take this poster;
how might it be assessed?
‘hmm, a low 2.1, perhaps 61%?’
Not very a good assessment method is it?
Malcolm Knowles was doing better in 1975
with a few criteria and a Likert scale for each.
This ‘connoisseur approach’ is also used for:
Essays; essay exams; dissertations; projects
(The connoisseur approach does not do well in fulfilling
the good practice of Gibbs, Nicol and McFarlane-Dick.)
Criteria in the mark scheme for Experiment 1 of the
practical 1 from the previous slide
The table below shows marks achieved for each of 10 criteria of an
exercise.
Each student gets an individual e-mail with the marks for each
components and some remarks.
Setting out a learning design shows not only how the
task should be set up but where criteria for
assessment can be identified.
Task
Task
Problem
specified
S
Task
Manipulation
Problem
of
solution
system
Assessment
T
R
R
Rules
System
Examples
Procedures
Resources
Clues
Guidance
Help
Support
Provision of
Feedback on
solutions
Support
T Tacit knowledge required
S Sticking point(s) likely
A temporal sequence describing a rule-based learning design, In Oliver et al. 2007.
Describing ICT-based learning designs that promote quality learning outcomes
A final thought:
adding the veg
We have lots of ideas
Technology is helping greatly
but
What we need to do is keep reviewing the
teaching-learning process:
‘ …… the time, energy and resources
students devote to activities designed to
enhance learning at university.’
(Kerri-Lee Krause, 2005)
A few extra references to get
the basic concepts
• University of Plymouth Assessment
handbook
(www.plymouth.ac.uk/files/extranet/docs/TLD/Assessment
guidelines March09.pdf)
• University of Bath, ‘How to use assessment
criteria’
(http://www.bath.ac.uk/learningandteaching/the
mes/assessment-criteria/)
• Stevens, D. D., & Levi, A. J. (2005) Introduction
to Rubrics, (Sterling, VA: Stylus).