Designing assessment with the
‘Assessment Design Decisions’
framework
Phillip Dawson
Office of the Vice-Provost
(Learning & Teaching)
Monash University
The ‘Assessment Decisions’ team
• Dr Phillip Dawson,
Monash (Co-lead)
• A/Prof Margaret
Bearman, Monash (Colead)
• A/Prof Liz Molloy,
Monash
• Prof David Boud, UTS
• A/Prof Gordon Joughin,
UQ
• A/Prof Sue Bennett, UOW
• Dr Matt Hall, Monash
Agenda
1.
2.
3.
4.
5.
“Assessment design decisions?”
Your assessment designs
The framework
Applying the framework
Summary and close (and feedback!)
1. “Assessment design decisions?”
We know a lot about
ideal practice
Hattie, 2009
Actual
practice is
different
Research suggests …
• Changing assessment ‘thinking’ in academics
doesn’t necessarily change practice (Offerdahl
& Tomanek, 2011)
• “…achieving a balance between summative
and formative assessment requires complex,
contextual thinking” (Price, et al., 2011)
Focus on assessment design
• Assessment policy
and procedures
• Decisions in the
design and
implementation of
assessment
• Judgements about
student work
‘assessment
design
decisions’
2. Your assessment designs
Your designs
• Think of an assessment design
– It might be new
– You might be thinking of revising it
• What is the impetus for change?
• Pair-share
Absorb the complexity, but don’t try to read
Giving
instructions to
students
Professional
identity
Past experience
Marking/
feedback
Beliefs about
assessment
Structuring
Professional
learning
Educator
Activities
Degree of
student agency
Strategies
Promoting
student
engagement
Degree of
difficulty
Impetus
Influences
Personal
characteristics
Backwash
Student
learning
Environment
Save resources
Ineff ective
assessment
Unit/ program
Institutional
culture
Institutional
requirements
Resources
3. The framework
Framework design
• Joined research data with conceptual
frameworks, and literature
• Drew from own experience as expert
practitioners
• Oriented towards educators’ agency rather
than prescriptive
Engaging with the framework
• One-page summary
• Online ‘guide’ with detailed information,
resources and educator experiences
• 55-page document version of ‘guide’
• (as well as a ‘Five insights’ document to give
to L&T leaders)
• http://assessmentdecisions.org
4. Applying the framework
Applying the framework
• Focus on a specific unit
• We will work through the six framework
components
• Address key questions on one-pager
• Use hard copy or online versions
5. Summary and close (and feedback)
Support for this project/presentation/resource
has been provided by the Australian Government
Office for Learning and Teaching.
The views expressed here do not necessarily
reflect the views of the Australian Government
Office for Learning and Teaching.
assessmentdecisions.org
phillip.dawson@monash.edu
philldawson.com
References
• Carless, D. (2009). Trust, distrust and their impact on assessment
reform. Assessment & Evaluation in Higher Education, 34(1), 79-89.
doi: 10.1080/02602930801895786
• Hattie, J., The Black Box of Tertiary Assessment: An Impending
Revolution, in Tertiary Assessment & Higher Education Student
Outcomes: Policy, Practice & Research, L.H. Meyer, et al., Editors.
2009, Ako Aotearoa: Wellington, New Zealand.
• Offerdahl, E. G., & Tomanek, D. (2011). Changes in instructors'
assessment thinking related to experimentation with new
strategies. Assessment & Evaluation in Higher Education, 36(7), 781795. doi: 10.1080/02602938.2010.488794
• Price, M., Carroll, J., O'Donovan, B., & Rust, C. (2011). If I was going
there I wouldn't start from here: a critical commentary on current
assessment practice. Assessment & Evaluation in Higher Education,
36(4), 479-492. doi: 10.1080/02602930903512883
Extra slides after this point
How do university teachers
make decisions about
assessment?
Research design
• 30 semi-structured
interviews
• Gritty, coalface,
‘actual’ not ‘ideal’
• Thematic analysis;
coding against
framework
• Meaning-making
from coded data
Arts Professions
Arts – Nonprofession
Sciences Professions
Sciences –
Nonprofession
What we can say
• Improving assessment is more than just a
problem of knowledge
transmission/translation
• Rarely about rationally selecting from options
• Assessment decisions are complex; situated;
pragmatic
Impetus
Influences
Activities
• Opportunity or driver for
change
• Environment
• Educator
• Making it work
New to role/ job
Meeting student
needs
New unit
Impetus
Program level
changes/ review
Save resources
Part of a project
Ineff ective
assessment
Professional
identity
Past experience
Beliefs about
assessment
Professional
learning
Educator
Cohort
Influences
Personal
characteristics
Backwash
Student
learning
Environment
External factors
Unit/ program
Institutional
culture
Institutional
requirements
Resources
Giving
instructions to
students
Marking/
feedback
Reducing
cheating
Structuring
Ensuring equity
Aligning
teaching and
assessment
Activities
Degree of
student agency
Strategies
Considering
alternatives
Promoting
student
engagement
Degree of
difficulty
“I think a lot of us
have good intentions,
we just don't have
the time” – science
lecturer
Influences
Environment
Time
Resources
Impetus
Save resources
“I was redeveloping a unit, it had
already had a particular format of
assessment. I elected to run with
that rather than to go though the
processes of trying to alter it
… I wasn't gonna [jump
through] those other
hoops.” – humanities lecturer
Influences
Environment
Committees
and
paperwork
Institutional
requirements
Activities
Strategies
“I don't think an
assessment should be
painful for the students
or painful for the staff
that assess it” – science
lecturer
Beliefs about
assessment
Educator
Influences
Beliefs
“technology becomes really critical where
assessment is concerned. If
you set
something up and it doesn't
work, they don't trust you.
Getting them on board again is a killer …
students can be very hostile
to you making mistakes.
They're not very forgiving” – arts lecturer
Educator
Personal
characteristics
Influences
Technology
Institutional
culture
Environment
External factors
Institutional
requirements
Resources
Improving assessment
Requested supports Our analysis adds
• Exemplars
• Understanding of
freedom to move
• Time, money,
sessionals
• Someone to help
• Involvement from
senior academics
More information
http://assessmentdecisions.org
A short paper:
This research is supported
by an Australian government
Office for Learning and
Teaching grant titled
“Improving assessment:
understanding educational
decision-making in practice”
(ID12-2254)
sclr.li/19
References
• Carless, D. (2009). Trust, distrust and their impact on assessment
reform. Assessment & Evaluation in Higher Education, 34(1), 79-89.
doi: 10.1080/02602930801895786
• Hattie, J., The Black Box of Tertiary Assessment: An Impending
Revolution, in Tertiary Assessment & Higher Education Student
Outcomes: Policy, Practice & Research, L.H. Meyer, et al., Editors.
2009, Ako Aotearoa: Wellington, New Zealand.
• Offerdahl, E. G., & Tomanek, D. (2011). Changes in instructors'
assessment thinking related to experimentation with new
strategies. Assessment & Evaluation in Higher Education, 36(7), 781795. doi: 10.1080/02602938.2010.488794
• Price, M., Carroll, J., O'Donovan, B., & Rust, C. (2011). If I was going
there I wouldn't start from here: a critical commentary on current
assessment practice. Assessment & Evaluation in Higher Education,
36(4), 479-492. doi: 10.1080/02602930903512883
Extra slides
Research and ideas from the literature
• “there's some interesting papers that look at
the use of wikis and peer learning and those
sorts of things” – science lecturer (earlycareer)
Research and ideas from the literature
• “That was really quite confronting to me and
reading education literature, and I'd still find it
difficult because it's completely different to
my discipline … I just find it really, really
difficult. Because I’m going, "My god, that's
just their feelings."” – science lecturer (latecareer)
Time and workload
• “I think a lot of us have good intentions, we
just don't have the time” – science lecturer
• “But the main thing is that, it has to be
feasible from a resource point of view as well.
And if at a resource point of view, as well, as
that educational perspective as well.” – health
professions lecturer
Being strategic
• one of the other things that we learned as
well was not to put too much data into the
FEC documents, which I remember in the
early days, again, we had 1500-word essay on
topic X and then afterwards when you go
'that's crazy' we need to change it, we've got
to go back to FEC and that load of paperwork
and things. So, we do... You know, we are
much more general about what we're putting
into the FEC documents
Policy and flexibility
• “I was redeveloping a unit, it had already had a
particular format of assessment. I elected to run with
that rather than to go though the processes of trying to
alter it … I wasn't gonna [jump through] those other
hoops.” – humanities lecturer
• “there's supposed to be between 10 and 20% HDs and
no fewer... No more than 5% fails, or 10%, I think … or
more than probably 10% fails. I don't know would you
call that a bell curve or what, but it is a prescribed
range of results that you should have.” – humanities
lecturer
Policy and flexibility
• “In regards to formal channels of approval, like
education committees and the like, turns out that the,
what the paperwork that currently exists with the
education committees, is actually very, extremely nonspecific.” - health professions lecturer
• “But I don't feel that the committee structures are
really designed to be able to give... feedback or you
know contribute, it's really more tick-cross. So in some
of these cases, if what I was doing wasn't a change to
the paperwork they already had, then we just carried
on.” – health professions lecturer
Teamwork (or not)
• “The process of how this was developed was kind
of the most amazing process, because basically,
<colleague> said, "Would you develop the unit,
write a unit guide", and I did, with no help at all. I
just thought ‘what would be needed?’, and I
invented it…
• …the process has been a bit out of necessity, less
consultative than what it needs to be. I mean, we
had to get a <topic> course up and running really
quickly.” – humanities lecturer
Teamwork (or not)
• “Yeah, I had to learn a fair bit, with that, which
I learned off colleagues. So, again, yeah, that
the sort of the theoretical background to
teaching. Which sort of was provided to me.
Through exposure to colleagues. And, that
made a big change. That helped me do
assessments better.” – health professions
lecturer
Technology
• technology becomes really critical where
assessment is concerned. If you set
something up and it doesn't work, they don't
trust you. Getting them on board again is a
killer … students can be very hostile to you
making mistakes. They're not very forgiving –
arts lecturer