Learning about Masters Level
Assessment:
the Assimilate NTFS project
http://assimilate.teams.leedsmet.ac.uk/
Sally Brown, Janice Priestley, Phil
Race, Ruth Pickford and Tim Deignan
PASS conference
24th - 55th July 2012
Bradford
Assimilate is a 3-year NTFS
funded project



The Assimilate team have been exploring innovative
assessment at Masters level using research funding
from the National Teaching Fellowship scheme.
Recognising that limited prior research had been
undertaken in this area, the project was designed to
review the range of assessment methods used to
assess at this level, particularly exploring authentic
assessment.
Interviews were undertaken in the UK and
internationally by students and team members to
elicit information about diverse approaches and to
produce case studies showcasing innovations.
The project was designed to:




Survey the range of assessment methods and
approaches used to assess at Masters level in
diverse institutions, particularly in professional
subject areas and in a variety of disciplines;
Investigate the ways in which Masters level students
receive formative feedback;
Provide a compendium of diverse approaches to
assessing at this level;
Develop recommendations for good practice
regarding assessment and formative feedback for
students working towards Masters level awards.
Changes en route




We originally planned to use 2nd year
Journalism students as interviewers for our
research, but this proved impractical;
We then moved to using our (changing)
project team members including me when I
(semi) retired;
Using data from 45 interviews and from other
people who have been working with the
project we have produced more than 34 case
studies illustrating diverse M-level
assessment, including negotiated course
work;
We changed our data analysis approach.
Analysing our data




Tim Deignan has been using Activity Theory and Q
Methodology to help us make sense of the case
study data and to conduct a follow-up study.
His initial research study used Activity Theory to
investigate practitioners’ experiences of introducing
innovative assessment methods at Masters level.
He then designed a Q-study using 48 statements
which were rank-ordered by 39 participants .
Using statistical analysis of these data he has
interpreted five distinct factors, or viewpoints,
relating to Masters level assessment.
User-friendly: a Q-sort underway
Acknowledgement: My thanks to Dr. Louise Bryant of University of Leeds for sharing this graphic.
Stages in a Q-study





Identifying and sampling the concourse
Developing a set of statements that is
representative of the concourse
Selecting participants for a diversity of views
on the issues
Q-sorting and post-sort interviews
Pattern analysis - data reduction and
interpretation
Viewpoints 1, 2 and 3

1: The innovative assessment and accreditation of
learning for complex real life / workplace
applications requires assessment training for both
staff and students.

2: Standards and consistency can not be guaranteed
by any means, but flexible assessment criteria and
innovative assessment methods have their uses.

3: Introducing innovative assessment methods can
be powerful but requires new perspectives on
learning with institutional support and
encouragement for successful wholesale change.
Viewpoints 4 and 5

4: Clear guidance to students in the form of high
quality assessment criteria and timely tutor
assessment feedback can help students to develop
the skills that they and also employers want.

5: Improving assessment methods does not
necessarily require a paradigm shift in thinking, but
stakeholder consultation is important as benefits are
not guaranteed and one size does not fit all.
Dissemination events




Our own Assimilate conference July 2011
and our planned final international event
September 6th 2012
Presentations/workshops at Newman
University College, Aberdeen university,
Central Queensland University, SRHE,
Institute of Education, Queens University
Belfast, York St John and Cranfield
University;
Conference presentations at Cumbria
University T&L event, NTFS symposium,
York, SEDA 2012 Chester, HEA York
Posters accepted at ISL Lund and (probably)
SRHE Newport.
Publications plan
1. Project reports for HEA
2. Articles in refereed journals:
a. ‘What are the differences between Masters and UG
level assessment?’ IETI, accepted
b. ‘Innovations in Masters level teaching’ (final draft)
c. ‘Making sense of educators experiences and views
of assessment innovation at M level’ (drafted & being
redrafted)
d. Chapter in book on M level teaching & assessment
e. Further technical articles for Q journals
3. Compendium of resources and good practice advice
for website and for delivery at our conference.
What we have achieved and
not achieved




We had little success using students as
researchers;
We have survived major changes in our
project team;
We have successfully populated a useful
(and well-used) website with case studies
and overviews;
We have explored M-level assessment widely
in the UK as well as in Spain, the
Netherlands, Denmark, Ireland, Singapore,
new Zealand and Australia.
Further achievements





We have met all project milestones to date and
to budget;
We are adding significantly to understanding
of M-level assessment, particularly through
our analysis of data to identify viewpoints;
We have produced and are still producing a
range of useful and relevant project outputs;
We have successfully networked in the UK and
internationally;
Team members have themselves benefited
significantly from the project in terms of CPD;
And where next?




We will continue to publish our outcomes
and disseminate our findings;
We will share our outputs through our
networks;
Our website will continue to show case the
project for the foreseeable future;
We will seek further opportunities for
research, potentially exploring the
differences between M level and PhD
assessment.
Selected references and further
reading
Barry, J. & Proops, J. (1999). Seeking sustainability discourses with Q
methodology. Ecological Economics, 28(3),337-345 Casey, J. (2002) On-line
assessment in a masters-level policy subject: participation in an on-line
forum as part of assessment. Centre for the study of higher education,
Charles Sturt University, Australia.
Dunn, S. and Singh, K. A. (2009) Analysis of M-level modules in
interdisciplinary nanotechnology education. Nanotechnology Centre,
Department of Materials, School of Applied Sciences, Cranfield University.
Engeström, Y. (2010). Studies of expansive learning: Foundations, findings
and future challenges. Educational Research Review, (5):1-24
Fry, H., Pearce, R. and Bright, H. (2007) Re-working resource-based learning - a
case study from a masters programme. Innovations in Education and
Teaching International, 44(1), pp.79-91.
Geographical Association. (no date) GTIP Think Piece - Writing at
Masters Level. Available online:
http://www.geography.org.uk/gtip/thinkpieces/writingatmasterslevel/
Haworth, A., Perks, P. and Tikly, C. (no date) Developments with
Mathematics M-Level PGCE Provision and Assessment. University
of Manchester, University of Birmingham, University ofSussex.
References (contd.)
Institute of Education (2006) Masters level criteria for Geography PGCE
http://www.geography.org.uk/download/GA_PRGTIPBrooksMLevelCrit
eria.pdf Accessed March 2012
Lord, D. (2008) Learning to Teach a Specialist Subject: Using New
Technologies and Achieving Masters Level Criteria. In: MOTIVATE
conference 2008, 11 - 12th November 2008, Dunaujvaros, Budapest.
(Unpublished) This version is available at
http://eprints.hud.ac.uk/10892/ Accessed march 2012
NZQA (2007) http://www.nzqa.govt.nz/assets/Studying-in-NZ/NewZealand-Qualification-Framework/theregister-booklet.pdf (accessed
March 2012
M level PGCE. (2007) ESCalate ITEM level PGCE seminar at the
University of Gloucestershire on January 9th 2007.
QAA (2010) Masters Degree Characteristics
http://www.qaa.ac.uk/academicinfrastructure/benchmark/masters/Masters
DegreeCharacteristics.pdf
Seymour, D. (2005) Learning Outcomes and Assessment: developing
assessment criteria for Masters-level dissertations. Brookes eJournal
of Learning and Teaching , 1(2).
References (contd.)
Van Eeten, Michel J.G. (2001) Recasting Intractable Policy Issues: The
Wider Implications of the Netherlands Civil Aviation Controversy,
Journal of Policy Analysis and Management, 20(3):391-414
Wharton, S. (2003) Defining appropriate criteria for the assessment of
master's level TESOLAssignments. Assessment & Evaluation in
Higher Education, 28(6), pp.649-664.