A Program of Assessment for a
Program of Study
Implementing an Integrated
System for Programmatic
Assessment of the MBBS
Program
Dr Tracey Papinczak
Quality assurance in education
“the totality of systems, resources and
information devoted to maintaining
and improving the quality and
standards of teaching, scholarship
and research, and of students
learning experience”
Quality Assurance Agency in UK Higher Education
Quality assurance system
Does the assessment
– Align with best practice?
Have the following been undertaken?
– Self-reflection
– Examination of student feedback
– Peer review
– External review
1. Best practice in assessment:
Individual instruments
EBP
•
•
•
•
•
•
•
•
Reliable
Valid
Objective
Acceptable
Constructively aligned
Equitable and fair
Cost-effective
Time-efficient
1. Principles: Best practice in assessment
programs
• Clarify what are expected standards (criteria)
• Facilitate deep learning
• Deliver high quality feedback
• Positively motivate students
• Support development of learning communities
• Involve students in decision making
• Deliver authentic assessment
• Encourage reflection & self-assessment
• Be paced to support continuous learning
• Have strong validity and reliability for all individual assessment pieces
2. Self-reflection
Taking
time
Reflecting
in-action
and onaction
What is
working?
What is
not
working?
How can
we
improve?
Where
can we
access
support?
3. Student feedback
Student feedback to the Evaluation Unit
–
–
–
–
Concerns about subjectivity of consultant assessment years 3 & 4
Lack of time to complete some written examinations
Excessive assessment in some rotations
In yrs 1 & 2, too little assessment
Course evaluations administered by TEDI
– 2010 SET-C
• greater feedback on assessment needs to be provided
– 2008 - 2009 iCEVAL
• LO to assessment match needs greater clarity
• Assessment requirements and marking criteria need to be provided at
the start of the course
• Assessment needs to be spread across the semester not concentrated
at the end
4. Peer review
4. Peer review
Examples:
• Peer review of examination papers in years 1
and 2
• Peer review of PBL tutors in year 1
• Peer review of lecturers in Discipline of
General Practice
5. External Review
Australian Medical Council Review
Lack of centralised control over and management
of assessment
Variable quality of assessment
Redundancies and gaps
Lack of examiner training
Planning for quality assurance
Quality Assurance of the Program of Assessment
Quality of
each piece of
assessment
Purposeful plan
Mapped to
aims and goals
Gaps
identified and
addressed
Redundancies
removed
EBP in
assess
ment
Best
choice
for
KSA
Threshold learning outcomes
ALTC – Learning & Teaching Academic
Standards project
Health, Medicine and Vet Science
Six TL outcomes revised through stakeholder
consultation
Implications for assessment within all
programs once TEQSA is underway (new
Federal Government auditing agency within
DEEWR)
Upon completion of their program of study,
health graduates at professional entry level
will be able to:
• Demonstrate professional behaviours
• Assess individual and population health status, and where
necessary, formulate and implement management plans in
consultation with patients/clients/carers
• Promote and optimise the health and welfare of patients/clients
and populations
• Retrieve, critically evaluate, and apply evidence in the
performance of health care activities
• Deliver safe and effective health care in collaboration with other
health care professionals
• Reflect on current skills, knowledge and attitudes, and plan
ongoing professional development activities.
Assessment in the MBBS Program
Threshold
learning
outcomes
(once TEQSA
established)
UQ
Graduate
attributes
Aims, goals
and
learning
outcomes
Graduate attributes
In-depth knowledge & skills in the field of study
• A comprehensive and well-founded knowledge in the field of study.
• An understanding of how other disciplines relate to the field of study.
• An international perspective on the field of study.
Effective Communication
• The ability to collect, analyse and organise information and ideas and to convey
those ideas clearly and fluently, in both written and spoken forms.
• The ability to interact effectively with others in order to work towards a
common outcome.
• The ability to select and use the appropriate level, style and means of
communication.
• The ability to engage effectively and appropriately with information and
communication technologies.
Graduate attributes
Independence and Creativity
• The ability to work and learn independently.
• The ability to generate ideas and adapt innovatively to changing environments.
• The ability to identify problems, create solutions, innovate and improve
current practices
Critical Judgement
• The ability to define and analyse problems
• The ability to apply critical reasoning to issues through independent thought
and informed judgement
• The ability to evaluate opinions, make decisions and to reflect critically on the
justifications for decisions
Ethical and Social Understanding
• An understanding of social and civic responsibility
• An appreciation of the philosophical and social contexts of a discipline
• A knowledge and respect of ethics and ethical standards in relation to a major
area of study
• A knowledge of other cultures and times and an appreciation of cultural
diversity.
3
7
35
Programmatic assessment
•
•
•
•
•
•
•
•
•
Focussed on the whole of a program of study
Collaborative & consultative
Charts students progress over time
Involves multiple perspectives
Measures knowledge, skills & affective domains
Has a strong workplace focus
Based on principles of best practice in education
Measures previously underserved areas of assessment
Avoids overlap & over-assessment of commonly measured
KSAs
From the perspective that a program of study is more than the sum of
each individual course, a program of assessment must be more than
each individual measure.
The whole is
greater than
the sum of
the parts
Research aim
To develop & to implement
a comprehensive, integrated assessment
program
which explicitly maps
to the three aims, seven goals & 35 learning
outcomes of the MBBS Program
and the six threshold learning outcomes of
the TEQSA
Educational approach
Dijkstra, Van der Vleuten & Schuwirth (2009)
– Six dimension framework
– Guides programmatic assessment
– Holistic focus
– Explicit mapping to program goals BUT
– Establishing a set of goals for an assessment program in
medical education
Method
Dijkstra dimension
Dimension elements
Details of Implementation
Goals
Establishing a clear goal for the
assessment program
Mapping of existing assessment items to aims, goals & learning
outcomes;
Identification of gaps, redundancies & assessment practices &
instruments inconsistent with best practice;
Consultation with stakeholders -assessment across all four years of
the Program, potential barriers & other issues;
Establishment of a shared goal for the assessment program
Supporting the
program
Technical support for quality
assessment & involvement of
stakeholders
Professional development and support;
Continued stakeholder input
Documenting the
program
Assessment rules, blueprinting &
planning;
Plotting the assessment processes
for clarity & accountability
Literature review to identify best evidence for assessment in higher
education & medical education;
Analysis of strategic plans & priorities at School, Faculty &
University level;
Development of a holistic plan for assessment
Program in action
Collecting data, combining the
data, reporting & decision-making
Implementation of the integrated assessment program
Improving the program
Constant monitoring & application
of relevant research
Evaluation & action research
Accounting for the
program
Internal & external review;
Accountability to stakeholders
UQ & AMC Review;
Stakeholder feedback
Likely outcomes
• Creation of comprehensive map: learning goals, aims &
outcomes matched to current assessment processes
• Establishment of holistic assessment program: sampling
knowledge, skills & other key aspects of competence
both summatively & formatively
• Development of spectrum of complementary
assessment strategies: sampling student achievement in
an iterative manner but with increasing sophistication
• Synthesis of well-informed, logical & highly defensible
plan of assessment.
• Enhanced collaboration & understanding between all
stakeholders
Conclusion
The opportunity exists to
produce, over the next three
years, an innovative program
of assessment that best
serves the needs of all
stakeholders & sets an
example of the high quality
which can be achieved by
one of Australia’s premier
medical schools.
References
Dijkstra, J., Van der Vleuten, C. & Schuwirth, L. (2009). A new framework
for designing programmes of assessment. Advances in Health Sciences
Education, Advance online publication. Doi: 10.1007/s10459-009-9205-z.