Assessing and Assuring
Graduate Learning Outcomes
SUMMARY OF KEY PROJECT FINDINGS
PRESENTED BY SIMON BARRIE
SYDNEY 26TH SEPTEMBER 2013
CURRICULUM RENEWAL FOR
GRADUATE LEARNING OUTCOMES
Everybody is….
Developing 'new' statements of graduate attribute 'outcomes'
Renewing curriculum ……(to achieve such outcomes?)
Demonstrating achievements …..(compliance?)
2
WHAT’S IT ALL ABOUT? (ACCORDING TO UNESCO)
“Owing to the scope and pace of change, society has become
increasingly knowledge-based so that higher learning and
research now act as essential components of cultural, socioeconomic and environmentally sustainable development of
individuals, communities and nations. Higher education itself
is confronted therefore with formidable challenges and must
proceed to the most radical change and renewal it has ever
been required to undertake, so that our society, which is
currently undergoing a profound crisis of values, can transcend
mere economic considerations and incorporate deeper
dimensions of morality and spirituality."
3
GRADUATE ATTRIBUTES
A TRANSFORMATIVE OPPORTUNITY?
They are descriptions of the core abilities and values a university
community agrees all its graduates should develop as a result of
successfully completing their university studies (adapted from
Bowden et al 2000).
Graduate attributes are an orientating statement of education
outcomes used to inform curriculum design and engagement with
teaching and learning experiences at a university (Barrie 2009).
Graduate attributes are one way of starting a rich conversation,
across the university and beyond it, about how the university's
intentions,values and mission are embodied in the learning
experiences and outcomes of its students.
4
Transformative opportunity or managerialist trap?
5
Collaborative national project supported by ALTC / OLT with strategic priority funding
› What are the assessment tasks in a range of disciplines that generate convincing
evidence of achievement of graduate learning outcomes?
› What are the assurance process trusted by disciplines in relation to those
assessment tasks and judgements?
Situational analysis, literature review, expert reference group, institutional visits,
interviews with 48 academics 28 universities in 7 disciplines
Ten key issues papers, endnote library, submissions to government discussion
papers, research report – 6582 unique page views…and counting. Full story at:
http://www.itl.usyd.edu.au/projects/aaglo/
6
WHAT (AND WHY) ARE WE ASSESSING?
1.
Approach of assessment for learning
2.
Coherent development and assessment of program level graduate learning
outcomes requires coherent institutional and discipline statements of outcomes.
What that statement ‘is’ matters.
› Foundation skills? Translation attributes? Enabling attributes and dispositions?
› Traditional or contemporary conceptions of knowledge?
› Content knowledge plus generic skills = traditional
› Graduate attributes, capabilities, dispositions and stances = contemporary
Some of the ways we are articulating GLOs are taking us backwards to
outdated old fashioned expectations of the goals of higher education,
not forwards to the contemporary outcomes and purposes we espouse.
7
ASSESSMENT STRATEGIES
2. What we assess, and how we assess it, shapes curriculum and teaching as
much as it does learning
› Assessment prioritizes some GLO and ignores others
› Communication skills – Privileged
› Information literacy - Privileged
› Research and inquiry – (less) Privileged
› Ethical social professional understandings - Neglected
› Personal intellectual autonomy - Neglected
Consequences: ?
8
ASSESSMENT STRATEGIES
3. Features of effective assessment practices identified by respondents
› (assessment for learning)
› interconnected multi-component
› authentic, relevant, roles
› standards-based with effective communication of criteria
› involve multiple decision makers – including students
› (program level coherence)
› With effective assurance processes around the quality of these tasks and the
judgments made
9
ASSESSMENT STRATEGIES
4. Variety of assessment quality assurance strategies
› Quality of task:
Pre-implementation (policies, curriculum approval
processes for new and revised tasks)
Post- implementation (evaluation by students,
examiners, peers, curriculum audit & review)
› Quality of judgment:
Pre-judgement - (calibration)
Post-judgement – (consensus moderation)
› Confidence?
Combination of effective task and assurance was less common than it could be
Evidence of ‘assurance’ engaging staff and leading to ‘enhancement’ was rare
› Barriers?
10
ASSESSMENT STRATEGIES – FUTURE
4. Trend towards (systematic) program level approach to assessment
› Many drivers – complexity of outcomes, coherence & integration,
resources, managing regulatory accountability….. others?
› Possible changes / implications: longer, bigger, fewer subjects? Limited
task variety - cumulative/progressive? Shift from subject focus to
assessment of course outcomes (less assessment), integrative
assessment, rebalancing of assessment types from FY to graduation?
Curriculum design /sequencing on learning challenge not content?
› Barriers to making these changes?
11
BARRIERS TO CHANGE….RESOURCES
› Cost-effective pedagogies
› Those that generate lots of productive studying with modest teaching input (and can
still receive very high student feedback ratings)
› More feedback …. ?
› More teaching ……?
› More assessment ……?
› Students are an underutilized resource …… active students increase performance at
little or no cost
› Total teaching hours do not predict student performance … more teaching usually
means less time for learning
› Increasing students’ social engagement improves retention …. and performance
› Library investment in learning resources per student predicts student study hours
12
Figure 1: Different forms of Program-focused assessment
(adapted from the Bradford PAsS project 2012)
1
READ ALL ABOUT IT
HEADLINES FROM THE AAGLO ISSUES PAPERS
Paper 1: The AAGLO project and the international standards agenda
Insight: We are spending a lot of money reinventing the wheel and repeating the
same mistakes - but disciplines are coming together around ‘learning’
Paper 2: Assurance of graduate learning outcomes through external review
Insight: We have an opportunity to extend ‘peer review’ (in a useful way) to
influence curriculum and teaching at an institutional / whole degree level
Paper 3: Challenges of assessing graduate learning outcomes in work-based
contexts
Insight: The boundaries between work and education persist and work is still
underutilised as an assessment ground by universities
Paper 4: Standardised testing of graduate learning outcomes in higher
education
Insight: The illusion that standardised generic skills tests usefully define
and demonstrate outcomes causes more harm than good
Paper 5: Approaches to the assurance of assessment quality
Insight: These need to be embedded in academic practice in ways
that meaningfully engage staff and students.
14
HEADLINES FROM
THE AAGLO ISSUES PAPERS
Paper 6: Assessment policy and effective GLO assessment and assurance
Insight: Our well-intentioned policy decisions may have a limiting impact on
assessment practice
Paper 7: Characteristics of effective assessment tasks
Insight: Some things really do work!
Paper 8: eAssessment issues in effective GLO assessment and assurance
Insight: E-assessment can offer new opportunities through the design of
tasks that require Web 2.0 creative activities
Paper 9: Whole-of-programme approaches to assessment planning
Insight: Fragmented approaches to assessment provide neither a coherent
learning experience for students nor credible evidence of GLOs
Summary 10:The student perspective on assurance of outcomes
Insight: We keep leaving the students out of the conversation
15
CURRICULUM RENEWAL
› It is one thing to have the right curriculum
goals (learning outcomes) in place…..
› ……and it is better to have credible
assessment evidence of the achievement
of those goals by graduates…..
› …..but the real question remains ‘how
does a university curriculum, teaching and
learning effectively develop those
outcomes’?
16
Thank you!
Simon.Barrie@sydney.edu.au
More at: http://www.itl.usyd.edu.au/projects/aaglo/