Jean Hughes
Office of the Vice President for Learning
Innovation,
DCU
Overview
DCU Approach
 Framework Implementation Network
 Learning Outcomes
 General – Assessment of Learning
Outcomes
 Common Questions

DCU




Already Modularised and Semesterised (1997)
Module Descriptors use ‘Learning Outcomes’
European Credit Transfer System (ECTS) well established
But


Mainly administrative exercise
Academic Framework for Innovation (AFI) – for Deep
Curriculum Reform (3 year project)




Bologna Compliance
Demonstrable Alignment with NFQ
Use of Learning Outcomes – for real change in assessment
practice
Flexibility – Designed in and through new Marks &
Standards
Framework Implementation
Network


Created by the IUA and the NQAI
Membership




7 Universities
NUI Colleges
Purpose – to collectively address
outstanding issues relating to Bologna and
the National Framework of Qualifications
Three separate Working Groups



Award Titling
Disciplinary Learning Outcomes
Assessment of Learning Outcomes
Key Issues
Ensuring that Assessment, Teaching
and Learning are aligned
 Moving from assessing content to
assessing outcomes
 Ensuring all outcomes are assessed
and all assessment instruments
actually assess outcomes
 Avoiding over-assessment

Learning Outcomes

Clearly identify what a learner can demonstrate as a result of
successfully completing a part of a learning programme

They describe student attainment which is demonstrable and
assessable.
Benefits of Learning
Outcomes

Guide students in their learning - what is
expected of them - in turn helping them to
succeed in their studies.

Help lecturers to focus on exactly what they
want students to achieve in terms of both
knowledge and skills.

Provide a useful guide to inform employers
about the general knowledge and
understanding that a graduate will possess.
Challenges

Traditional view

inputs = quality of learning
outcomes
Moving from content (Teaching)
focus to Outcome (Learning)
focus
 Aligning teaching, learning and
assessment

Award and Module
Outcomes




NFQ describes generic, high level
outcomes for awards at each level under 3
strands and 8 substrands
How do we demonstrate achievement of
award outcomes?
We don’t assess at award level we assess
at module level
Need to ensure that module learning
outcomes collectively deliver the award
outcomes
Alignment of Programme and
Module Outcomes
PO1
PO2
PO3
PO4
Mod2
Mod1
Mod6
Mod5
Mod4
Mod3
Mod7
Mod8
PO5
PO6
Mod4
Mod3
Mod5
Mod11
Mod9
PO – Programme Outcome
Mod - Module
Mod2
Mod10
Assessing Learning
Outcomes
Constructive Alignment
is the underpinning concept behind the current requirements for
programme specification, declarations of intended learning
outcomes and assessment criteria, and the use of criterion based
assessment.
(Biggs, 1999)
Example – Project
Management
Learning Outcome
(Mismatched)
Learning Outcome
(Matched)
Tasks
Students will understand how to
plan a multi-disciplinary project
Students will demonstrate ability to
plan a multi-disciplinary project
Assignment 1
Students will complete an in-class
test on Project Management
Students will devise an appropriate
project plan from a
scenario/business case study
Assignment 2
Students will create a Project Plan
using MS Project during labs
Students will create a Project Plan in
MS Project from the plan in
Assignment 1. They will be
demonstrate appropriate use of
Base Lines, Serial and Parallel Tasks,
Variances, Resource Allocation and
Critical Path, through an in-lab
demonstration.
1 Question on final exam
Case Study exam question requiring
analysis of a Business Problem,
design of appropriate project plan
and discussion of variety of project
planning issues.
Final exam
Student Perspective

From our students’ point of view,
assessment always defines the actual
curriculum (Ramsden, 1992)

Biggs (2003)
Teacher Perspectives:
Objectives
Learning Outcomes
Teaching Activities
Assessment
Alignment
Assessment
Learning Activities
Student Perspectives
Outcomes
If curriculum is reflected in the assessment, the teaching activities and the learner activities are
directed towards the same goal – in preparing for the assessment students will be learning the
curriculum
Assessment – Impact
(Boud, 1995)
“There is probably more bad practice and ignorance of
significant issues in the area of assessment than in
any other aspect of higher education”.
“The effects of bad practice are far more potent than
they are for any aspect of teaching. Students can,
with difficulty, escape from the effects of poor
teaching, they cannot (by definition if they want to
graduate) escape the effects of poor assessment”.
Assessment - Terminology

Learning Outcomes



Assessment Criteria


What a student should be able to demonstrate
(May imply the assessment criteria)
The basis on which a judgement of the adequacy of the
student work is made
Assessment Methods



The task(s) undertaken by the student – writing an essay,
answering an exam question etc- that is subject to
assessment.
Many assessment methods may be appropriate to the same
outcomes – e.g. Written, Oral, Visual, Project etc
Must be valid and reliable – Fit for purpose
Over-Assessment




Typical with semesterisation and modularisation
High risk with learning outcomes
 Tend to be quite granular and have several
independent outcomes
To avoid
 Try combining the assessment of more than one LO
 LOs only need to be assessed once
 Use assessments which inform/follow on from each
other
Check what you are assessing
Checking Assessment
Tasks
Assignment 1
Assignment 2
LO1 LO2 LO3 LO4 LO5 LO6





Assignment 3
Final exam


Issues:
What does assignment 3 assess?
How is Learning Outcome 2 assessed?
Are Learning Outcomes 1 and 4 over-assessed?

Refining Assessment
Perhaps design a new assessment 3 to re-balance?
Tasks
Assignment 1
Assignment 2
LO1 LO2 LO3 LO4 LO5 LO6




Assignment 3

Final exam





Common Questions
Must all outcomes be
assessed?

Yes, but

Not independently

Look at how you write the learning outcomes
•
Students are expected to be able to
1.
2.
3.
•
Students are expected to be able to
1.
–
–
Write a business report
Present orally
Design a visual presentation
versus
Communicate effectively using a combination of written reports,
oral presentations and visual media
This reduces the number of independent outcomes and also
recognises the combination of communication mechanisms
BUT

For certain disciplines, for example languages, you may
need these to be independently demonstrated
Example


In a Business Module (1 outcome)
 Students are expected to be able to communicate
effectively in a business context using appropriate
communication tools including Written, Oral and Visual
 The outcome is communication, the tools are some means
of demonstrating that outcome
In a languages Module (4 outcomes)
 Students are expected to be able to demonstrate written
proficiency in French
 Students are expected to be able to translate a piece of
French text
 Students are expected to be able to comprehend a piece of
French text
 Students are expected to be able to deliver an oral
presentation in French
Must all outcomes be
passed?


In theory, yes, but…
Threshold Achievement

Implies that ALL learning outcomes must be
passed in order to pass a module
• Very strict interpretation of LO approach
• Leads to reductionist or ‘Tick Box’ approaches

Typical Achievement

Learning outcomes are written (and
assessed) with the ‘typical’ or ‘’average’
student in mind
• Grading criteria will allocate marks to degree of
achievement
But, we don’t currently assess
everything….




Exams often require only 3 out of 5 questions
Therefore we don’t assess everything
But
 Generally this refers to content
 Outcomes should be about demonstrating
knowledge/understanding aided by knowledge of
content, not content in its own right
 Assessment criteria and methods should satisfy you
that students can demonstrate achievement of the
outcomes
Worst-case scenario – no worse than current situation!
What if some outcomes are
failed?

Depends on the criticality of the outcome




If particular outcomes must be passed then
passing other outcomes should not be able
to compensate
Need to look at ‘Threshold’ and ‘Typical’
Outcomes
Need to write outcomes carefully
Need to ensure that supplemental/repeat
assessment is also valid
What about repeats?

Did they actually fail?
 “Students will present orally on 20th century politics”
• If they don’t attend they fail….. Is this what you meant?
or

“Students will effectively communicate on 21th
century politics”
• You may prioritise Oral assessment and allocate more
marks, but this allows more choice of methods

If a student has to repeat the supplemental
assessment must be valid if different from the original
 Look at how you write the outcomes
Is it an Outcome, An
Assessment or an Efficiency?


Often mix up pedagogical (or efficiency) approaches with learning
outcomes
Example Group Working

Learning Outcome
• ‘Students will demonstrate effective group working skills’
• Group working needs to be ‘taught’ and assessed
• Can’t assume that putting people in groups will automatically lead to
effective group working!
• How does an individual student repeat?
• ‘Students will be able to evaluate group dynamics and effective group
working’
•

Would this suffice but give an alternative assessment approach for repeat
students?
Pedagogical Approach
• Group working is used to facilitate learning of a particular outcome
• Not assessing the group working – assessing the actual outcome

Efficiency
• Often used to reduce the volume of marking
• Need to be careful that it is pedagogically sound
• Not assessing group working
Making the Tacit Explicit


Some disciplines rely on expertise of the individual
lecturer
 ‘I know a good piece of work when I see it’
 ‘You can’t possibly express Creativity, Performance,
Medieval English….. Using learning outcomes
NOT about subjectivity – expertise, tacit knowledge etc
 How do students know what to aim for?
 How do novice lecturers know about standards?
 How can we demonstrate consistency, transparency
etc?
Example
First, rewrite the outcome:
“By the end of this module students will understand the importance of postrenaissance European art in the context of art history”
Redesigned:
By the end of this module students will be able to:

Evaluate and criticise post-renaissance European art within its historical
context

Date and identify key works of this specific period

Demonstrate an understanding of the particular art-historical debates
that have taken place about this period

Show ability to argue why this particular place and time raise interesting
issues about the discipline as a whole
University of Sussex
Then, use a rubric






Also called grading criteria or scoring guide
A set of guidelines for marking or scoring
Can be a list, chart, or guidelines
They state all dimensions being assessed
They contain a scale
They assist the marker on rating
assessments on the scale
Rubrics - 3 Types



Checklists
 Indicating list of criteria
Rating Scales
 Checklist with rating scale added
 Show degrees by which students meet criteria
Holistic Scoring Guide
 Used where assignments can vary i.e. projects
 Narrative descriptions of characteristics of different
grades of work
 Not list of criteria
Rubric Design
Achievement Levels
Criteria
Objective 1
Excellent
Good
Accepted
Minor
revision
Objective 2
Expert
Objective 3
6-5
Needs Work
Not
acceptable
Major
revision
Rejected
Advanced Intermediate Novice
4-3
2-1
0
Simple Checklist Rubric
2.6.2 A Checklist Rubric for a Web Site (Suskie, 2004)










The purpose of the site is obvious.
The site’s structure is clear and intuitive.
Titles are meaningful.
Each page loads quickly.
The text is easy to read.
Graphics and multimedia help convey the site’s main points.
The design is clean, uncluttered and engaging.
Spelling, punctuation and grammar are correct.
Contact information for the author or webmaster is given.
The date each page was last update is given.
1. Write articulate, persuasive and grammatically
correct business materials.
2. Use critical, flexible and creative thinking to
generate sound conclusions, ideas and solutions to
problems.
3. Use software and networking services to obtain,
manage and share information.
4. Apply understanding of domestic and international
diversity concepts and issues to business
situations.
5. Recognise ethical challenges and reach ethical
business decisions.
Insufficient
Information to
Evaluate
Inadequate
Satisfactory
Rubric is
for a written
Business
Assignment
Understanding
Learning Outcome




















This assignment asked
students to take a
position on a
debateable issue
regarding interpretation
of literature they had
studied. The lecturer
has identified 3 major
criteria – Position,
Support and
Acknowledgement of
Alternative Points of
View and has weighted
them different levels of
attainment of each
criterion. Opposite is the
detailed rubric for the
Position criterion.
Criterion: Position taken on a debateable issue
Description
Student takes a defensible position on the issue posed and states the position
clearly. Position does not merely state the obvious or parrot one of the
readings, but shows a creative mind at work.
Student takes a defensible position on the issue posed and states the position
clearly. Position may be somewhat obvious or closely parallel to one of the
readings.
Student takes a defensible position on the issue posed and states the position
clearly but the position may state the obvious or simply paraphrase one of the
readings.
Weighting
5
4
3
Student takes a defensible position on the issue posed, but the statement is
ambiguous, carelessly stated or must be inferred.
2
Student does not clearly state a defensible position, or position is not
defensible or is irrelevant to the question posed.
1
In Summary




Need to write learning outcomes so that
they can be demonstrated and assessed
Need to identify assessment criteria and
appropriate assessment methods
Need to be sure exactly what it is students
need to demonstrate
Making the tacit explicit – Clearly written
outcomes and the use of rubrics to
articulate achievement
Questions?