Assessment of Learning
Outcomes in Engineering
- an International Perspective
Mona Dahms
mona@plan.aau.dk
UCPBL, Aalborg University
Presentation Outline
1. Curriculum Development and Assessment
2. An International Perspective
3. The Situation in Denmark
aloe conference 2008
2
Definitions
• Learning outcomes: ”sets of competences, expressing
what the student will know, understand or be able to do
after completion of a process of learning” (ECTS Guide
2005); describes the end product of learning
• Assessment: Judgement of students’ performance, i.e.
judgement of whether students have achieved the
learning outcomes
• Evaluation: Judgement of quality of study programs and
activities
aloe conference 2008
3
Curriculum Development (CD)
– a sequential process
Competence
profile of
graduate
Design of study
program
aloe conference 2008
Design of
1st sem.
Study
activity 1
Design of
2nd sem.
Study
activity 2
Design of
3rd sem.
Study
activity 3
Design of
4th sem.
Study
activity 4
Design of
Nth sem.
Study
activity 5
4
Competence
profile of
graduate
CD – a top-down process
Design of study
program
Design of
semester
program
This is where
assessment is
carried out!!!
Design of study
activity
Evaluation of
graduate
competences
Evaluation of
study program
Evaluation of
semester
program
Evaluation of
study activity
Teaching, learning
and assessment
5
The Cowan
Curriculum
Development
model
= learning
outcomes
Cowan, 2004
aloe conference 2008
6
Constructive alignment
Learning outcomes
“Alignment
is about trust in the relationship
between
Learning/teaching
Assessment
student and teacher” (Derick)
aloe conference 2008
7
What is assessment?
Formative
Summative
• setting criteria
• collecting evidence
• making judgement
and always
• providing feed-forward for
further improvement
• setting criteria (LOs)
• collecting evidence
• making judgement
and possibly
• providing feed-back for
improvement
Example: The chef tastes the
 Example: The customer tastes
food to be served and .........
the food served and ......
8
Assessment
Important questions to consider are:
–
–
–
–
–
WHO sets criteria?
WHEN and WHERE is evidence prepared?
HOW is evidence presented?
WHO makes the judgement?
WHO provides feed-forward/feed-back?
aloe conference 2008
9
Changing competences…
…necessitates changes in assessment
creating
evaluating
Higher level
competences
analysing
applying
Learning
outcomes
assessed
Declared
understanding
Lower
level
learning
competences
outcomes remembering
aloe conference 2008
10
Traditional assessment
Pass
Good understanding
Failed examination:
0%
Fail
Understanding
Poor understanding
Failed examination:
25 %
Jakobsen and
Rump, 1998
Fail
Good understanding
Passed examination:
25 %
Poor understanding
Passed examination:
50 % !!!!
aloe conference 2008
Pass
Exam
11
The International Perspective
• Malaysia (2006?) and
• South Africa (1997) have both introduced
• Outcomes Based Education (OBE) at all levels
aloe conference 2008
12
Malaysian Qualification Framework (MQF)
Learning Outcomes
1. knowledge of discipline areas
2. practical skills
3. social skills and responsibilities
4. value, attitude and professionalism
5. communication, leadership and team skills
6. problem solving and scientific skills
7. information management and life-long learning skills
8. managerial and entrepreneurial skills
“The challenge to us is in drafting assignments, projects
and exams as means of evaluating the MQF learning
outcomes.”
aloe conference 2008
13
South African Critical Cross-Field Learning Outcomes
1. Identify and solve problems and make decisions using critical and
creative thinking
2. Work effectively with others as members of a team, group, organisation
and community
3. Organise and manage themselves and their activities responsibly and
effectively
4. Collect, analyse, organise and critically evaluate information
5. Communicate effectively using visual, symbolic, and/or language skills
in various modes
6. Use science and technology effectively and critically showing
responsibility towards the environments and the health of others
7. Demonstrate an understanding of the world as a set of related systems
by recognising that problem-solving contexts do not exist in isolation
aloe conference 2008
14
Achievements in SA ?
• A study of critical thinking abilities of 1st year
education students documented deficient critical
thinking abilities
• Possible reasons: Teaching practice;
educational change; socio-cultural environment;
language ability (Lombard & Grosser 2008)
aloe conference 2008
15
MoHE - Denmark
Uni 1
Aalborg University
HUM FAC
Study Brd. Civil Eng.
Uni 2
TECH FAC
Study Brd. Elec. Eng.
Uni 3
SOC FAC
Study Brd. Mec. Eng,
Study Prg. 1 - Team
Study Prg. 2 -Team
Study prg. 3 - Team
Course 1 - Lecturer
Project - Facilitator
Course 2 - Lecturer
aloe conference 2008
Other study brds
Study Prg. 4 - Team
16
Skills
To what extent do you feel that the following skills were evaluated
at the exam? The students’ ability to a high or some degree:
Students: group
exam 2006 (N=455)
Quickly answer the
asked questions?
Staff: group exam
2006 (N=159)
External examiners:
group exam 2006
(N=130)
Students: individual
exam 2007 (597)
Remember
concepts and
definitions?
Relate different
concepts to each
other?
Staff: individual
exam 2007 (N=113)
0%
20%
Kolmos and Holgaard 2007
40%
60%
aloe conference 2008
80%
100%
External examiners:
individual exam
2007 (N=63)
17
Skills
To what extent do you feel that the following skills were evaluated at
the exam? The students’ ability to a high or some degree:
Students: group
exam 2006
(N=455)
Argue for
methodological
choices?
Staff: group exam
2006 (N=159)
Assume a
theoretical
overview?
External
examiners: group
exam 2006
(N=130)
Students:
individual exam
2007 (597)
Apply analytical
skills?
0%
Kolmos and Holgaard 2007
20%
40%
60%
aloe conference 2008
80%
100%
Staff: individual
exam 2007
(N=113) 18
External
Skills
To what extent do you feel that the following skills were evaluated
at the exam? The students’ ability to a high or some degree:
Students: group
exam 2006 (N=455)
Argue for and against a
solution to a problem?
Staff: group exam
2006 (N=159)
Transfer knowledge
gained in projects to
other situations?
External examiners:
group exam 2006
(N=130)
Students: individual
exam 2007 (597)
Communicate
knowledge through a
presentation?
Staff: individual
exam 2007 (N=113)
0%
Kolmos and Holgaard 2007
20%
40%
60%
aloe conference 2008
80%
100%
External examiners:
individual exam
2007 (N=63)
19
To what extent do you feel that the following skills were evaluated at
the exam? The students’ ability to a high or some degree:
Skills
Students: group exam
2006 (N=455)
Enter into
dialogue and
cooperation?
Staff: group exam 2006
(N=159)
Complement
and expand on
others
answers?
External examiners:
group exam 2006
(N=130)
Students: individual
exam 2007 (597)
Participate in a
teamwork?
Staff: individual exam
2007 (N=113)
0%
20%
Kolmos and Holgaard 2007
40%
60%
80%
100%
aloe conference 2008
External examiners:
individual exam 2007
(N=63)
20
Statements on team based exams
• “I dread a transition to an individual exam because I
believe that this would result in a significant decline in the
academic level. Besides I am of the opinion that the team
based exam is the type of exam which best relate to the
working environment which the students will be in later
on.” (External examiner)
• “I think it is important that the members of the group are
tested in each others’ co-presence and in a collaborative
process. It opens the possibility for partially evaluating
their team play and co-operation, ability to stand out in a
group, ability to follow up and to have an effect on a
discourse.” (External examiner)
Kolmos and Holgaard 2007
aloe conference 2008
21
Conclusion
• Individual assessment does not test complex
knowledge and collaborative competences
(Kolmos and Holgaard 2007)
aloe conference 2008
22
Thank you for your attention!
Comments and questions are
very welcome at:
mona@plan.aau.dk