Accreditation Workshop
- Part 2
28 February 2014
Islamabad
Er. Professor Dr. Kai Sang LOCK
Washington Accord Mentor to PEC
A seminar organized by
Pakistan Engineering Council
Part 2
Implementing Outcomes-based
Assessment & Preparing for Program
Accreditation
Understanding Outcome-Based
Assessment (OBA)
TRADITIONAL APPROACH FOR QUALITY ASSURANCE OF
ENGINEERING PROGRAMMES
• Focused on the input & process quality
• The criteria for accreditation may typically include the
following list:
–
–
–
–
–
–
–
–
–
–
Organization and governance
Financial resources
Physical resources and facilities
Faculty and staff
Student intake quality
Teaching – learning process
Co-curricular and extra-curricular activities
Student services & counseling
Research & Development
Industrial interaction
Key features of OBA
• OBA focuses more on:
– Learning, as against teaching
– Students
– Outcomes, not inputs or capacity
• OBA incorporates continuous improvement
– to systematically analyze its systems for variance
– make decisions based on facts
– consciously define the organization’s customers –
both internal and external
– actively seek input from customers
Outcomes-based system
• The emphasis is on measured “outcomes”
rather than "inputs”
• Outcomes include a range of skills and
knowledge
• Outcomes are measurable, observable
• Outcomes (skills and knowledge) are
specified, but not inputs
• Identifying appropriate and measurable
outcomes is difficult, and often controversial
Outcomes
• Stipulate knowledge, skills, attitudes &
behaviors on completion of program
• Well documented
• Defined by measurable performance
indicators
• Both direct & indirect assessment tools to
measure each outcome
Writing Intended Learning Outcomes
• Intended learning outcomes need to be
written at both programme and course levels.
• Both of them need two essential elements:
– A statement of what content are the students
expected to be able to do at the end of learning
experience;
– The levels of understanding or performance in
those content areas.
Definition of terms used
• Programme Educational Objectives
• Programme educational objectives are broad statements
that describe the career and professional
accomplishments that the programme is preparing
graduates to achieve (within 3 to 5 years after graduation)
• Student Learning Outcomes
• Student learning outcomes are narrower statements that
describe what students are expected to know and be able
to do by the time of graduation. These relate to the skills,
knowledge, and behaviours that students acquire in their
matriculation through the programme
Definition of terms used
• Assessment
• Assessment is one or more processes that identify, collect,
and prepare data to evaluate the achievement of
programme educational objectives and student learning
outcomes
• Evaluation
• Evaluation is one or more processes for interpreting the
data and evidence accumulated through assessment
practices. Evaluation determines the extent to which
programme educational objectives or student learning
outcomes are being achieved, and results in decisions and
actions to improve the programme
Accreditation Criteria of
relevance to Outcome-Based
Assessment (OBA)
Accreditation Criteria
• 11 criteria in EAB Accreditation Manual:
1)
2)
3)
4)
5)
6)
7)
8)
9)
10)
11)
Mission & Programme Educational Objectives
Student Learning Outcomes
Curriculum and Teaching/Learning Processes
Students
Faculty members
Facilities & learning environment
Institutional support & financial resources
Governance
Interaction between institution & industry
Research & development
Specific Programme criteria
Mission
• University mission?
• Faculty (school) mission & departmental
mission statement?
• Alignment with mission of institution?
• Published and known to stake-holders?
Programme Education Objectives
(PEO)
• Broad statements that describe what graduates are
expected to attain within a few years after graduation
• Align with university’s & engineering school’s mission
• Well defined & documented & publicized
• Stakeholders’ participation, including faculty
• Manageable number of objectives
• Define outcomes
• Measurable
• Feedback mechanism for improvement
Programme Educational Objective
(PEO)
•
•
•
•
•
PEO consistent with mission?
Alignment with mission of institution?
Published and known to stake-holders?
Based on needs of constituencies?
Curriculum and processes that lead to attainment
of PEO?
• Common mistakes in setting PEO
– Too broad and not specific to program
– Too narrow and similar to SLO
– Not known to stake-holders
Achievement of PEO
• Are assessment and evaluation processes in
place to determine attainment of PEO?
• Is there a continuous improvement
mechanism in place?
• Evidence and documentation is important.
Assessment of Programme Educational
Objectives
• What graduates are expected to attain within a few years
after graduation
• Ingredients to look out for:
– Objectives are current and relevant
– Methods and frequency to monitor if they are appropriate
(usually every 2 to 3 yrs)
– Constituents involved is appropriate
– Student learning outcomes will enable its attainment
– Effective assessment processes in place to evaluate
achievement
– Assessment method depends on factors like size of cohorts
– Methods can include data from advisory boards, recruiters,
employers, graduate surveys
Student Learning Outcomes
• Student learning outcomes describe what
students are expected to know and able to do by
the time of graduation
• These relate to knowledge, skills, and behaviors
that student acquire as they progress through the
program
• Student learning outcomes to be stated and
documented - in the form of graduate attributes
• Provide mapping of program SLO with EAB’s SLO,
if different
• Relationship of SLO to PEO
Table 2.1 : Mapping of Student Learning Outcomes to graduate attributes
in EAB’s Criterion 2(i) (a) to (j)
Graduate attributes in EAB’s 2(i) (a) to (j)*
Educational institution’s Student Learning Outcomes
[if different from graduate attributes in Criterion 2(i) (a)
to (j)]
(1)
Student learning outcome (..)
(2)
Student learning outcome (…)
(3)
|
(4)
|
Assessment of Student Learning
Outcomes
• Collect data to assess the level students have attained outcomes
• Evaluate results to assist in deciding how to improve teaching-learning
processes
• Key features to take note:
– Assess performance of cohorts, not individual students
– For each outcomes, identify key performance indicators@ (concrete actions
students is able to perform)
– Not necessary to collect data for every student in every course – data
collection needs to be representative
– Not necessary to have more than one data point to determine performance of
an outcome
– Not necessary to assess every outcome every year
– Focus is on how data is used to understand strengths/weaknesses and
continuous improvement
@ Definition by ABET: Specific, measurable statements identifying the performance(s) required to meet the
outcome; confirmable through evidence
Reference (from ABET)
• In program assessment planning, it is
important to let common sense prevail.
Example of how not having to assess every
outcome every year:
Course Learning outcomes
• Course Learning Outcomes describe what a student
should be capable of as a result of learning experiences
within a course
• To be determined by the course instructor (s)
• Mapping course learning outcomes to SLO to show
learning experience meets the accreditation criteria
• Each course may contribute to a list of SLOs, may
contribute strongly to one or some SLOs and less
strongly to other SLOs
• When a course may contribute to several SLOs, usually
only a subset of its strong outcomes need to be used
for EAB assessment
Defining Curriculum Objective and
Intended Course Learning Outcomes
• A learning outcome is what a student can do as a
result of a learning experience.
• It describes a specific task that student is able to
perform at a given level of competence under a
certain situation.
• The three broad types of learning outcomes are:
– Disciplinary knowledge and skills
– Generic skills
– Attitudes and values
Table 3.1: Curriculum and teaching processes to achieve Student Learning Outcomes,
and evaluation method/criteria
Module
Category#
Evaluation method & criteria
Student Learning Outcomes*
(1) (2) (3) (4) (5) (6) (7) --
Course Learning outcomes
• Course Learning Outcomes describe the
complex performances a student should be
capable of as a result of learning experiences
within a course.
• These are determined by the course instructor
(s)
• Mapping course learning outcomes to
program outcomes and how overall learning
experience meet the accreditation criteria
Contribution of each course
• Each undergraduate course in the department
contributes to a list of SLOs.
• Usually, a course may contribute strongly to
some EAB* SL outcomes and less strongly to
other outcomes.
• While a course may contribute to several NBA
SLOs, usually only a subset of its strong
outcomes need to be used for EAB*
assessment.
*Adapted from: UCLA Electrical Engineering Department’s “Guide for instructors and teaching
Assistants of undergraduate courses
Teaching-Learning Processes
• Each program should cover general and
specialized professional content
• Adequate breadth and depth, and
• Appropriate components in Science and
Humanities
• Evaluation of teaching-learning processes
• Modes of teaching-learning: lecture, tutorial,
seminar, projects, internship, peer-group
discussion, ..
Curriculum
• Does the curriculum satisfy the program specific
criteria of the particular engineering discipline?
• Are performance indicators established to
measure the outcomes of the courses with
respect to the program outcomes of NBA
criteria)?
• Major design experience?
• Prerequisites
• Course syllabi
• Cores and electives
Interaction between institution &
industry
• Involvement of industry stake-holders to
ensure relevance of curriculum
• Opportunity for students to acquire industrial
experience via internships and design projects
by professional engineers and faculty
members with industrial experience
• Communication channel with industry, e.g.
industry advisory board
Assessment
&
Demonstration of Achievement
Breadth
Depth
Where gained
Learning Process
Assessment Methods
Abilities
•
•
•
•
•
•
•
Knowledge (gathering & recall)
Comprehending information
Application (making use of knowledge)
Analysis (taking apart)
Synthesis (putting together)
Evaluation (judging the outcome)
Creation
Activities
Action Verbs that provides evidence
Knowing
define, describe, identify, label, name, outline, reproduce, recall, select, state, present, be
aware of, extract, organise, recount, write, recognise, measure, underline, repeat, relate,
know, match.
Comprehension
interpret, translate, estimate, justify, comprehend, convert, clarity, defend, distinguish,
estimate, explain, extend, generalise, exemplify, give examples of, infer, paraphrase,
predict, rewrite, summarise, discuss, perform, report, present, restate, identify, illustrate,
indicate, find, select, understand, represent, name, formulate, judge, contrast, translate,
classify, express, compare.
Application of
knowledge
(understanding)
apply, solve, construct, demonstrate, change, compute, discover,
manipulate, modify, operate, predict, prepare, produce, relate, show, use, give examples,
exemplify, draw (up), select, explain how, find, choose, assess, practice, operate, illustrate,
verify.
Analysis
recognise, distinguish between, evaluate, analyse, break down,
differentiate, identify, illustrate how, infer, outline, point out, relate, select, separate,
divide/subdivide, compare, contrast, justify, resolve, devote, examine, conclude, criticise,
question, diagnose, identify, categorise, point out, elucidate.
Synthesis
propose, present, structure, integrate, formulate, teach, develop,
combine, compile, compose, create, devise, design, explain, generate, modify, organise,
plan, rearrange, reconstruct, relate, reorganise, revise, write, summarise, tell, account for,
restate, report, alter, argue, order, select, manage, generalise, precise, derive, conclude,
build up, engender, synthesise, put together, suggest, enlarge.
Evaluation
Judge, appraise, assess, conclude, compare, contrast, describe how, criticise, discriminate,
justify, defend, evaluate, rate, determine, criticise, choose, value, question.
Bloom’s Taxonomy – Cognitive Domain
(modified by Anderson & Krathwohl)
Creation
Evaluation
Compare, decide …
Synthesis
Analysis
Application
Comprehension
Knowledge (Remembering)
Design …
Analyse …
Calculate …
Explain …
List …
Ability/
competency
Knowledge
Level
Definition
1
the remembering of previously learned material; it may involve the recall of a wide
range of material from specific facts to complete theories, but all that is required is the
bringing to mind of the appropriate information.
2
Comprehension
Application
3
4
Analysis
5
Synthesis
Evaluation
6
Creation
7
the ability to grasp the meaning of material; may be shown by translating material from
one form to another (words to numbers), by interpreting material (explaining or
summarizing), and by estimating future trends (predicting consequences or effects);
this goes one step beyond the simple remembering of material, and represent the
lowest level of understanding
the ability to use learned material in new, concrete situations; may include the
application of rules, methods, concepts, principles, laws, and theories; requires a
higher level of understanding than those under comprehension.
the ability to break down material into its component parts so that its organizational
structure may be understood; may include the identification of parts, analysis of the
relationship between parts, and recognition of the organizational principles involved;
represents a higher level than comprehension and application because it requires an
understanding of both the content and the structural form of the material.
the ability to put parts together to form a new whole; may involve the production of a
unique communication, a plan of operations (research proposal), or a set of abstract
relations (scheme for classifying information); stresses creative behaviors, with major
emphasis on the formulation of new patterns or structure
the ability to judge the value of material for a given purpose, based on definite criteria;
contains elements of all the other categories, plus conscious value judgments based on
clearly defined criteria.
Assessment of SLO
• Assessment is big subject and probably the major
challenge of the teaching faculty
• Are assessment methods adequate to provide
evidence of achievement of SLO?
• Each learning outcome may be measured or
evaluated in terms of performance indicators
• Is there a system in place to ensure that each
student will acquire the stated SLO before
graduation? (bearing in mind the various core
and optional subjects available, and overseas
attachment)
Sustainable Program Assessment
Processes
• Direct and indirect methods of assessment to
be applied to measure a wide variety of
different student abilities
• Consider best fit between program needs,
satisfactory validity and affordability (time,
money and effort)
• Need to use multiple methods to maximise
validity and reduce bias of any approach –
triangulation.
Assessment
• Assessment is one or more processes that
identify, collect, and prepare data to evaluate the
attainment of student learning outcomes and
program educational objectives.
• Effective assessment uses relevant direct,
indirect, quantitative and qualitative measures as
appropriate to the objective or outcome being
measured.
• Appropriate sampling method may be used as
part of an assessment process.
Steps for assessment design
1)
2)
3)
4)
5)
Define results to be measured
Identify data required and sources
Review existing assessment method
Define additional methods and measures
Implement & evaluate
Assessment Methods
(Gloria Rogers)
•
•
•
•
•
•
•
•
•
•
•
Locally developed examinations
Oral exam
Written surveys and questionnaires
Commercial, norm-referenced, standardized exams
Exit and other interview
Focus groups
External examiner
Portfolios
Simulations
Performance appraisals
…
Examples of Assessment Methods in EC UK
•
•
•
•
•
•
•
•
Exams
Class tests
Project reports
Presentations
Lab reports
Design studies
Vivas/orals
Posters
Assessment tools and methods
• Formative assessment
The collection of information about student
learning during the progression of a course or
program in order to improve students
learning. Example: reading the first lab reports
of a class to assess whether some or all
students in the group need a lesson on how to
make them succinct and informative.
Assessment tools and methods
• Summative assessment
The gathering of information at the conclusion of
a course, program, or undergraduate career to
improve learning or to meet accountability
demands. When used for improvement, impacts
the next cohort of students taking the course or
program. Examples: examining student final
exams in a course to see if certain specific areas
of the curriculum were understood less well than
others; analyzing senior projects for the ability to
integrate across disciplines.
Rubric
• A rubric is a set of criteria for assessing
student work or performance. Rubrics are
particularly suited to learning outcomes that
are complex or not easily quantifiable, for
which there are no clear “right” or “wrong”
answers, or which are not evaluated with
standardized tests or surveys. Assessment of
writing, oral communication, critical thinking,
or information literacy often requires rubrics.
Evaluation
• Evaluation is one or more processes for
interpreting the data and evidence
accumulated through assessment processes.
• Evaluation determines the extent to which
student outcomes and program educational
objectives are being attained.
• Evaluation results in decisions and actions
regarding program improvement.
Two Assessment Mechanisms
• The End-of-Course Instructor Survey and
• The Instructor Evaluation of Students’
Performance on an EAB*-related Problem
– It is the main mechanism used to obtain instructor
feedback on whether the students in the course
achieved some of the desired course outcomes.
Adapted from: UCLA Electrical Engineering Department’s “Guide for instructors and teaching
Assistants of undergraduate courses
Student Survey
• End-of-Course Student Surveys.
• The Student Surveys collect student input on
course material, course organization, and
instruction.
• Besides asking students questions about the
quality of a course and its instruction, the
surveys also assess, for each course, the main
topics that students are expected to have
been exposed to during the course.
Adapted from: UCLA Electrical Engineering Department’s “Guide for instructors and teaching
Assistants of undergraduate courses
Student Survey
• Students are asked to rate, on a scale from Poor to
Excellent, whether they feel they have had an
opportunity to learn the Specific Course Outcomes
well.
• The student input is then summarized and tracked in:
1. Individual reports on Course Performance for each course
offering.
2. Yearly reports on Course Performance during an academic
year.
3. Quarterly reports on Department Performance.
4. Yearly reports on Department Performance.
Adapted from: UCLA Electrical Engineering Department’s “Guide for instructors and teaching
Assistants of undergraduate courses
Other student feedback mechanisms
• Student feedback may be collected through
two additional mechanisms:
– Exit surveys administered to graduating seniors.
– Student Advisory Committee.
Saving Samples of Student Works
• Each course is required to save samples of
student homework solutions, laboratory reports,
project or design reports, and exam solutions,
typically from poor to good quality.
• At the end of each quarter, the lecturers of all
undergraduate courses must compile a binder
containing in addition to the solutions, the
corresponding homework questions, exam
questions, lab description, and project
description.
Requirements of
Continuous Quality Improvement
Figure 1 – Continuous Quality Improvement Process in Outcome-based Accreditation
Other feedback mechanisms
Institution Resources, Teaching & Learning Environment
SLO
SLO
Individual
Course
Learning
Outcomes
Course Assessment &
Evaluation by Course Lecturer
1
SLO
Course Module 1
Course Assessment &
Evaluation by Course Lecturer
1
Graduates
Working
In
industry
2
SLO Assessment & Evaluation by
Management/Accreditation
Committee
PEO Assessment & Evaluation
(Surveys, Interviews, etc) by
Management/Accreditation Committee
Continuous Quality Improvement
3
Stake-holders
Course Module 2
SLO Delivered
Curriculum & Course Modules
SLO
PEO
Institution Vision & Mission
Other Curricular & ExtraCurricular Activities
CONTINUOUS QUALITY IMPROVEMENT PROCESS WITHIN
OUTCOMES-BASED ACCREDITATION
• Loop (1) - involvement of the course teacher in
the continuous quality improvement process –
fast response
• Loop (2) - achievement of SLO at the programme
level is evaluated at the exit point
• Loop (3) - achievement of the programme
education objectives from inputs and feedbacks
from the stakeholders, e.g. industry employers
and alumni
CHALLENGES IN OBA IMPLEMENTATION
• Challenges to the education providers include:
– Understanding clearly the requirements, procedure
and policy of OBA
– Setting appropriate PEO and SLO which are relevant,
measurable and meeting OBA requirements
– Avoiding low outcome standards
– Obtaining support from top management to institute
outcomes-based teaching and learning
– Buying-in from faculty on the benefits of OBA and
securing their commitments to implement the
continuous quality improvement mechanism,
particularly at individual course module
CHALLENGES IN OBA IMPLEMENTATION
• Challenges to the education providers include:
– Training faculty on assessment and evaluation
methods which support OBA
– Instituting the continuous quality improvement
mechanism as illustrated in Figure 1, and having the
people and resources to monitor and effect the CQI
loops
– Obtaining support and feedbacks from the
stakeholders
– Having champions to lead, implement and prepare for
OBA
– Giving due recognition for contribution to OBA
Part 2–
Preparation for
Accreditation visit
What the PEV looks for?
• PEVs are sent to evaluate programs, certifying
that they satisfy the criteria
• They look for evidences that the required
criteria are met
• They identify deficiencies, weaknesses,
concerns
• They are not advisers and should not give
solutions to problems identified
Major focus
1) Outcome of the education provided;
2) Quality assurance processes, including internal
review;
3) Assessment procedures;
4) Activities and work of the students;
5) Entry standards and selection procedure for
admission of students;
6) Motivation and enthusiasm of faculty;
7) Qualifications and activities of faculty;
8) Facilities;
9) Industry participation.
Mission
• University mission?
• Faculty (school) mission & departmental
mission statement?
• Alignment with mission of institution?
• Published and known to stake-holders?
Program Educational Objective
•
•
•
•
•
PEO consistent with mission?
Alignment with mission of institution?
Published and known to stake-holders?
Based on needs of constituencies?
Curriculum and processes that lead to attainment
of PEO?
• Common mistakes in setting PEO
– Too broad and not specific to program
– Too narrow and similar to SLO
– Not known to stake-holders
Achievement of PEO
• Are assessment and evaluation processes in
place to determine attainment of PEO?
• Is there a continuous improvement
mechanism in place?
• Evidence and documentation is important.
Student Learning Outcomes
• Student learning outcomes to be stated and
documented - in the form of graduate
attributes
• Provide mapping of program SLO with EAB’s
SLO, if different
• Relationship of SLO to PEO
Assessment of SLO
• Assessment is big subject and probably the major
challenge of the teaching faculty
• Are assessment methods adequate to provide
evidence of achievement of SLO?
• Each learning outcome may be measured or
evaluated in terms of performance indicators
• Is there a system in place to ensure that each
student will acquire the stated SLO before
graduation? (bearing in mind the various core
and optional subjects available, and overseas
attachment)
Program Outcomes & TeachingLearning Processes
• Each program should cover general and
specialized professional content
• Adequate breadth and depth, and
• Appropriate components in Science and
Humanities
• Evaluation of teaching-learning processes
• Modes of teaching-learning: lecture, tutorial,
seminar, projects, internship, peer-group
discussion, ..
Curriculum
• Does the curriculum satisfy the program specific
criteria of the particular engineering discipline?
• Are performance indicators established to
measure the outcomes of the courses with
respect to the program outcomes (e.g. EAB’s (a)
to (k) criteria)?
• Major design experience?
• Prerequisites
• Course syllabi
• Cores and electives
Interaction between institution &
industry
• Involvement of industry stake-holders to
ensure relevance of curriculum
• Opportunity for students to acquire industrial
experience via internships and design projects
by professional engineers and faculty
members with industrial experience
• Communication channel with industry, e.g.
industry advisory board