Development of processes and criteria for CEAB
graduate attribute assessment
Lynann Clapham
Associate Dean (Academic)
Brian Frank
Director (Program Development)
Faculty of Applied Science
Queen's University
FACULTY OF APPLIED SCIENCE
Overview
•
•
•
•
Outcome requirements at CEAB and general higher
education in Canada
Nationwide engineering program responses
Outcomes measurement
Queen's University progress
2
FACULTY OF APPLIED SCIENCE
CEAB accreditation changes
Engineering programs are being
required to assess outcomes
and produce a process for
improvement, not just inputs
Section “3.1: Graduate attributes”
(outcomes)
3
FACULTY OF APPLIED SCIENCE
Section 3.1: Graduate Attributes
Two part requirement:
“The institution must
(1) demonstrate that the graduates of a program possess the
attributes under the following headings. The attributes will
be interpreted in the context of candidates at the time of
graduation...
(2) There must be processes in place that demonstrate that
program outcomes are being assessed in the context of
these attributes, and that the results are applied to the
further development of the program.”
4
FACULTY OF APPLIED SCIENCE
Graduate attribute measurement

This requires:
– Programs gather information that is measurable
and meaningful for them to use, not just to show
CEAB accreditation teams
– Programs have a system to gather and analyze
data over multiple years
– Programs gather enough information to improve
the program, which may involve gathering
information from students in first year, middle
years, and graduating year of the program, though
only graduating information is required
5
FACULTY OF APPLIED SCIENCE
Why is CEAB requiring outcomes
assessment?
1. Most major industrialized countries use them to
demonstrate their students' capabilities, both in
engineering and in other programs.
2. There was a danger that our students would not have their
educational credentials recognized outside Canada if we
cannot demonstrate outcomes.
Washington Accord: substantial equivalency among Australia,
Canada, Hong Kong, Republic of Ireland, New Zealand,
South Africa, United Kingdom, and United States, Japan,
Singapore,Korea, and Chinese Taipei
Bologna Declaration: compatibility among higher education
programs across Europe
6
FACULTY OF APPLIED SCIENCE
Other forces: National standards
Provincial Ministers of Education across Canada released
Ministerial Statement on Quality Assurance of Degree
Education in Canada, which recommended outcome
measures in for higher education
Ontario adopted the recommendations and released them as
Undergraduate Degree Learning Expectations for all
undergraduate programs in the province.
Similar expectations are likely to follow in other provinces.
7
FACULTY OF APPLIED SCIENCE
Ontario requirements
Undergraduate Degree Level Expectations (UDLEs) based on
national recommendations
1. Depth and Breadth of Knowledge
2. Knowledge of Methodologies
3. Application of Knowledge
4. Communication Skills
5. Awareness of Limits of Knowledge
6. Autonomy and Professional Capacity
Detailed expectations shown in document referenced in
paper. They align with CEAB graduate attributes very well.
8
FACULTY OF APPLIED SCIENCE
Why are graduate attributes (outcomes)
being adopted?
It provides information about our students' knowledge, skills,
and attitudes (outcomes), in addition to syllabi, instructor
education, library resources, etc. (inputs).
It identifies gaps between our perceptions of what we teach
and what students can do.
The information can be used to continually improve our
programs.
9
FACULTY OF APPLIED SCIENCE
Curriculum design
Currently most of us have designed
curriculum around subject
content:
This will require we consider what
we want students to do with the
content:
Thermodynamics
Fluid mechanics
Electric circuits
Economics
Solid mechanics
E.g. Students can:
...select appropriate mathematical
principles and tools to analyze
and solve engineering problems
... (Knowledge)
...evaluate and use information
based on its authority,
objectivity, and currency.
(Lifelong learning)
10
FACULTY OF APPLIED SCIENCE
What we have been given:
12 broad areas (graduate attributes), and a requirement to
create a process for program improvement. Programs will
need to decide:
What are specific
criteria?
(measurable)
How to
measure?
Graduate
attributes
Process for
program
improvement?
In which
courses?
11
FACULTY OF APPLIED SCIENCE
Generally accepted process to do this





Identify stakeholders (advisory boards, faculty, students,
staff, alumni)
Identify the program's major objectives (which must include
CEAB graduate attributes!)
Create specific measurable assessment criteria for
objectives
– e.g. “Students describe limitations of theory in
predicting behaviour of processes and systems”
Identify assessment measures appropriate for criteria
– e.g. report, oral presentation, simulation,
standardized test, lab performance, etc.
Curriculum process flow by Peter Wolf, U Guelph
12
FACULTY OF APPLIED SCIENCE
E.g. ABET's assessment flow chart
13
FACULTY OF APPLIED SCIENCE
Creating assessment criteria
•
•
•
What are a program's specific and measurable criteria?
Can involve cognitive (knowing, analyzing, creating),
beliefs, skills
Assessment criteria describes
1. level of expectation (“describes”, “compares”,
“applies”, “creates”, etc.)
2. Content area
e.g.
The student identifies constraints on a design problem including applicable
standards, economic, environmental, culture, and societal
considerations, health and safety risks
14
FACULTY OF APPLIED SCIENCE
Example: Draft assessment criteria at
Queen's University
15
FACULTY OF APPLIED SCIENCE
Curriculum mapping: Link criteria to
appropriate courses:
E.g. For a capstone course, a program might decide that the assessment
criteria used are:
D1. identify constraints including applicable standards, economic,
environmental, culture, and societal consideration, health and safety
risks (design)
D2. use creativity to establish innovative candidate solutions. (design)
L1. evaluate information from a variety of sources for objectivity,
authority, currency (lifelong learning)
L2. describe opportunities for professional development in the field
(lifelong learning)
...
These would be assessed using whatever is deemed appropriate: reports,
presentations, portfolios, surveys, etc.
16
FACULTY OF APPLIED SCIENCE
Graduate attributes similar to USA, UK,
Australia, Korea, Peru,...
CEAB
Knowledge
Problem Analysis
ABET
Knowledge
Solve problems
UK-SPEC
Knowledge
Eng. analysis
Eng Australia
Knowledge
Problem solving
Investigation
Experiments
Design
Engineering tools
Design
Techniques, tools
Design
Lab skills
Design
Techniques, tools
Teamwork
Teamwork
Teamwork
Communication
Communication
Communication
Professionalism
Impact
Ethics & Equity
Contemporary issues
Impact
Ethical responsibility
Econ. & Proj.
Manage
*(part of design)
*(part of design)
Lifelong learning
Lifelong learning
Lifelong learning
Eng practice
Social context
Ethical conduct
Professionalism
Responsibilities
Ethics
17
FACULTY OF APPLIED SCIENCE
Since expectations are similar, we can use
material from other countries
General engineering outcomes
– Learning Outcomes for UK-SPEC (UK)
– Engineers Australia Competency Standards
(Australia)
– EC2000 Outcome Attributes for ABET (USA)
– ABET criteria 2009-2010
– OCAV Undergraduate Degree-level expectations
(Ontario)
– CDIO Syllabus
Available training: ABET's week long Institute for Development
of Excellence in Assessment Leadership (IDEAL), run 2x/year
18
FACULTY OF APPLIED SCIENCE
National response of deans
Similarity among Canadian programs (more so than among US
institutions)
but
no large-scale funding to support development of assessment
procedures (like NSF funding in US for EC2000)
So... how can we work together?
19
FACULTY OF APPLIED SCIENCE
NCDEAS Education Committee
•
National Council of Deans of Engineering and Applied
Science (NCDEAS) Education Committee developing
three “exemplars” to be shared among other schools as
a starting point
– Queen's U (coordinating institution and
clearinghouse)
– U Calgary
– U Sherbrooke
20
FACULTY OF APPLIED SCIENCE
Queen's Development plan
1. Measurable faculty wide assessment criteria, e.g.:
Identify constraints including applicable standards, economic,
environmental, culture, and societal consideration, health and safety
risks (Design)
2. Leveled criteria (recommended expectations in first,
middle, and final year to track progression and recommend
curriculum targets)
3. Assessment measures for criteria, e.g.
design reports, observations, exams, surveys, oral presentations,
standardized instruments, etc.
4. Recommendations for software to gather and manage data
5. Individual program customizations, map to courses
21
FACULTY OF APPLIED SCIENCE
Progress
•
•
•
•
•
•
January: B. Frank attended ABET's Institute for
Development of Excellence in Assessment Leadership
(IDEAL)
February: began gathering functional requirements for
software to support process
March: formed 7 working groups to consider “what do
we expect our students to be able to do when they
graduate?” in the 12 areas required by CEAB
April: worked with assessment specialist on framework
for assessment criteria
June: rolling out software, developing documentation,
transitioning courses
July: Rough draft of assessment criteria assembled
22
FACULTY OF APPLIED SCIENCE
General comments
Not all courses need to assess graduate attributes
Create plan to assess some graduate attributes in each
academic term (not necessarily all attributes every year).
E.g. assess design in year 1, 3, 6 of accrediation cycle
Select how many students are being sampled
Triangulate: multiple sources of assessment for each
assessment criteria
Use a mix of direct measures (by directly observing student
skill and indirect measures (e.g. Student self-reports,
alumni opions, exit surveys, etc.)
23
FACULTY OF APPLIED SCIENCE
Queen's Timeline
Spring '09
Create faculty
wide
outcomes
Evaluate,
select
software
Fall '09
Spring '10
Department
Pilot faculty
outcomes
wide
outcomes in
some courses Map to
curriculum
Pilot software
Create
process for
Evaluate
program
improvement
Share
Fall '10
Pilot program
outcomes
Fall '11
Report
Gather data
Evaluate
24
FACULTY OF APPLIED SCIENCE
Material to be shared
•
•
Queen's U:
– Leveled assessment criteria,description of
sources, with links to UDLEs
– Assessment measures
– Example course mapping
– Documentation for open-source software to
support gathering and reporting
– Collection of references to support development
(competencies from other accreditation bodies,
curriculum development resources, etc.)
Resources from U Calgary and U Sherbrooke
25
FACULTY OF APPLIED SCIENCE
Software support
Lesson from U.S.: don't need to pour $millions into
assessment, gathering far too much data.
Over the summer we assessed learning management systems
that will be able to store a variety of types of assessment
measures and materials to simplify reporting and analyzing.
Best option for us was Moodle (open source, customizable,
extendible, low initial cost), but similar functionality is
available in Blackboard, DesireToLearn
26
FACULTY OF APPLIED SCIENCE
Moodle support for graduate attribute
assessment
27
FACULTY OF APPLIED SCIENCE
Material being collected:
Assessment wiki
Wiki link here.
28
FACULTY OF APPLIED SCIENCE
Input/questions/comments
Lynann Clapham
academic@appsci.queensu.ca
Brian Frank
brian.frank@queensu.ca
29