Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Questionnaire for Evaluation
of an Engineering Program
Submitted by:
Queen’s University at Kingston, Ontario
Mechanical Engineering
August 31, 2011
Canadian Engineering Accreditation Board
1100 – 180 Elgin Street, Ottawa, ON K2P 2K3
Tel.: (613) 232-2474 / Fax: (613) 230-5759
ceab@engineerscanada.ca
1
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Table of contents
Table of contents .......................................................................................................... 2
Glossary of terms .......................................................................................................... 2
3. Accreditation criteria ................................................................................................. 3
3.0 Continual improvement .......................................................................................... 3
Summarize the continual improvement process: .................................................................... 3
3.1
Graduate attributes ........................................................................................... 4
Appendix 3.1 Graduate Attributes.................................................................................... 20
Glossary of terms
Accreditation Units (AU) are defined on an hourly basis for an activity which is granted academic
credit and for which the associated number of hours corresponds to the actual contact time of that
activity between the student and the faculty members, or designated alternates, responsible for
delivering the program:

one hour of lecture (corresponding to 50 minutes of activity) = 1 AU

one hour of laboratory or scheduled tutorial = 0.5 AU
This definition is applicable to most lectures and periods of laboratory or tutorial work. Classes of
other than the nominal 50-minute duration are treated proportionally. In assessing the time assigned to
determine the AU of various components of the curriculum, the actual instruction time exclusive of
final examinations should be used.
Assessment tool: sources of data on student learning.
Illustrative examples: design report, presentation, essay, examination, standardized exam,
oral examination, observed behaviour, focus group, survey, etc.
Curriculum content: This need not mean an entire course dedicated to specific material; for example,
it may include separate units within an array of courses which address the material.
Curriculum map: a plotted representation (often in the form of a table) that shows the relationship
between learning experiences (e.g. courses, co-ops, co-curricular activities), instructional and
assessment methods, and intended learning for each aspect of a given program so that the
relationships and connections among all the elements are easily seen.
Illustrative Example: If a program identifies three indicators to demonstrate the graduate
attribute Lifelong Learning, as described above, a table could be used to show which learning
experiences (e.g. courses) are used to develop abilities and assess indicators
Graduate Attribute: generic characteristics, specified by the CEAB, expected to be exhibited by
graduates of Canadian engineering schools
Indicator: descriptors of what students must do to be considered competent in the attribute; the
measurable and pre-determined standards used to evaluate learning
2
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Illustrative example: Criterion 3.1.12 requires that students possess the attribute Lifelong Learning. A
program might consider that the indicators required to demonstrate that students possess this attribute
are:



Critically evaluates procured information for authority, currency, and objectivity.
Describes professional and academic societies in the discipline and how new knowledge
enters discipline.
Identifies resources and professional associations that address student’s own ongoing
professional development.
Minimum path: The set of courses, including common-core, program compulsory, option compulsory
and elective courses which provide the least number of AU within each Accreditation Board curriculum
content category.
Modern engineering tools: This refers to tools such as equipment, processes, codes of practice,
software, simulation packages, etc. that are considered essential for the given discipline.
“Qualified” Accreditation Units: Curriculum content delivered by faculty members that meet the
Accreditation Board accreditation licensure requirements. Engineering licensure is examined only for
courses that include engineering science and/or engineering design curriculum content. Please see the
Statement of interpretation on licensure expectations and requirements for further information.
Weakest-link principle: All options in the program are examined. Following the principle that a
program is only as strong as its “weakest link”, a program is accredited only if all such variations meet
the criteria.
3. Accreditation criteria
The following sections describe the measures used by the Accreditation Board to evaluate Canadian
engineering programs for the purpose of accreditation. The number indicated beside each criterion
corresponds with the Accreditation Board’s Accreditation Criteria and Procedures for 2010.
3.0 Continual improvement
Summarize the continual improvement process:
In 2009 the Faculty of Engineering and Applied Science began developing a process for graduate
attribute assessment by forming working groups to establish measurable and meaningful
indicators of graduate attributes appropriate to first year, middle years, and at graduation. In
the 2009-2010 academic year the Faculty piloted a process for attribute assessment within
some faculty-wide first year courses. In the summer of 2010 the processes were reviewed and
expanded, and in the 2010-2011 academic year the process of attribute assessment was
broadened to involve include faculty wide first and second year courses, and the fourth year
capstone design, communications and project management courses MECH 460/464.
3
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
For this cycle attribute assessment was a separate process from the remainder of accreditation
preparation. As a result the detailed response to Criterion 3.0 and 3.1 are contained in a
separate document in Appendix 3.1, which details the processes followed faculty wide and
common to all programs for the 2010-2011 academic year. It lists the indicators measured,
where they were measured, a description of how the curriculum contributes to the
development of the attributes, a description of the student activities used to measure the
attributes, data collected from student work and surveys, an evaluation of the data, and
proposals for program improvement. Based on evaluation of the data from student work and
surveys, revisions are planned as follows:





The existence and importance of attributes for engineering practice will be
communicated and used more extensively, and linked to learning objectives in courses.
Some of the attributes appear to be poorly understood by students, and this will be
addressed
At the first year level, the program is being revised in the areas including making
effective arguments, evaluating complex problem solutions against objectives, written
communications, and evaluating information. Grader calibration is being enhanced to
reduce variation between graders, and some indicators will be revised to reflect
student ability.
At the second year level, changes are being made to the faculty-wide communications
and economics courses. More emphasis will be placed on summarizing important
information clearly and concisely, effectively participating in informal small group
discussions, and on risk assessment and project planning.
A 4-year sequence of courses in engineering design and practice is partway through
implementation, which will support development of professional skill attributes, and
provide a venue for developing and assessing all attributes in broad integrative
experiences, like team projects, that emulate professional practice.
The outcomes assessed in the Mechanical and Materials Engineering capstones MECH
460/464 (See Below) will be harmonized with the new outcomes developed in the
faculty wide courses. A project this summer has collected a list of over 1000 outcomes
being assessed, but not yet recorded, in our current offerings. Analysis of this data will
allow a more complete picture of our program in combination with the faculty wide
curriculum rolling out this fall.
Because the process of attribute assessment generally started in faculty wide courses, the
faculty-wide data and analysis is more detailed and should be viewed as representing an
example of process that will augment our graduating year assessment in the coming years. The
faculty plans to continue assessing student attributes throughout the four-year program for the
purpose of program improvement. As curriculum changes occur over the next two years, the
program evaluation and improvement process will be expanded to include more comprehensive
coverage of all attributes at the first and graduating years of the program.
3.1
Graduate attributes
The higher education institution must demonstrate that the graduates of a program possess the
attributes under the following headings. The attributes will be interpreted in the context of candidates
4
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
at the time of graduation. It is recognized that graduates will continue to build on the foundations that
their engineering education has provided.
Engineering programs are expected to continually improve. There must be processes in place that
demonstrate that program outcomes are being assessed in the context of these attributes, and that the
results are applied to the further development of the program.
Response to criterion 3.1:
This section presents only data specific to the Mechanical and Materials Engineering (MME)
program and is additional to the information on process and faculty wide curriculum presented
in Appendix 3.1. The details presented in sections 3.1.1 through 3.1.12 cover only the
assessments made in the capstone courses MECH 460/464 and do not include the faculty wide
information found in the appendix.
MME Program Curriculum Map
In the summer of 2011 a project in MME collected a list of over 1000 outcomes based on a
retrospective survey of assessments actually applied in MME program courses during the
2010/2011 year. For each assessment (report, exam question, problem set) in each course, the
marks assigned were associated with an outcome or outcomes such as “Applies Bernoulli’s
equation to a pipe flow” or “Composes clear and logical arguments”. Those outcomes were
categorized by CEAB Attribute, CDIO Syllabus Topic, and Bloom Verb for knowledge
characterization, among other details. This analysis is not yet complete, however the
curriculum map table below lists courses where teaching and assessment has been carried out
in each of the Attribute areas. The collected outcomes will be used as explicit indicators to
gather data on student performance starting in 2011/2012, with initial concentration on
courses with strong relationships to attributes 2 through 12 to capture performance in areas
other than the Engineering Knowledge Base.
Attribute
Developed
Assessed
Knowledge base
Extensively
Extensively
5
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
problem analysis
Extensively
APSC-100, MECH-396,397,398,399
Investigation
APSC-100, MECH
215/396/397/398/399
(not this year)
Design
APSC-100, MECH 212/323
APSC-100,MECH-323,212,460, Math272
Engineering
tools
APSC -100, 142, 161, MECH
323
APSC 161, Math 272, MECH 213, MECH 323
Communication
APSC-100, 291, 292, MECH
464
APSC-100, 291,292, MECH213,396,397,398,399,464
Individual and
team work
APSC-100, MECH 212
APSC-100, MECH 212
Impact of
engineering on
society
APSC-100, 151
APSC-100, 151
Ethics and
equity
APSC-100
APSC-100
Professionalism
APSC-100, MECH 212
APSC-100, MECH 212, 460,464
Economics and
project
management
APSC-100, 221
APSC-100, 221
Lifelong learning
APSC-100, 291, 292
APSC-100, 291, 292, MECH 464
Graduating year assessment
In 2010-2011, the final year class was assessed on the basis of performance in the jointly
offered MECH 460/464 capstone design, communications and project management courses.
Students were assessed both on a team basis (Group Assignment in the legend) and as
individuals (Assignment) for outcomes taken from the faculty wide list in development and
from the CDIO Syllabus where additional coverage was required. All outcomes were assessed
directly by regular or adjunct faculty members who are Professional Engineers.
Data were accumulated for each outcome during the marking process and stored in the Moodle
course management system. The charts that follow were completed based on Moodle records
at the end of the academic year. The data for all students in both courses was downloaded,
aligned to provide a complete record for each student, and then anonymized for privacy. The
resulting frequencies were accumulated over all students and plotted for each outcome in each
assignment. The overall data set includes 96 students, the majority of whom participated in
both courses. The data does not capture the performance of students taking the APSC 480
Multidisciplinary Design course.
Each of the FEAS outcomes were ranked on a scale of:




Not Demonstrated, which included instances where the outcome was not addressed
Marginal
Meets Expectations
Exceeds Expectations
6
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
CDIO Syllabus outcomes were ranked on an Engineering Practice scale of:





Resubmit: The work is unsatisfactory and must be revised.
File Only: The work is sufficient to provide a record for the file.
Internal Circulation: I would show this to my boss.
Almost Client Ready: Only a few minor tweaks are required.
Client Ready: The work is ready for submission to an outside client with no revisions.
All of the outcomes are satisfactory except Resubmit and Not Demonstrated. File Only or
Marginal is below expectations, but sufficient to pass the course.
The full numerical data is available in the file MECH460—F-464—FW Combined Outcomes
Anonymized.xlsx available as part of the on-site exhibits.
The following outcomes were used as indicators in the graduating year for MME students. The
faculty wide indicators have been supplemented by several CDIO indicators as needed. Each
indicator (or outcome) was explicitly assessed as part of the MECH 460/464 capstone project
course evaluation by project faculty supervisors.
7
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Sec.
3.02
3.02
Attribute
Code
Indicator
Course Deliverable
Problem analysis
3.02-GY1
Identifies problem, known and unknown
information, uncertainties, and biases
Final Report
3.02-GY2
Creates process for solving problem
including justified approximations and
assumptions
Final Report
Final Report
Problem analysis
3.02
Problem analysis
3.02-GY3
Selects and applies appropriate model and
analysis to solve problems
3.02
Problem analysis
3.02-GY4
Evaluates validity of results and model for
error, uncertainty
Final Report
3.04-GY1
Identify problem and constraints including
health and safety risks, applicable
standards, economic, environmental,
cultural and societal considerations
Proposal Report
3.04-GY2
Applies appropriate knowledge,
judgement, and design tools, in creating
and analyzing conceptual design solutions
to select best concept
Proposal Report
3.04-GY3
Creates and tests simulations, models,
and/or prototypes at various points in
design with complexity appropriate to
design stage
Final Report
3.04-GY4
Assesses design performance based on
requirements, yield, reliability, and/or
safety as appropriate
Final Report
3.04-GY5
Identifies possibilities for further
improvement and conducts design review
to evaluate performance of the overall
process.
Final Report
Final Report
3.04
3.04
3.04
3.04
3.04
Design
Design
Design
Design
Design
3.05
Engineering tools
3.05-GY1
Evaluates techniques, resources, and tools
to identify their limitations with respect
to needs
3.05
Engineering tools
3.05-GY2
Applies appropriate techniques, tools, and
processes to accomplish a task
Final Report
3.05
Engineering tools
3.05-GY3
Evaluates appropriateness of results from
several engineering techniques and tools
Final Report
Demonstrates capacity for initiative and
technical or team leadership while
respecting others' roles
Meeting Leader,
Memo and Minutes /
Supervisor Assessment
of Individual
Contribution
3.06
Teamwork
3.06-GY1
3.06
Teamwork
3.06-GY2
Evaluates team effectiveness and plans for
improvements
Not Explicitly
Assessed, although
students completed
peer evaluations at
mid term and the end
of term.
3.07
Communications
3.07-GY1
Writes and revises documents using
appropriate discipline-specific conventions
Proposal Report
3.07
Communications
3.07-GY2
Demonstrates accurate use of technical
vocabulary
Proposal Report
3.07
Communications
3.07-GY3
Demonstrates confidence in formal and
Meeting Leader,
8
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
informal oral communications
Memo and Minutes/
Supervisor Assessment
of Individual
Contribution
3.07
Communications
3.07-GY4
Uses appropriate referencing to cite
previous work
Proposal Report
3.07
Communications
3.07-GY5
Uses graphics to explain, interpret, and
assess information
Proposal Report
CDIO 3.2.3
Compose meeting minutes and progress
memo
Meeting Leader,
Memo and Minutes
CDIO 3.2.6
Demonstrate answering questions
effectively
Meeting Leader,
Memo and Minutes /
Proposal Presentation
or Final Poster
Presentation
Final Report
3.07
3.07
Communications
Communications
3.08
Professionalism
3.08-GY1
Integrates standards, codes of practice,
and legal and regulatory factors into
decision-making processes (as
appropriate)
3.08
Professionalism
3.08-GY2
Demonstrates professional bearing
Meeting Leader,
Memo and Minutes
3.09
Impact of
engineering
3.09-GY1
Considers economic, social, and
environmental factors and/or impacts in
decisions
Final Report
3.09
Impact of
engineering
3.09-GY2
Evaluates trade-offs among goals and
concepts
Proposal Report
3.09
Impact of
engineering
3.09-GY3
Explains the societal, enterprise, and/or
technical context of the system
Proposal Report
3.11
Economics and
project
management
CDIO 4.3.4
Apply project Control for cost,
performance, and schedule in the
estimation and allocation of resources
considering risks and alternatives.
(typically includes a Gantt Chart and Work
plan)
Proposal Report
3.12
Lifelong learning
3.12-GY1
Critically evaluates procured information
for authority, currency, and objectivity.
Proposal Report
Demonstrate skills of self-education
Supervisor Assessment
of Individual
Contribution
3.12
Lifelong learning
3.12-GY2
Overall Student performance on outcome measures is strong, with 90% meeting or exceeding
expectations on individual outcomes. This performance is consistent across the attributes
assessed. The remaining students showed marginal outcomes in these areas, sufficient to meet
our threshold requirements to pass the course. These marginal outcomes were not strongly
clustered in individual students or project groups so that in aggregate, all but eight students
met our expectations and those eight met expectations in half or more of the outcomes.
Outcomes of Not Demonstrated appear under several attributes. These instances were
investigated with faculty supervisors. They are generally a result of perceptions by the students
and faculty supervisor that addressing that outcome was not essential to that particular
project. For example, a team designing an on-board braking system for an SAE Mini-Baja
9
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
vehicle would have had to make a big reach to include substantial elements of impact on
society and the environment. This is the first application of these outcomes in our final year
projects and many project scopes were defined before the assessment scheme was finalized.
We will address this in the future by ensuring that the scope of each project includes elements
representative of the full outcomes list, in combination with assessing some outcomes, e.g. 10.
Ethics and Equity in other parts of the curriculum or on deliverables not directly tied to
capstone project reports.
Students show considerable progress in communications abilities over time in comparison to the
similar cohort assessed this year in APSC 291/292. Anecdotal observation also indicates
substantial progress in design, team, and project skills from their performance in the MECH 212
design course. Repeated experience over multiple years in contexts based on engineering
practice and design in teams develops the skills necessary for engineering practice. This is not
surprising.
Our students are strongest in the outcomes based on presentations in various forms ranging
from informal meetings to poster sessions and formal slide presentations. Marginal outcomes
appear in more substantial numbers in both the technical and communications elements of
their proposals and final reports, although the overall quality is still very high.
We are in the process of expanding the outcomes assessment to other courses to allow us to
collect more comprehensive data on student performance with respect to the graduate
attributes. This will be especially useful where all students are working on activities with the
same objectives and scope, all marked by the same faculty member. Individual faculty
variation adds considerable noise to the assessments on final year projects.
10
This chart shows the outcomes from the combined MECH 460/464 results grouped by assignments.
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
11
This chart shows the outcomes from the combined MECH 460/464 results grouped by CEAB Attribute.
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
12
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Mechanical Engineering Program specific development
Mechanical and Materials Engineering has been an active partner in the development of the
faculty wide initiatives summarized above and detailed in Appendix 3.1, and has been actively
revising and improving curricula towards design and engineering practice based goals since at
least the early nineties. We pioneered development of curriculum and prototype facilities
eventually incorporated in the Integrated Learning Initiative and the Integrated Learning
Centre, including:



MECH 212 design course in second year that incorporates two project activities and
design build within a context of engineering practice. The new APSC 200 design course
is largely modelled on MECH 212’s success.
The ILC Active Learning Centre was prototyped with found materials on the third floor
of Jackson Hall in support of MECH 212.
The ILC Plaza Labs were prototyped on the first floor of Jackson Hall to support more
open-ended lab activities in MECH 215.
Mechanical and Materials Engineering at Queen’s was one of the founding members of the CDIO
Initiative in Engineering Education (cdio.org) which now includes more than 70 member
institutions around the world. We hosted the First International CDIO Conference in 2005 and
we continue active participation and one of our faculty, Rick Sellens, holds a seat on the CDIO
Council.
CDIO Standards are closely aligned with the Washington Accord and with national and
international accreditation requirements based on them, such as ABET (a-k) and the CEAB
Graduate Attributes. Mapping by Cloutier, Hugo and Sellens1 shows that the highly detailed
CDIO Syllabus is a superset of the topics required to meet the CEAB Graduate Attributes and
shows how programs may select from the Syllabus to meet the Attributes. Many of the Queen’s
faculty wide outcomes listed above are adapted from CDIO and additional outcomes based on
CDIO have been used in evaluating some aspects of our program not covered by the faculty
wide outcomes.
In 2005 we completed an initial assessment of our curriculum against CDIO Standards and
Syllabus, including a mapping of our courses to the relevant areas of the Syllabus on an
Introduce, Teach, Utilize basis, and completion of stakeholder surveys. The stakeholder surveys
showed consistency with others internationally and strong agreement among faculty, students,
alumni and industry partners. A major result of that assessment was a refocusing of design
activities in all years, especially capstone (see section 3.3.4.4 of the MME Questionnaire), to
focus on industrially relevant design activities with practical consequences in the real world. In
addition, many faculty members reworked elements of their engineering science courses to
deliver materials in the context of real systems and engineering practice following CDIO
Standard 1.
1
Guy Cloutier, Ron Hugo and Rick Sellens, Mapping the relationship between the CDIO Syllabus and the CEAB
Graduate Attributes: An Update, paper 132, 7th International CDIO Conference 2011, Copenhagen, Denmark, June
20-23, 2011.
13
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
In the summer of 2011, we are gathering a detailed list of what outcomes were actually
assessed across our core curriculum in 2010/2011 for comparison against the earlier mapping
and to determine which changes have actually stuck and been carried through to student
assessment. These outcomes from individual courses are being categorized in terms of both the
CDIO Syllabus and the CEAB Attributes to provide further evidence that they are parallel
systems with very similar objectives. Once the mapping is complete we will have data on how
many of the AU within our core program relate to each of the different topics in the Syllabus
and to each of the different CEAB Graduate Attributes. Where possible, the outcomes will be
monitored in Moodle to allow us to collect comprehensive data on our student performance,
and support a further process of continuous improvement.
3.1.1 A knowledge base for engineering
Response to criterion 3.1.1:
Details for faculty wide courses are presented in Appendix 3.1. This attribute is developed in
various parts of the program and indicators have been identified as part of our summer 2011
curriculum project. These indicators will be evaluated in 2011/2012.
The data available from student performance in technical courses indicate that students are
meeting or exceeding our expectations for this attribute as a requirement of graduation.
The results from the 2011/2012 assessments will be combined with the faculty wide data from
Appendix 3.1 and follow the process defined there to improve coverage and student
performance as part of our ongoing curriculum monitoring process under the CDIO Standards.
3.1.2 Problem analysis
Response to criterion 3.1.2:
Details for faculty wide courses are presented in Appendix 3.1. This attribute is developed in
various parts of the program and indicators were collected as part of the assessment process in
MECH 460/464 as detailed above. Indicators evaluated for this attribute were:
“Identifies problem, known and unknown information, uncertainties, and biases” was evaluated
in Final Report by the Faculty Supervisor.
“Creates process for solving problem including justified approximations and assumptions” was
evaluated in Final Report by the Faculty Supervisor.
“Selects and applies appropriate model and analysis to solve problems” was evaluated in Final
Report by the Faculty Supervisor.
“Evaluates validity of results and model for error, uncertainty” was evaluated in Final Report
by the Faculty Supervisor.
14
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
The data indicate that more than 90% of students are meeting or exceeding our expectations
for this attribute at the fourth year level. However, about 25% of students did not adequately
evaluate the validity and uncertainty of their solutions.
These results will be combined with the faculty wide data from Appendix 3.1 and follow the
process defined there to improve coverage and student performance as part of our ongoing
curriculum monitoring process under the CDIO Standards. Emphasis will be added to their
reporting requirements to increase their focus on validity of results.
3.1.3 Investigation
Response to criterion 3.1.3:
Details for faculty wide courses are presented in Appendix 3.1. This attribute is developed in
various parts of the program, particularly in MECH 215/396/397/398 lab courses. Indicators
have been identified as part of our summer 2011 curriculum project and will be applied to the
lab course in 2011/2012.
The data collected by direct performance assessment will indicate if students are meeting or
exceeding our expectations for this attribute.
These results will be combined with the faculty wide data from Appendix 3.1 and follow the
process defined there to improve coverage and student performance as part of our ongoing
curriculum monitoring process under the CDIO Standards.
3.1.4 Design
Response to criterion 3.1.4:
Details for faculty wide courses are presented in Appendix 3.1. This attribute is developed in
various parts of the program, including APSC 100, MECH 212/323/460, and indicators were
collected as part of the assessment process in MECH 460/464 as detailed above. Indicators
evaluated for this attribute were:
“Identify problem and constraints including health and safety risks, applicable standards,
economic, environmental, cultural and societal considerations” was evaluated in Proposal
Report by the Faculty Supervisor.
“Applies appropriate knowledge, judgement, and design tools, in creating and analyzing
conceptual design solutions to select best concept” was evaluated in Proposal Report by the
Faculty Supervisor.
“Creates and tests simulations, models, and/or prototypes at various points in design with
complexity appropriate to design stage” was evaluated in Final Report by the Faculty
Supervisor.
“Assesses design performance based on requirements, yield, reliability, and/or safety as
appropriate” was evaluated in Final Report by the Faculty Supervisor.
“Identifies possibilities for further improvement and conducts design review to evaluate
performance of the overall process” was evaluated in Final Report by the Faculty Supervisor.
15
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
The data indicate that over 90% of students are meeting or exceeding our expectations for this
attribute at the fourth year level. About 12% of students do not adequately transfer knowledge
from the technical courses to their design project.
These results will be combined with the faculty wide data from Appendix 3.1 and follow the
process defined there to improve coverage and student performance as part of our ongoing
curriculum monitoring process under the CDIO Standards. Knowledge application will be
improved through our increasing integration of design activities into engineering science
courses under CDIO.
3.1.5 Use of engineering tools
Response to criterion 3.1.5:
Details for faculty wide courses are presented in Appendix 3.1. This attribute is developed in
various parts of the program, including APSC 100/142/161, MATH 272, and MECH 212/323 and
indicators were collected as part of the assessment process in MECH 460/464 as detailed
above. Indicators evaluated for this attribute were:
“Evaluates techniques, resources, and tools to identify their limitations with respect to needs”
was evaluated in Final Report by the Faculty Supervisor.
“Applies appropriate techniques, tools, and processes to accomplish a task” was evaluated in
Final Report by the Faculty Supervisor.
“Evaluates appropriateness of results from several engineering techniques and tools” was
evaluated in Final Report by the Faculty Supervisor.
The data indicate that more than 90% of students are meeting or exceeding our expectations
for this attribute at the fourth year level. Many students did not explicitly evaluate the
appropriateness of tools for the application, typically in instances where the appropriateness
was obvious, as in applying a finite element package to stress analysis in a component.
These results will be combined with the faculty wide data from Appendix 3.1 and follow the
process defined there to improve coverage and student performance as part of our ongoing
curriculum monitoring process under the CDIO Standards.
3.1.6 Individual and team work
Response to criterion 3.1.6:
Details for faculty wide courses are presented in Appendix 3.1. This attribute is developed in
various parts of the program, including APSC 100 and MECH 212/460/464, and indicators were
collected as part of the assessment process in MECH 460/464 as detailed above. Indicators
evaluated for this attribute were:
“Demonstrates capacity for initiative and technical or team leadership while respecting others'
roles” was evaluated in Meeting Leader, Memo and Minutes / Supervisor Assessment of
Individual Contribution by the Faculty Supervisor.
16
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
“Evaluates team effectiveness and plans for improvements” was not explicitly assessed,
although students completed peer evaluations at mid term and the end of term.
The data indicate that students are meeting or exceeding our expectations for this attribute at
the fourth year level. This is unsurprising given their teaming experiences in APSC 100 followed
by the MECH 212 design course which provides the model for the new faculty wide APSC 200
course.
These results will be combined with the faculty wide data from Appendix 3.1 and follow the
process defined there to improve coverage and student performance as part of our ongoing
curriculum monitoring process under the CDIO Standards.
3.1.7 Communication skills
Response to criterion 3.1.7:
Details for faculty wide courses are presented in Appendix 3.1. This attribute is developed in
various parts of the program, including APSC 100/291/292 and MECH 464, and indicators were
collected as part of the assessment process in MECH 460/464 as detailed above. Indicators
evaluated for this attribute were:
“Writes and revises documents using appropriate discipline-specific conventions” was
evaluated in Proposal Report by the Faculty Supervisor.
“Demonstrates accurate use of technical vocabulary” was evaluated in Proposal Report by the
Faculty Supervisor.
“Demonstrates confidence in formal and informal oral communications” was evaluated in
Meeting Leader, Memo and Minutes / Supervisor Assessment of Individual Contribution by the
Faculty Supervisor.
“Uses appropriate referencing to cite previous work” was evaluated in Proposal Report by the
Faculty Supervisor.
“Uses graphics to explain, interpret, and assess information” was evaluated in Proposal Report
by the Faculty Supervisor.
“Compose meeting minutes and progress memo” was evaluated in Meeting Leader, Memo and
Minutes by the Faculty Supervisor.
“Demonstrate answering questions effectively” was evaluated in Meeting Leader, Memo and
Minutes / Proposal Presentation or Final Poster Presentation by the Faculty Supervisor.
The data indicate that all students are meeting or exceeding our expectations for oral
communications at the fourth year level, with the exception of a few students who are
marginal in responding to questions during oral presentations. Less than 10% of our students
still have some difficulty meeting expectations in the details of referencing, vocabulary, and
discipline specific conventions in their writing, but are otherwise strong in their written
communications.
These results will be combined with the faculty wide data from Appendix 3.1 and follow the
process defined there to improve coverage and student performance as part of our ongoing
17
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
curriculum monitoring process under the CDIO Standards. More emphasis will be place on the
details in MECH 464 lectures and additional detailed material will be incorporated into the new
APSC 293 communications course paired with the faculty wide APSC 200 design project course,
which should produce improvements in future years.
3.1.8 Professionalism
Response to criterion 3.1.8:
Details for faculty wide courses are presented in Appendix 3.1. This attribute is developed in
various parts of the program, including APSC 100/151 and MECH 212, and indicators were
collected as part of the assessment process in MECH 460/464 as detailed above. Indicators
evaluated for this attribute were:
“Integrates standards, codes of practice, and legal and regulatory factors into decision-making
processes (as appropriate)” was evaluated in Final Report by the Faculty Supervisor.
“Demonstrates professional bearing was evaluated” in Meeting Leader, Memo and Minutes by
the Faculty Supervisor.
The data indicate that more than 90% of students are meeting or exceeding our expectations
for this attribute at the fourth year level. Some design projects fall in areas outside the realms
of established standards and codes of practice.
These results will be combined with the faculty wide data from Appendix 3.1 and follow the
process defined there to improve coverage and student performance as part of our ongoing
curriculum monitoring process under the CDIO Standards. Greater attention will be given to
including elements of standards within all design projects.
3.1.9 Impact of engineering on society and the environment
Response to criterion 3.1.9:
Details for faculty wide courses are presented in Appendix 3.1. This attribute is developed in
various parts of the program, including APSC 100/151, and indicators were collected as part of
the assessment process in MECH 460/464 as detailed above. Indicators evaluated for this
attribute were:
“Considers economic, social, and environmental factors and/or impacts in decisions” was
evaluated in Final Report by the Faculty Supervisor.
“Evaluates trade-offs among goals and concepts” was evaluated in Proposal Report by the
Faculty Supervisor.
“Explains the societal, enterprise, and/or technical context of the system” was evaluated in
Proposal Report by the Faculty Supervisor.
The data indicate that over 95% of students are meeting or exceeding our expectations for this
attribute at the fourth year level. In some design projects their assessment of context
indicated minimal social or environmental impacts and thus no opportunity to consider
mitigation of those impacts.
18
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
These results will be combined with the faculty wide data from Appendix 3.1 and follow the
process defined there to improve coverage and student performance as part of our ongoing
curriculum monitoring process under the CDIO Standards. Greater attention will be given to
including elements of social and environmental impact within the scope of all design projects.
3.1.10 Ethics and equity
Response to criterion 3.1.10:
Details for faculty wide courses are presented in Appendix 3.1. This attribute is developed in
various parts of the program, primarily in APSC 100, but not explicitly assessed in the MME
program outside the faculty wide courses.
Anecdotal sources suggest that most students are meeting or exceeding our expectations for
this attribute, however empirical data is required.
We will review the coverage in the faculty wide data from Appendix 3.1 and the new second
year curriculum coming on line this fall and follow the process defined there to improve
coverage and student performance as part of our ongoing curriculum monitoring process under
the CDIO Standards.
3.1.11 Economics and project management
Response to criterion 3.1.11:
Details for faculty wide courses are presented in Appendix 3.1. This attribute is developed in
various parts of the program, particularly in APSC 100/221 and MECH 464, and indicators were
collected as part of the assessment process in MECH 460/464 as detailed above. Indicators
evaluated for this attribute were:
“Apply project Control for cost, performance, and schedule in the estimation and allocation of
resources considering risks and alternatives. (Typically includes a Gantt Chart and Work plan)”
was evaluated in Proposal Report by the Faculty Supervisor.
The data indicate that over 90% of students are meeting or exceeding our expectations for this
attribute at the fourth year level.
These results will be combined with the faculty wide data from Appendix 3.1 and follow the
process defined there to improve coverage and student performance as part of our ongoing
curriculum monitoring process under the CDIO Standards.
3.1.12 Life-long learning
Response to criterion 3.1.12:
Details for faculty wide courses are presented in Appendix 3.1. This attribute is developed in
various parts of the program, particularly in APSC 100/291/292 and MECH 464, and indicators
were collected as part of the assessment process in MECH 460/464 as detailed above.
Indicators evaluated for this attribute were:
19
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
“Critically evaluates procured information for authority, currency, and objectivity” was
evaluated in Proposal Report by the Faculty Supervisor.
“Demonstrate skills of self-education” was evaluated in Supervisor Assessment of Individual
Contribution by the Faculty Supervisor.
The data indicate that over 90% of students are meeting or exceeding our expectations for this
attribute at the fourth year level.
These results will be combined with the faculty wide data from Appendix 3.1 and follow the
process defined there to improve coverage and student performance as part of our ongoing
curriculum monitoring process under the CDIO Standards.
Appendix 3.1 Graduate Attributes
Appendix 3.1 Contents
Appendix 3.1 Contents................................................................................................................................ 20
Section 3.0 – Continual improvement process ........................................................................................... 22
Section 3.1 Part A: Demonstration of student attributes ........................................................................... 24
Purpose and outcomes ........................................................................................................................... 25
Curriculum mapping ............................................................................................................................... 29
Assessment and evaluation .................................................................................................................... 33
Faculty-wide First year ........................................................................................................................ 33
Faculty-wide middle year assessment: Communications ................................................................... 69
Faculty-wide Middle year assessment: Economics ............................................................................. 75
Student input on Graduate attributes ................................................................................................ 77
Program development informed by data ............................................................................................... 82
Faculty wide development .................................................................................................................. 82
Section 3.1 Part B: Future plans for graduate attribute assessment .......................................................... 83
Appendix 3.1A: Leveled indicators.............................................................................................................. 85
Appendix 3.1B: Student surveys ................................................................................................................. 91
Appendix 3.1C: Definitions and Guidelines................................................................................................. 96
Criterion 3.1 ............................................................................................................................................ 96
Supplementary guidelines for Criterion 3.1 ............................................... Error! Bookmark not defined.
Definitions .............................................................................................. Error! Bookmark not defined.
20
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
21
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Section 3.0 – Continual improvement process
Engineering programs are expected to continually improve. There must be processes in place that
demonstrate that program outcomes are being assessed in the context of the graduate attributes, and
that the results are applied to the further development of the program. Summarize the continual
improvement process.
The Faculty of Engineering and Applied Science is committed to continual program improvement that
has historically focused on many of the attributes in Criterion 3.1. In the mid 90’s the faculty began an
initiative, labelled Integrated Learning, to promote integration of theory and practice, and concepts
from academia and industry. The faculty saw a need to enhance development of professional skills and
an understanding of the engineering profession in the first year program. Informed by data from pilot
projects and an awareness of outcomes-based assessment being introduced in the USA via EC2000, a
committee of faculty, administration, and industry advisors designed new facilities and curriculum to
developing competence in areas now covered by the CEAB graduate attributes. This led to the
introduction of two new team and project-based first year courses focusing on problem analysis, design,
investigation, engineering tools, teamwork, communications, professionalism, ethics and equity, and
lifelong learning.2
Over the past 10 years these courses have seen regular re-development based on student feedback and
assessment3. Previously existing courses have also been going through a development process; a course
on earth systems has been developing a stronger emphasis on the impact of engineering on society and
the environment, and courses on engineering graphical communications and computer programming
include open-ended design projects.
With the introduction of the graduate attribute assessment requirements the faculty has begun
formalizing an approach to program development informed by measurable and meaningful data.
In the 2009-2010 academic year the Faculty piloted a process for attribute assessment within some first
year courses. At the end of the year the processes were reviewed and expanded to cover graduating
year courses for the 2010-2011 academic year. The process being developed is shown in Figure 1 and
follows other accepted processes for outcomes assessment.4
In 2009 the Faculty created seven working groups to establish measurable and meaningful indicators
appropriate to first year, middle years, and at graduation. Although current accreditation regulations
2
See J. McCowan, C. Knapper, An Integrated and Comprehensive Approach to Engineering Curricula, (Three parts),
Int. J. Eng. Ed, Vol. 18, No. 6, 2002.
3
See A. Topper, L. Clapham, From Experiments to Experimentation; A New Philosophy for First Year Laboratories,
C2E2 2001, Victoria; B. Frank, J. Mason, Impact of Peer-Managed Project-Based Learning in First Year Engineering,
ASEE 2008 General Conference, Pittsburgh, PA
4
P. Wolf, New Directions for Teaching and Learning, Volume 2007, Issue 112, pp. 15-20
22
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
require assessment only at graduation, the faculty is assessing attribute development in earlier years for
several reasons:
1. The ability to apply the results of graduate attribute assessment to the further development of
the program is improved when data is available for earlier years.
2. Creating expectations by years greatly improves the efficiency and quality of curriculum design
3. The first year of the engineering program is common to all engineering students and is
administered by the faculty office. It is efficient to benchmark all incoming students.
4. Most of the professional skill attributes are specifically targeted in the first two years of the
current program and not in upper years. The curriculum is being developed to ensure these
attributes continue to be developed in upper years.
Figure 1 - Current graduate attribute assessment process
Working groups were composed of representatives from departments across the faculty, subject experts
in areas required by the graduate attributes (library science, economics, communications, etc.) and
students. An educational developer was consulted throughout the process. The groups used several
resources including the CDIO syllabus and resources from other engineering accreditation bodies. The
23
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
requirements from the Ontario Council of Academic Vice Presidents Guidelines for University
Undergraduate Degree Learning Expectations (UDLEs) were also used.5
1.
Depth and Breadth of Knowledge
2.
Knowledge of Methodologies
3.
Application of Knowledge
4.
Communication Skills
5.
Awareness of Limits of Knowledge
6.
Autonomy and Professional Capacity
Appendix 3.1A shows the indicators created by these groups in a single table.
At the beginning of the 2010-2011 academic year, a set of indicators from Appendix 3.1A was selected
for assessment in the 2010-2011 academic year based on the current state of the curriculum and ability
to assess in a timely manner. The indicators were mapped to some faculty-wide courses, including the
common first year, and faculty-wide middle year courses in economics and communication, and to
program specific courses. Appropriate assessment tools were determined by individual course
instructors. Finally, grading resources like rubrics were created for each of the course assignments that
were used to assess the indicators.
In a parallel process, a survey of instructors in the faculty was conducted to look at the alignment
between the curriculum and the attributes.
After the data was gathered and collated, statistical analysis was performed and provided to the faculty
and programs to interpret and create an improvement plan.
Section 3.1 Part A: Demonstration of student attributes
Instructions for criterion 3.1
A. Describe the processes that are being or are planned to be used to demonstrate that
students possess the attributes required by criteria 3.1.1 to 3.1.12, and that the results are
being applied to the further development of the program. The description should include:
a) a set of indicators that describe specific abilities expected of students to demonstrate
each attribute
b) where attributes are developed and assessed within the program, such as via courses,
co-curricular activities, co-op positions (e.g., a curriculum map). Ensure that courses
described are easily identified in the course descriptions presented on the Course
Information Sheets.
5
See, e.g. http://www.cou.on.ca/related-sites/the-ontario-universities-council-on-qualityassura/policies/pdfs/quality-assurance-framework---guide-may-2011.aspx
24
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
c) how the indicators were or will be assessed. This could be based on assessment tools
that include, but are not limited to, reports, oral presentations, observed laboratory
performance, assignments, exams, standardized tests, student/alumni surveys, peer
reviews, self-assessments, and work term experience/reports which may be used in
whole or in part. For example, evidence may be demonstrated via a single question
embedded on an exam or in the entirety of one or more of the assessment tools. Provide
samples of work or other documents deemed to exceed, meet, and fall below the
expected level of performance. How the samples of work or other documents are used to
assess indicators should be clearly explained.
d) evaluation of the data collected including analysis of student performance relative to
program expectations
e) discussion of how the results will be used to further develop the program
f)
a description of the ongoing process used by the program to assess and develop the
program as described in (a)-(e) above.
The response to part A above must specifically address each graduate attribute in criteria
3.1.1 to 3.1.12.
Purpose and outcomes
The Mission statement for the Faculty of Engineering and Applied Science is:
We educate engineering students for leadership and citizenship in a global society through high quality,
technically rigorous engineering programs.
The Faculty’s perspective was used to develop indicators faculty-wide for first year, middle years, and
graduating year; these are shown in Appendix A to give a perspective on the general approach being
taken. The tables below show the indicators that were assessed in the 2010-2011 academic year, and
the courses in which they were assessed.
25
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Faculty-wide first year indicators
Sec.
Attribute
Code
3.01
Knowledge base
3.01-FY1
3.01
Knowledge base
3.01-FY2
3.01
Knowledge base
3.01-FY3
3.02
Problem analysis
3.02-FY1
3.02
Problem analysis
3.02-FY2
3.02
Problem analysis
3.02-FY3
3.02
Problem analysis
3.02-FY4
3.04
Design
3.04-FY1
3.04
3.04
Design
Design
3.04-FY2
3.04-FY3
3.04
Design
3.04-FY4
3.04
Design
3.04-FY5
3.04
Design
3.04-FY6
3.04
Design
3.04-FY7
3.06
Teamwork
3.06-FY1
3.06
Teamwork
3.06-FY2
3.06
Teamwork
3.06-FY3
3.06
Teamwork
3.06-FY7
3.07
Communications
3.07-FY3
3.08
3.08-FY2
3.09
Professionalism
Impact of
engineering
3.10
Ethics and equity
3.10-FY2
3.10
Ethics and equity
3.10-FY3
3.09-FY4
Indicator
Create mathematical descriptions or
expressions to model a real-world problem
Select and describe appropriate tools to solve
mathematical problems that arise from
modeling a real-world problem
Use solution to mathematical problems to
inform the real-world problem that gave rise to
it.
Identifies known and unknown information,
uncertainties, and biases when presented a
complex ill-structured problem
Creates process for solving problem including
justified approximations and assumptions
Selects and applies appropriate quantitative
model and analysis to solve problems
Evaluates validity of results and model for error,
uncertainty
Adapts general design process to design
system, component, or process to solve openended complex problem.
Accurately identifies significance and nature of
a complex, open-ended problem
Identifies customer and user needs
Gathers and uses information from appropriate
sources, including applicable standards,
patents, regulations as appropriate.
Produces a variety of potential design solutions
suited to meet functional specifications
Performs systematic evaluations of the degree
to which several design concept options meet
project criteria
Compares the design solution against the
problem objective
Recognizes a variety of working and learning
preferences
Applies principles of conflict management to
resolve team issues
Assumes responsibility for own work and
participates equitably
Exercises initiative and contributes to team
goal-setting
Summarizes and paraphrases written work
accurately with appropriate citations
Demonstrates punctuality, responsibility and
appropriate communication etiquette
Incorporates sustainability considerations in
decision making (societal and ecological)
Identifies and iterates items from the
professional codes of conduct
Describes ethical issues and how they affect
the individual, the company and the public
Course
APSC-171
APSC-171
APSC-171
APSC-100
APSC-100
APSC-100
APSC-100
APSC-100
APSC-100
APSC-100
APSC-100
APSC-100
APSC-100
APSC-100
APSC-100
APSC-100
APSC-100
APSC-100
APSC-100
APSC-100
APSC-100
APSC-100
APSC-100
26
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
3.11
Economics
3.11-FY1
3.11
Economics
3.11-FY2
3.11
Economics
3.11-FY3
3.12
Lifelong learning
3.12-FY1
3.12
3.12
Lifelong learning
Lifelong learning
3.12-FY2
3.12-FY4
3.12
Lifelong learning
3.12-FY5
3.12
Lifelong learning
3.12-FY6
Plans and efficiently manages time and money
Establishes appropriate project scope, after
consultation with client, based on available
resources.
Plan maps out project with clear milestones
and delegation.
Uses information effectively, ethically, and
legally to accomplish a specific purpose,
including clear attribution of Information
sources.
Identifies a specific learning need or knowledge
gap.
Describes own learning style
Identifies appropriate technical literature and
other information sources to meet a need
Critically evaluates the procured information for
authority, currency, and objectivity.
APSC-100
APSC-100
APSC-100
APSC-100
APSC-100
APSC-100
APSC-100
APSC-100
Faculty-wide middle year indicators
Two attributes, communications and engineering economics, were assessed within the middle years of
the program as there are faculty-wide courses targeting these attributes within the current curriculum.
APSC-291 and -292 are second year courses in communications, each with 12 AU. APSC-221 is a course
in engineering economics and business taken by MME students, usually in their fourth year.
Indicators for these courses are shown below. Note that the code for middle year communications
follows the CDIO Syllabus convention, in contrast to other indictor codes. These codes are coursespecific.
Sec.
Attribute
3.07
Communications
3.07
Communications
Code
2.2.2
2.3.3a
3.07
Indicator
Course
Apply information search and
identification, with proper citations
Identify all relevant factors and the
dominant factors in the system
APSC291/292
Communications
APSC291/292
2.3.3b
3.07
Communications
3.07
Communications
APSC291/292
2.4.4
2.4.5a
Identify and Prioritize relevant factors
Compose logical arguments based on
supporting evidence
Assess the extent of one's presentation
abilities, and one's responsibility for selfimprovement to overcome important
weaknesses
APSC291/292
APSC291/292
27
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
3.07
Communications
2.4.5b
3.07
Communications
3.07
Communications
3.07
Communications
3.07
Communications
3.1.2
3.2.1a
3.2.2a
Communications
3.2.2b
3.07
Communications
3.07
Communications
3.2.2c
3.2.2d
3.07
Communications
3.07
Communications
3.2.3a
3.2.3b
3.07
Communications
3.07
Communications
3.2.3c
3.2.3d
3.07
Communications
3.2.3e
3.07
Communications
3.07
Communications
3.2.3f
3.2.3g
3.07
Communications
3.07
Communications
3.2.5
3.2.6a
3.07
3.07
Communications
Communications
APSC291/292
APSC291/292
APSC291/292
APSC291/292
3.2.1b
3.07
Assess the extent of one's team
communications skills, and one's
responsibility for self-improvement to
overcome important weaknesses
Demonstrate effective communication in a
small team (active listening, collaboration,
providing and obtaining information)
Demonstrate a communications strategy
that considers the needs and character of
the audience
3.2.6b
3.2.6c
Identify and Summarize key content
Compose with the appropriate structure
and relationship amongst ideas
Create logical, persuasive arguments in
your cover letter
Demonstrate conciseness, crispness,
precision and clarity of language
Show relevant, credible, accurate
supporting evidence in your resume
Compose clear, precise, sequential
description of events or instructions
Demonstrate writing with coherence and
flow
Demonstrate writing with correct spelling,
punctuation and grammar
Use appropriate formatting for a memo,
including to: and cc:
Use appropriate formatting for a short
report, including headings and figure
captions
Use appropriate formatting for an
executive summary
Use prescribed formatting for a letter and
resume
Demonstrate construction of block
diagrams
Prepare a presentation with appropriate
language, style, timing and flow
Prepare and deliver a presentation with
appropriate language, style, timing and
flow
Prepare presentation materials with
APSC291/292
APSC291/292
APSC291/292
APSC291/292
APSC291/292
APSC291/292
APSC291/292
APSC291/292
APSC291/292
APSC291/292
APSC291/292
APSC291/292
APSC291/292
APSC291/292
APSC291/292
28
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
3.07
Communications
3.2.6d
3.07
Communications
4.3.1
3.11MY1
3.11
Economics
3.11
Economics
3.11
Economics
3.11
Economics
3.11MY2
3.11MY3
3.11MY4
3.11
Economics
3.11MY5
appropriate language, style, timing and
flow
Use appropriate nonverbal
communications (gestures, eye contact,
poise)
Apply the language/format of goals and
requirements
Gather appropriate information, categorize it,
and determine the economic attractiveness of
an engineering project
Measure the financial impact of risks that are
associated with the engineering project and
consider the risk and return relationship as a
component of determining economic
attractiveness
Plans a project from scoping to completion,
including scheduling
Plans for the management of risks and changes
required for project
Describes new enterprise formation, business
planning and the economic attractiveness of the
enterprise.
APSC291/292
APSC291/292
APSC-221
APSC-221
APSC-221
APSC-221
APSC-221
Graduating year
MME specific information for the graduating year is detailed in section 3.1 of the main body of the
questionnaire.
Curriculum mapping
In addition to mapping indicators to courses as shown above, a survey of instructors in the faculty was
conducted to look at the alignment between the curriculum and the graduate attributes.6 Each course
instructor was asked to respond to a set of questions about each course they teach. The questions were
as follows:
a. What methods of instruction do you use in your course? (respondents select from list
provided)
b. What methods of assessment are used in your course? (respondents select from list
provided)
6
Queen`s was generously allowed to pilot the CurricKit curriculum mapping tool created at the University of
Guelph for this survey. Information is at: http://currickit.wikispaces.com/
29
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
c. Which of the Engineering Graduate Attributes are developed in your course?
(respondents select from list provided)
d. What level of complexity/depth is expected for each of the identified Engineering
Graduate Attribute? (respondents select one of three levels)
e. Please specify the ways that each grad attribute is Taught in your course. (respondents
select from the list in part (a))
f. Please specify the ways that each grad attribute is Assessed in your course.
(respondents select from list in part (b))
g. What is the mark and workload distribution over the duration of the semester?
h. Are optional assignments or optional weighting of assignments made available to your
students? If so, please specify.
i. Do you have any general comments?
The plot below shows the results across all of engineering. Knowledge base and problem analysis are the
most commonly developed and assessed attributes, and economics, ethics and equity, lifelong learning,
and professionalism are the least. This distribution agrees with student perceptions as discussed later in
this report.
30
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
The plot below shows the expected complexity for the attributes as reported by course instructors.
Again, knowledge base and problem analysis attributes have the greatest number of courses reporting
an expectation for work-ready performance.
Within only faculty-wide courses (APSC), the distribution is shown below. The courses include all of the
core first year courses, second and third year communications and economics courses. The prevalence
of professional skill development is under-represented in the plot below, as many of the skills are
developed in a course that is equivalent to three typical term-length courses (APSC-100, which has 132
AU).
31
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
A rough mapping of the attributes to courses in the existing faculty wide curriculum is shown below.
Details of the program specific course mapping for Mechanical and Materials Engineering is provided in
Section 3.1 of the main body of the questionnaire. Section 3.1B discusses proposals for curriculum
development that will expand the development and assessment of graduate attributes.
Attribute
Knowledge base
Problem analysis
Investigation
Design
Engineering tools
Communication
Individual and team work
Impact of engineering on society
Ethics and equity
Professionalism
Economics and project
management
Lifelong learning
Developed
Extensively
Extensively
APSC-100,
APSC-100,
APSC -100, 142, 161
APSC-100, 291, 292
APSC-100
APSC-100, 151
APSC-100
APSC-100
APSC-100, 221
Assessed
(not this year)
APSC-100
(not this year)
APSC-100,
(not this year)
APSC-100,291
APSC-100
APSC-100
APSC-100
APSC-100
APSC-100,221
APSC-100
APSC-100
32
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Assessment and evaluation
Faculty-wide First year
Historically many of the knowledge and skills now targeted by the graduate attributes have been
developed in the common first year of the engineering program. As a result, all 11 of the 12 attributes
were assessed in two first year course courses, in addition to the assessment process in the graduating
year. Mathematical knowledge base for engineering was assessed in APSC-171, Calculus I, and ten of the
professional skill attributes were assessed in APSC-100, Engineering Practice. Details and sample work
may be found in the example material provided for each course.
The sections below show the indicators, assessment tools, and data gathered from these courses in the
2010-2011 academic year.
Knowledge base for Engineering
Three specific indicators were assessed in the context of introductory mathematical knowledge. These
reflect the three key course objectives of the introductory calculus course APSC-171, Calculus I.
Code
3.01-FY1
3.01-FY2
3.01-FY3
Indicator
Create and/or select mathematical descriptions or expressions for simple
real-world problems involving rates of change and processes of
accumulation
Course
APSC-171
Select and describe appropriate tools to solve mathematical problems
that arise from modeling a real-world problem
Use solution to mathematical problems to inform the real-world problem
that gave rise to it.
APSC-171
APSC-171
These indicators are also appropriate for the problem analysis graduate attribute, but since they are
specific to mathematics they are listed under the knowledge attribute. The final exam in the course was
designed to allow these three indicators to be assessed individually by specific questions or subquestions on two tests and the final exam. A copy of these tests is shown in the course material for
APSC-171. The material below shows results from specific questions on tests and the exam that were
developed to evaluate student performance on these indicators.7
7
The material in this section was adapted from a report prepared by the course coordinator of APSC-171, Prof. L.
Jonker.
33
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Indicator 1: Create and/or select mathematical descriptions or expressions for simple real-world
problems involving rates of change and processes of accumulation
The context for this question involved calculating intersection of two trajectories. One of the important
ways in which mathematics connects to the "real" world is through its applications to geometric space
and dynamics. This connection was explored especially during the first few weeks of the course and was
tested especially on the first test. Thus, the third and fourth pages of the first test consist of a question
that gives students the equations of the trajectories of two particles and then asks them to determine
whether the particles will collide, and (separately) whether their trajectories cross, and at what point of
time the velocity of the second particle is in a horizontal direction. This material is difficult for the
students, especially for those who do not come from Ontario high schools, where the curriculum
includes a unit on linear algebra and vectors. The marks for these pages show this, for they were quite
low, averaging 3.8 out of 6 (63%).
Figure 2
Indicator 2: Select and describe appropriate tools to solve mathematical problems that arise from
modeling a real-world problem
The context for this was differentiation similar to that required in the high school curriculum. Generally,
we find that students are well prepared to negotiate the second indictor, and the test and exam
questions that test this are done very well by most. For example, in Test 1, Question 1, students were
asked to calculate the derivatives of mathematical functions involving logarithms, exponents, powers.
These problems test students' knowledge of the various rules of differentiation, as well as their
knowledge of logarithm and exponential laws, and of the derivatives of trigonometric functions. Some of
that (particularly the differentiation rules and the laws of exponents and logarithms) was on their high
school curriculum. It shows, as the average mark (out of 11) was 8.9 (81%).
34
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Figure 3
Another question on the first test that was purely focused on mathematical knowledge and problem
analysis was Question 7. The first part of this question tested students' understanding of inverse
trigonometric functions, while the second part tested their understanding of implicit differential
equations. This question was done less well because both were entirely new topics to the bulk of the
students, and the first one, particularly, involved some subtlety that students tend to miss. The average
mark on this question was 3.3 out of 6 (55%).
35
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Figure 4
Indicator 3: Use solution to mathematical problems to inform the real-world problem that gave rise to
it.
One of the challenges in teaching calculus is that, whereas students know how to differentiate when
they come out of high school, their understanding of what a derivative is in practice is not as secure as it
might be. We consider one of the main goals of the course to be that they should gain a better, more
practical, sense of what a derivative represents. One of the ways in which we do this is by spending
quite a bit of time asking students to interpret differential equations (mostly without teaching them to
solve a differential equation!) and to translate statements of physical laws into mathematics (inevitably
resulting in a differential equation). One of the questions on Test 1 (Question 6) probes this
understanding, and asks students to do this sort of translation, together with a linear approximation
(the essence of the derivative):
Question 6 on Test 1: This problem concerns atmospheric pressure near the earth. A rough model is
that:
“The rate of change in pressure with height is proportional to the pressure at that height."
(a) Write down the mathematical equation that expresses this relationship. Be sure and label all your
variables and constants as clearly as possible, and do not use any variables that you do not need. Finally,
state whether the constants in your equation are positive or negative.
(b) The constant of proportionality in part (a) has an absolute value of approximately 0.124 /km. At a
height of 5:6 km above sea level, the pressure is approximately 0.5 atm (atmospheres). What is the rate
of change of pressure at this height?
36
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
(c) Use your answer from (b) and linear approximation to estimate the pressure at a height of 6 km
above sea level.
The average on this question was 4.9 out of 7 (70%). The histogram for the marks indicates that some
students continue to find this exercise difficult.
Problem analysis, Design, Communications, etc.
Ten of the professional skill attributes were assessed in APSC-100, Engineering Practice (the
Investigation attribute was not assessed this year, but will be in future years). These were assessed in
integrated assignments which covered multiple attributes. The tables below show the deliverables by
which indicators were assessed. Details about each deliverable expectation are described in the APSC100 course material.
APSC-100 is divided into three modules: Module 1 focuses on complex problem solving and numeric
computation (MATLAB is used as the tool); Module 2 focuses on investigation and design of
experiments; Module 3 centres around a semester long client-based team design project. Modules 1
and 3 were used to assess attributes this year.
Indicators used in APSC100 Module 1 (Fall and winter)
Course Deliverable
Assignment: Learning styles and professional
communication
Handed in: Sept 2010
Assignment: Problem solving and modeling activity 1
Indicators
FEAS - 3.12-FY4: Describes learning style
FEAS - 3.02-FY1: Identify problem
FEAS - 3.02-FY2: Create process
37
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
(PSM1: Mersey Venture Elevator Failure)
Handed in: Oct 2010
Assignment: Problem solving and modeling activity 2
(PSM2: Water filter)
Handed in: Nov 2010
Assignment: Problem solving and modeling activity 4
(PSM4: nicotine diffusion)
Handed in: Feb 2010
FEAS - 3.02-FY3: Select model
FEAS - 3.02-FY4: Evaluate solution
FEAS - 3.10-FY3: Describes ethical issues
FEAS - 3.02-FY1: Identify problem
FEAS - 3.02-FY2: Create process
FEAS - 3.02-FY3: Select model
FEAS - 3.02-FY4: Evaluate solution
FEAS - 3.10-FY3: Describes ethical issues
FEAS - 3.02-FY2: Create process
FEAS - 3.02-FY3: Select model
FEAS - 3.02-FY4: Evalute solution
Nicotine Diffusion - Ethical dilemma
Indicators used in APSC100 Module 3 (Winter)
In Module 3 student deliverables were assessed by three groups of people: each project had a faculty
sponsor (FS), an upper year student serving as a project manager (PM), and independent academic
assistants who evaluated written and oral communications. In the data below, the evaluators are
indicated by FS for faculty sponsors, PM for project managers, and AA for academic assistants.
Course Deliverables
Midterm Individual Assessment
Description
Assessment on Students’ individual performance to the middle
of semester
- Conflict management 3.06-FY2
- Responsibility 3.06-FY3
- Initiative 3.06-FY7
- Professional behavior 3.08-FY2
Final Individual Assessment
Assessment on Students’ individual performance to the end of
semester
- Conflict management 3.06-FY2
- Responsibility 3.06-FY3
- Initiative 3.06-FY7
- Professional behavior 3.08-FY2
Team Assignment 1: Project
definition (TA1)
The first report done by groups. Contents focus on the project
definition and team organization.
- 3.04-FY2: Identify problem
- 3.04-FY3: Identify needs
- 3.12-FY2: Identifies learning need
- 3.12-FY5: Identifies appropriate literature
- 3.06-FY1: Learning preferences
Team Assignment 2: Project scope
and information (TA2)
A brief report outlines team project plan and summarizes the
information research.
- 3.11-FY3: Project plan
- 3.11-FY2: Project scope
- 3.07-FY3: Summarizes accurately
- 3.12-FY6: Evaluates information
- 3.12-FY1: Ethical information use
- APSC-100 Comm: Conciseness
38
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Proposal Report Draft
In order to achieve a formal and professional proposal report, a
draft is required. Rubrics here focus on the grammar,
formatting and reference.
- 3.07-FY3: Summarizes accurately
- APSC100 Comm: Purpose and style
- APSC100 Comm: Coherence and Format
- APSC100 Comm: Graphical communications
- APSC100 Comm: Correctness
Proposal Report
The formal/final version of proposal report. Rubrics here are
different than the one above.
- 3.04-FY2: Identify problem
- 3.11-FY3: Project plan
- 3.04-FY5: Potential solutions
- 3.04-FY6: Decision making
- 3.07-FY3: Summarizes accurately
- APSC100 Econ: Proposal economic analysis
- APSC100M3 Proposal: Overall proposal
Proposal Presentation
A presentation on the proposal of the project.
- APSC-100 Oral Comm: Organization
- APSC-100 Oral Comm: Content
- APSC-100 Oral Comm: Slides
- APSC-100 Oral Comm: Speakers
- APSC-100 Oral Comm: Purpose and style
Final Report
The Formal final report which summarizes students’ term
efforts
- 3.04-FY4: Gathers info
- 3.11-FY1: Manage time and money
- 3.04-FY1: Uses process
- 3.09-FY4: Sustainability in decisions
- 3.04-FY7: Compares solution
- APSC 100 Final Report – Implementation
- APSC 100 Final Report - Overall Written Communication
Final Presentation
A presentation on the final solution/implementation of the
project
- APSC-100 Oral Comm: Organization
- APSC-100 Oral Comm: Content
- APSC-100 Oral Comm: Slides
- APSC-100 Oral Comm: Speakers
- APSC-100 Oral Comm: Purpose and style
The Table below summarizes which assignments were used to assess each indicators across the entire
course.
39
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
X
X
X
X
X
X
X
X
Final Report
Proposal Report
Proposal Report Draft
Team Assignment 1: Project
Definition (TA1)
Team Assignment 2: Project Scope
and Information (TA2)
Final Individual Assessment
Midterm Individual Assessment
Module 3
Problem Solving and Modeling
Activity 4 (PSM4)
Problem Solving and Modeling
Activity 2 (PSM2)
Indicators
FEAS – 3.02-FY1
FEAS – 3.02-FY2
FEAS – 3.02-FY3
FEAS – 3.02-FY4
FEAS – 3.04-FY1
FEAS – 3.04-FY2
FEAS – 3.04-FY3
FEAS – 3.04-FY4
FEAS – 3.04-FY5
FEAS – 3.04-FY6
FEAS – 3.04-FY7
FEAS – 3.06-FY1
FEAS – 3.06-FY2
FEAS – 3.06-FY3
FEAS – 3.06-FY7
FEAS – 3.07-FY3
FEAS – 3.08-FY2
FEAS – 3.09-FY4
FEAS – 3.10-FY3
FEAS – 3.11-FY1
FEAS – 3.11-FY2
FEAS – 3.11-FY3
FEAS – 3.12-FY1
FEAS – 3.12-FY2
FEAS – 3.12-FY4
FEAS – 3.12-FY5
FEAS – 3.12-FY6
Assignment: Learning Styles and
Professional Communication
Problem Solving and Modeling
Activity 1 (PSM1)
Module 1
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
40
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Some indicators were added specifically to APSC-100, beyond the faculty-wide indicators created and
shown in Appendix 3.1A. Most indicators were scored on a four-level rubric, with a “1” corresponding to
expectation not being demonstrated, a “2” meaning marginal (threshold) performance, “3” for meeting
expectations (target performance), and 4 for exceeding expectations. The rubrics were given to
students when the assignments were created.
Each deliverable, by students or team, was graded using the rubric. The total grade for the deliverable
was used to compute the mark on the assignment, and the score for each indicator was stored for the
purpose of analyzing students’ ability to meet expectations. In the next section student performance is
described.
Sample rubrics for some of the deliverables are shown below. More detail is shown in the set of APSC100 course materials. Rubric descriptors for each indicator on each deliverable are shown in the section
with student scores.
41
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
APSC-100 (2010 -2011) MEA1: Mersey Venture Elevator Failure
Information summary
3.02-FY1: Identify problem
1
(not demonstrated)
Information not identified
properly, no information,
or information copied
from assignment.
2
(marginal)
Some important
information or biases not
identified, or
trivial/incorrect
information included.
3
(meets expectations)
Identifies known and
unknown information,
uncertainties, and biases.
4
(outstanding)
Meets expectations plus:
No or inadequate
process
Process identified
misses some important
factors; some
assumptions left
unidentified or
unjustified.
Creates justified process
for solving problem,
suppored by information.
Meets expectations plus:
No analysis, or
model/analysis selected
is inappropriate
Model selected, some
errors in analysis or
inappropriate
assumptions.
Selects and applies
appropriate quantitative
model and MATLAB
analysis to solve problems,
using reasonable
approximations and
assumptions.
Meets expectations plus:
No evaluation of
solution.
Superficial evaluation of
solution
Evaluates validity of results
and model for error,
uncertainty
Meets expectations plus:
Makes supported
recommendations to
prevent future failures;
describes implications to
stakeholders.
Meets expectations plus:
Mark
(/4)
Includes information from
authoritative sources to
inform process, model,
and conclusions.
/4
Proposed process
3.02-FY2: Create process
Comprehensive process
model; comparison with
other possible approaches.
/4
Model
3.02-FY3: Select model
Model results and
conclusions
3.02-FY4: Evalute solution
Recommendations
3.10-FY3: Describes
ethical issues
No or trivial
recommendations
Some important
recommendations and
implications missed.
Authoritative research
used to defend
assumptions and
approximations made
/4
Evaluates conclusions and
presents potential
improvements.
Information, model, and
conclusions used to make
recommendations
supported by authoritative
information.
/4
/4
42
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
APSC-100 (2010 -2011) MEA2: Clay water filter
Information summary
3.02-FY1: Identify problem
Proposed process
3.02-FY2: Create process
Analysis
3.02-FY3: Select model
Model results and
conclusions
1
(not demonstrated)
Information not
identified properly, no
information, or
information copied
from assignment.
2
(marginal)
Some important
information or biases
not identified, or
trivial/incorrect
information included.
3
(meets expectations)
Identifies known and
unknown information,
uncertainties, and biases.
No or inadequate
process
Process identified
misses some important
factors; some
assumptions left
unidentified or
unjustified.
Creates justified process
for solving problem,
including concept map,
that includes T,S,E,F
factors.
Model selected, some
errors in analysis or
inappropriate
assumptions.
Uses analysis to make
decision on multiple
factors using reasonable
approximations and
assumptions.
Meets expectations plus:
Authoritative research used
to defend assumptions and
approximations made
Evaluates validity of
results and model for
error, impact of
uncertainty.
Meets expectations plus:
Evaluates conclusions and
presents potential
improvements.
/4
Describes potential side
effects of design.
Meets expectations plus:
Potential side effects
informed by model, analysis,
and conclusions, supported
by authoritative information.
/4
No analysis, or
model/analysis
selected is
inappropriate
No evaluation of
solution.
Superficial evaluation of
solution
3.02-FY4: Evalute solution
Side effects
3.10-FY3: Describes ethical
issues
No or trivial
recommendations
Some important
recommendations and
implications missed.
4
(outstanding)
Meets expectations plus:
Includes information from
authoritative sources to
inform process, model, and
conclusions.
Mark
(/4)
/4
Meets expectations plus:
Comprehensive process
model; comparison with
other possible approaches.
/4
/4
43
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
APSC-100 (2010-2011)
Purpose and style
Coherence and
Format
Sequence,
transitions,
formatting
1
(not demonstrated)
Unclear purpose. Tone and
style inappropriate.
Audience needs not met.
Poorly organized; rambling,
lacks unity; Inconsistent
writing/formatting; lacks
transitions; many gaps or
redundancies. Handwritten
material.
Graphical
communications
Figures and tables not
related to text. Improper
referencing of figures. List of
tables/figures missing.
Correctness
Numerous grammar errors;
disorganized sentence
structure; inappropriate
units or significant digits;
acronyms not defined
Erroneous or incomplete
conclusions drawn from
sources, or inappropriate
quality or quantity of
background information.
Information usage
3.07-FY3: Summarizes
accurately
Overall:
Written Communications Rubric
2
(marginal)
Some aspects of presentation
don’t achieve purpose.
Organization sometimes
unclear; some unsuitable
sections, minor gaps or
redundancies, minor formatting
problems; some wordy
expressions . "This is/There is/It
is" …
Some figures and tables not
discussed in text. Figure/table
captions missing. Incomplete/
inconsistent list of tables/
figures.
Some grammar/ punctuation
errors; spelling; incomplete
sentences.
Sources summarized with minor
misconceptions.
3
(meets expectations)
Clear purpose is met.
Formal tone and style
appropriate to audience
Organized, appropriate
sections, uniformly and
correctly formatted; little
irrelevant information.
4
(outstanding)
Professional tone and style.
Authoritative and
convincing
Focused, logically organized;
skillful and varied
transitions. Professionally
formatted. No irrelevant
information
Figures and tables referred
to in text, captioned.
Appropriate lists of
figures/tables.
Figures and tables
professionally formatted,
integrated into text,
complementing text
Relatively free of grammar/
punctuation errors; units
specified; acronyms and
symbols defined
No typos or grammatical
errors; attractively and
professionally formatted.
Summarizes and
paraphrases appropriate
sources accurately with
appropriate citations
Multiple authoritative,
objective, reliable sources
used; cited and formatted
properly
Mark
(/4)
/4
/4
/4
/4
/4
/
44
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
APSC-100 (2010-2011)
Purpose & Style
Organization
Sequence;
transitions. Methods
to emphasize content
(illustrations,
repetition)
Content
Oral Communications Rubric
1
(not demonstrated)
Unclear purpose. Tone
and style inappropriate.
Audience needs not
met.
Insufficient content;
poorly organized;
rambling; not
understandable
Minimal content that
does not demonstrate
appropriate progress on
project.
Slides
Mechanics,
acronyms, units,
graphics
Numerous typos and
grammatical errors;
unprofessional and
inconsistent formatting;
figures and tables
unclear or unrelated
Speakers
Volume, speed,
cadence, English
usage; questions.
Use of aids
Hard to understand/hear
the speaker; speaking
far too fast/slow;
questions not
satisfactorily answered;
reading from notes
Team Number:
2
(marginal)
Some aspects of
presentation don’t
achieve purpose.
3
(meets expectations)
Clear purpose is met
Formal tone and style
appropriate to audience
Organization sometimes
unclear; some
unsuitable sections;
disconnected. Some
useful information
missing.
Generally appropriate
content but some
information missing.
Organized; appropriate
information; generally
smooth transitions
between topics
Some typos; some
inconsistency in
formatting; some
acronyms and symbols
not defined;
inappropriate use of
figures/tables
Generally understood
with some volume, pitch
and speed issues; some
questions answered
incorrectly; reading off of
slides
Date:
Evaluator:
4
(outstanding)
Professional tone and
style
Authoritative and
convincing
Slides logical, easy to
follow; detailed,
impressive depth of
thought; excellent
transitions between topics
Appropriate content
covering project
background, scope,
relevant information,
design options, and work
completed.
Relatively free of typos and
grammatical errors
Slides uniformly formatted
Acronyms and symbols
defined; appropriate
graphics
Excellent content and
evidence of detailed
thought and analysis.
Understandable,
appropriate volume, pitch
and speed; questions
generally answered
correctly; aids used
properly
Professional variation in
volume, pitch, speed; very
knowledgeable and clear
when handling questions;
aids used very well
Mark
(/4)
Slides free of typos and
grammatical errors; slides
uniformly formatted and
professional; material and
graphics support main
ideas
Overall:
45
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
APSC-100 Module 3: Final team evaluation
1
(not demonstrated)
2
(marginal)
3
(meets expectations)
4
(outstanding)
Mark
Information Management No significant
information used, not
3.04-FY4: Gathers info
cited; blatant
plagiarism.
Insufficient usage; improper
citations.
Gathers and uses information
from appropriate sources,
including applicable standards,
patents, regulations as
appropriate, with proper citations
Uses information from multiple
authoritative, objective, reliable
sources; cited and formatted
properly
Project Management
No useful timeline or
budget described;
poorly managed
project; safety issues
Poor timeline or budget;
infrequent meetings; minor
safety problems
Plans and efficiently manages
time and money; team effectively
used meetings; safety
considerations are clear
Efficient, excellent project plan
presented; detailed budget;
potential risks foreseen and
mitigated
No discussion of
design process.
Generic design process
described.
Describes design process used to
design system, component, or
process to solve open-ended
complex problem.
Comprehensive design process
described, with appropriate
iterations and revisions based on
project progress
/4
No consideration of
these factors.
Factors mentioned but no
clear evidence of impact on
decision making.
Incorporated appropriate social,
environmental, and financial
factors in decision making
Well-reasoned analysis of these
factors, with risks mitigated where
possible
/4
Implementation
Insufficient
implementation
Sufficient implementation but
some opportunities not
taken, or feedback at
proposal not incorporated in
implementation
Appropriate effort, analysis,
Outstanding implementation
and/or construction demonstrated
to implement product, process, or
system
/4
Evaluation
No evaluation of
design solution
Some factors missed in
evaluating design solution
Compares the design solution
against the project objectives and
functional specifications, providing
qualitative evaluation where
appropriate
/4
Poorly constructed
report
Some organization
problems, minor formatting
problems, redundancy,
spelling grammar/errors
Report achieves goal using formal Professional tone, convincing
tone, properly formatted,
argument, authoritative, skillful
concisely written, appropriate use transitions
of figures, few spelling/grammar
errors
3.11-FY1: Manage time
and money
Design process
3.04-FY1: Uses process
Social, environmental,
and financial factors
/4
/4
3.09-FY4: Sustainability in
decisions
3.04-FY7: Compares
solution
Written communication
Comprehensive evaluation of
design solution, with welldefended recommendations for
future work or implementation
/4
46
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Attribute: Problem analysis
The plots below show student scores on indicators of problem analysis. In these plots, and plots below,
the text in parentheses shows the abbreviation of the deliverable by which the indicator was assessed.
Abbreviations are shown in the list of indicators above. The table below shows the levelled descriptors
used by graders to score student work on these indicators.
The problem analysis attribute was identified as one where students fail to meet expectations more
than others at the first year level. The indicators on which more than 20% of the students fell below the
target expectation were:


3.02-FY1: Identify problem (twice)
3.02-FY4: Evaluate solution (three times)
Figure 5
1 - Not Demonstrated
3.02 - FY1: Identifies known and Information not
unknown information,
identified properly,
uncertainties, and biases when no information, or
presented a complex illinformation copied
structured problem
from assignment
Threshold
Target
2 - Marginal
3 - Meets Expectations
Some important
information or biases not
Identifies known and unknown
identified, or
information, uncertainties,
trivial/incorrect information and biases
included
3.02 - FY2: Creates process for
solving problem including
justified approximations and
assumptions
Process identified misses
Creates justified process for
some important factors;
solving problem, suppored by
some assumptions left
information.
unidentified or unjustified.
Meets expectations PLUS:
Comprehensive process model;
comparison with other possible
approaches
No analysis, or
3.02 - FY3: Selects and applies
model/analysis
appropriate quantitative model
selected is
and analysis to solve problems
inappropriate
Selects and applies approriate
quantitative model and
Model selected; some
MATLAB analysis to solve
errors in analysis or
problems, using reasonable
inappropriate assumptions
approximations and
assumptions
Meets expectations PLUS:
Authoritative research used to
defend assumptions and
approximations made
3.02 - FY4: Evaluates validity of
results and model for error,
uncertainty
Superficial evaluation of
solution
No or inadequate
process
No evaluation of
solution
Evaluates validity of results
and model for error,
uncertainty
4 - Outstanding
Meets expectations PLUS:
Includes information from
authoritative sources to inform
process, model, and conclusions
Meets expectations PLUS:
Evaluates conclusions and
presents potential improvements
47
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Design
Plots of performance scores on design indicators are shown below. Generally performance on these was
acceptable. However, given the large percentage of students scored as “Exceeding expectations”, the
wording of the indicators needs to be reviewed to ensure that the expectations are appropriate.
Figure 6
1 - Not Demonstrated
3.04 - FY1: Adapts general
design process to design
No discussion of
system, component, or process design process.
to solve open-ended complex
problem.
Project definition is
3.04 - FY2: Accurately identifies
loosely defined.
significance and nature of a
Difficulty narrowing
complex, open-ended problem
goal of project
3.04 - FY3: Identifies customer
and user needs
3.04 - FY4: Gathers and uses
information from appropriate
sources, including applicable
standards, patents, regulations
as appropriate.
Threshold
2 - Marginal
Generic design process
described.
Target
3 - Meets Expectations
4 - Outstanding
Describes design process used
to design system, component,
or process to solve openended complex problem.
Comprehensive design process
described, with appropriate
iterations and revisions based on
project progress
Thoroughly identifies and
Accurately identifies and
Repeats project description
describes the presented problem
describes the presented
as presented by client or
while being aware of human
problem building on the client
course
factors, resources constraints and
requirements
client needs
Statement is multidimensional
No description of
Identifies customer and user
Identifies some customer
showing constraints and potential
customer or user
needs; elaborates and explain
and user needs.
strengths. Is aware of potential
needs
how it will impact the project
bias from client
Gathers and uses information
No significant
from appropriate sources,
Uses information from multiple
information used, not Insufficient usage;
including applicable standards, authoritative, objective, reliable
cited; blatant
improper citations.
patents, regulations as
sources; cited and formatted
plagiarism.
appropriate, with proper
properly
citations
48
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Figure 7
1 - Not Demonstrated
3.04 - FY5: Produces a variety of
No description of
potential design solutions
potential design
suited to meet functional
solutions
specifications
3.04 - FY6: Performs systematic
evaluations of the degree to
which several design concept
options meet project criteria
No
comparison/analysis
of potential design
solutions
3.04 - FY7: Compares the design
No evaluation of
solution against the problem
design solution
objective
Threshold
2 - Marginal
Target
3 - Meets Expectations
4 - Outstanding
Some critical considerations Produces a variety of potential Applies creative approaches to
missed in generating
design solutions suited to
identify and develop alternative
potential design solutions meet functional specifications concepts and procedures
Contrasts potential design
solutions without analysis
Performs systematic
evaluations of the degree to
which several design concept
options meet project criteria
Uses technical knowledge,
mathematical models,
appropriate design tools and
client/user feedback to select
best solution
Some factors missed in
evaluating design solution
Compares the design solution
against the project objectives
and functional specifications,
providing qualitative
evaluation where appropriate
Comprehensive evaluation of
design solution, with welldefended recommendations for
future work or implementation
In addition to the indicators described above, two professors in the Faculty of Engineering and Applied
Science have administered a test of design process understanding at the beginning and end of their
courses for the past three years.8 APSC-100 was the context in first year, and an elective multidisciplinary project-based design course, APSC-381, was the context for the third year assessment. Since
students elect to take APSC-381 the results should not be considered representative across the third
year of all programs, but provide a point of comparison for first year students.
The test instructions were as follows:
Situation:
You are leading a team of engineers on a design project, and need to plan out how you would approach
a problem.
Instructions:
1) Select and circle one scenario from the box below.
Content for this was adapted from a published study: B. Frank, D. Strong, “Development of a Design Skill
Assessment Tool”, Canadian Engineering Education Association Conference 2010, Kingston, ON, available online
at: http://library.queensu.ca/ojs/index.php/PCEEA/article/view/3165/3103
8
49
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
2) Outline, on the back of this page, the specific steps you would take to ensure a successful design for
your selected scenario. Please ensure you:

Outline all special considerations and specific requirements associated with your selected
scenario.

Outline the design process you would follow to solve the problem, listing specific steps your team
would take. You do not need to create a specific design, sketch, etc.
The students selected one scenario from a list of three provided on each test. An “A” version, with 3 of
the scenarios, and a “B” version, with three different scenarios, were each given to half of the class as
the pre-assessment, and the versions switched for the post assessment. This way the average difficulty
of the two versions could be evaluated.
Student responses were evaluated against 6 commonly accepted stages of engineering design: problem
definition, preliminary design, conceptual design, detailed design, validation, and implementation. They
are also assessed for the overall process (iteration and appropriate order) and use of design tools.
The figures below show the distribution of “pre” and “post” subtotal scores on the for the groups of
interest. The largest gains occur in APSC381, as would be expected since it is an advanced 12-week
course on design tools and methodologies. APSC100 students also show significant gains in the two
deliveries. Very modest gains are apparent from the outreach group. Note that the distribution of
“post” scores for APSC-100 is higher than “pre” scores for APSC-381, possibly due to low knowledge
retention between first and third year, or limited design instruction between first year and winter
semester of third year.
The histograms for APSC-100 show students making gains in conceptual and preliminary design, which
include activities like idea generation, planning, comparing potential designs, and early prototype
construction. This reflects the emphasis in the course. Note that score changes corresponding to a major
tick on the x-axis are grouped with the lower interval.
The corresponding data for APSC-381 students show much larger gains are made in problem definition,
conceptual design, preliminary design, and detailed design. This reflects the nature of the course which
does not require students to implement their designs. An even more significant increase in score for use
of Tools can be seen students were noted to make reference to design tools in the post-course
assessment.
50
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
PRE
Figure 9: Change in score from in 2010 “pre” to
“post” for APSC100 students in the major
categories. Statistically significant changes at a
95% confidence level are indicated with an
asterisk.
POST
Figure 8: Distribution of scores in 2010 for pre (top)
and post (bottom) engineering activity. Boxes show
middle quartiles, dark line shows median score.
Figure 10: Change in score in 2010 from “pre” to
“post” for APSC381 students in the major categories.
Statistically significant changes at a 95% confidence
level are indicated with an asterisk.
51
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Scoring rubric used to assess design skill assessment scenarios.
52
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Teamwork
Plots of performance scores on teamwork indicators are shown below. Generally performance on these
was acceptable. However, for this year no descriptors were provided to graders on these indicators (as
shown below). Given the large percentage of students scored as “Exceeding expectations”, descriptors
will need to be provided to provide guidance to graders in future years.
Figure 11
Threshold
Target
1 - Not
Demonstrated
2 - Marginal
3 - Meets
Expectations
4 - Outstanding
3.06 - FY1: Recognizes a variety
of working and learning
preferences
Fail
Marginal
Good
Excellent
3.06 - FY2: Applies principles of
conflict management to resolve
team issues
Fail
Marginal
Good
Excellent
3.06 - FY3: Assumes
responsibility for own work and
participates equitably
Fail
Marginal
Good
Excellent
3.06 - FY7: Exercises initiative
and contributes to team goalsetting
Fail
Marginal
Good
Excellent
53
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Communications
In addition to the faculty wide indictors of the communications attribute, some course specific indicators
for communications were developed and assessed as well. Generally students were observed to do well
on formal oral presentation expectations but poorer on expectations for written work. Throughout the
year approximately 15% of the students struggled with summarizing and paraphrasing written work
accurately. At the end of the semester approximately 15% of the student teams were deemed to be
below target on written communications, as measured by their final report.
450
400
350
300
250
1
200
2
150
3
100
4
50
0
FEAS - 3.07-FY3 FEAS - 3.07-FY3 FEAS - 3.07-FY3 FEAS - 3.07-FY3
(PropReportPM) (PropReportFS) (PropReportAA) (T2Scope&Info)
APSC100
APSC100
APSC100
APSC100
Figure 12
3.07 - FY3: Summarizes and
paraphrases written work
accurately with appropriate
citations
Threshold
1 - Not Demonstrated
2 - Marginal
Insufficient content
to assess summary of Records information from
work. Summary
few resources. Misses
misinterprets
significant points of view.
researched material.
Target
3 - Meets Expectations
Summarizes and paraphrases
written work accurately.
4 - Outstanding
Synthesizes main ideas to
construct new concepts.
Summarizes the leading thoughts
in the field and gives a broader
picture of the problem.
54
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Figure 13
1 - Not
Demonstrate
d
Threshold
Target
2 - Marginal
3 - Meets
Expectations
4 - Outstanding
Report achieves
goal using formal
tone, properly
formatted,
concisely written,
appropriate use of
figures, few
spelling/grammar
errors
Professional tone,
convincing
argument,
authoritative, skillful
transitions
Concise prose, few
extraneous words.
Little irrelevant
information
Concise, meaningful
text, with no
irrelevant
information.
Appropriate use of
varied sentence
structures.
Final Report
Written
Communication
Poorly
constructed
report
Some
organization
problems, minor
formatting
problems,
redundancy,
spelling/grammar
errors
Team Assignment 2
Communication:
Conciseness
Wordy
expressions,
including
"This is/There
is/It is"
structures.
Many
redundancies
or gaps
Some irrelevant
information,
wordy phrases,
redundancies,
and/or gaps
55
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Figure 14
Threshold
Target
3 - Meets
Expectations
1 - Not
Demonstrated
2 - Marginal
Proposal Report Coherence &
Format: Sequence,
transitions,
formatting
Poorly prganized;
rambling, lacks
unity;
Inconsistent
writing/formattin
g; lacks
transitions; many
gaps or
redundancies.
Handwritten
material
Organization
sometimes
unclear; some
unsuitable
sections, minor
gaps or
redundancies,
minor
formatting
problems; some
wordy
expressions
"This is/There
is/It is"
Organized,
appropriate
sections,
uniformly and
correctly
formatted; little
irrelevant
information.
Focused, logically
organized, skillful
and varied
transitions.
Professionally
formateed. No
irrelevant
information
Proposal Report Correctness
Numerous
grammar errors;
disorganized
sentence
structure;
inappropriate
units or
significant digits;
acronyms not
defined
Some grammar/
punctuation
errors; spelling;
incomplete
sentences.
Relatively free of
grammar/
punctuation
errors; units
specified;
acronyms and
symbols defined
No typos or
grammatical errors;
attractively and
professionally
formatted.
4 - Outstanding
56
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Proposal Report Graphical
Communications
Figures and tables
not related to
text. Improper
referencing of
figures. List of
tables/figures
missing.
Some figures
and tables not
discussed in
text.
Figure/table
captions
missing.
Incomplete/
inconsistent list
of tables/
figures.
Figures and tables
referred to in text,
captioned.
Appropriate lists
of figures/tables.
Figures and tables
professionally
formatted,
integrated into text,
complementing text
Proposal Report Purpose & Style
Unclear purpose.
Tone and style
inappropriate.
Audience needs
not met.
Some aspects of
presentation
don’t achieve
purpose.
Clear purpose is
met. Formal tone
and style
appropriate to
audience
Professional tone
and style.
Authoritative and
convincing
450
400
350
300
250
1
200
150
2
100
3
50
4
0
Prop Pres Oral Final Pres Oral Final Pres Oral Prop Pres Oral
Comm: Content Comm: Content Comm: Content Comm: Content
(FS)
(FS)
(PM)
(PM)
APSC 100
APSC 100
APSC 100
APSC 100
Figure 15
57
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
1 - Not
Demonstrated
Oral
Communication Content
Oral
Communication Organization:
Sequence;
transitions.
Methods to
emphasize content
(illustrations,
repetition)
Oral
Communication Purpose & Style
Oral
Communication Slides: Mechanics,
acronyms, units,
graphics
Oral
Communication Speakers: Volume,
speed, cadence,
English usage;
questions. Use of
aids
2 - Marginal
3 - Meets
Expectations
Appropriate
content covering
project
background,
scope, relevant
information,
design options,
and work
completed.
4 - Outstanding
Minimal
content that
does not
demonstrate
appropriate
progress on
project.
Generally
appropriate
content but
some
information
missing.
Insufficient
content; poorly
organized;
rambling; not
understandable
Organization
sometimes
unclear; some
unsuitable
sections;
disconnected.
Some useful
information
missing.
Organized;
appropriate
information;
generally smooth
transitions
between topics
Slides logical, easy
to follow; detailed,
impressive depth of
thought; excellent
transitions between
topics
Some aspects of
presentation
don’t achieve
purpose.
Clear purpose is
met. Formal tone
and style
appropriate to
audience
Professional tone
and style.
Authoritative and
convincing
Relatively free of
typos and
grammatical
errors. Slides
uniformly
formatted.
Acronyms and
symbols defined;
appropriate
graphics
Slides free of typos
and grammatical
errors; slides
uniformly formatted
and professional;
material and
graphics support
main ideas
Understandable,
appropriate
volume, pitch and
speed; questions
generally
answered
correctly; aids
used properly
Professional
variation in volume,
pitch, speed; very
knowledgeable and
clear when handling
questions; aids used
very well
Unclear
purpose. Tone
and style
inappropriate.
Audience needs
not met.
Numerous
typos and
grammatical
errors;
unprofessional
and inconsistent
formatting;
figures and
tables unclear
or unrelated
Hard to
understand/hea
r the speaker;
speaking far too
fast/slow;
questions not
satisfactorily
answered;
reading from
notes
Some typos;
some
inconsistency in
formatting;
some acronyms
and symbols not
defined;
inappropriate
use of
figures/tables
Generally
understood with
some volume,
pitch and speed
issues; some
questions
answered
incorrectly;
reading off of
slides
Excellent content
and evidence of
detailed thought
and analysis.
58
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
450
400
350
300
250
200
150
100
50
0
1
2
3
4
Prop Pres Oral Prop Pres Oral Final Pres Oral Final Pres Oral
Comm:
Comm:
Comm:
Comm:
Organization (FS) Organization Organization (FS) Organization
(PM)
(PM)
APSC 100
APSC 100
APSC 100
APSC 100
Figure 16
400
350
300
250
200
1
150
2
100
3
50
4
0
Prop Pres Oral Prop Pres Oral Final Pres Oral Final Pres Oral
Comm: Purpose Comm: Purpose Comm: Purpose Comm: Purpose
& Style (PM)
& Style (FS)
& Style (FS)
& Style (PM)
APSC 100
APSC 100
APSC 100
APSC 100
Figure 17
59
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
400
350
300
250
200
1
150
2
100
3
50
4
0
Prop Pres Oral Prop Pres Oral Final Pres Oral Final Pres Oral
Comm: Slides Comm: Slides (FS) Comm: Slides (FS) Comm: Slides
(PM)
(PM)
APSC 100
APSC 100
APSC 100
APSC 100
Figure 18
450
400
350
300
250
1
200
150
2
100
3
50
4
0
Prop Pres Oral Final Pres Oral Prop Pres Oral Final Pres Oral
Comm: Speakers Comm: Speakers Comm: Speakers Comm: Speakers
(FS)
(FS)
(PM)
(PM)
APSC 100
APSC 100
APSC 100
APSC 100
Figure 19
60
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Professionalism
Individual professional behaviour, punctuality, responsibility, and communication etiquette was
assessed on the team design projects at the middle and end of the projects. Additionally, a coursespecific indicator was assessed that required students to deal with an ethical dilemma using PEO’s code
of ethics (this indicator overlaps with the Ethics and Equity attribute). The student scores are shown
below. Generally graders felt that the students showed appropriate professional behaviour on teams,
but many students’ performed below expectation when it came time to apply a code of ethics to an
ethical dilemma.
Figure 20
3.08 - FY2: Demonstrates
punctuality, responsibility and
appropriate communication
etiquette
Ethical
dilemma
(PSM4)
1 - Not Demonstrated
Threshold
2 - Marginal
Target
3 - Meets Expectations
4 - Outstanding
Fail
Marginal
Good
Excellent
1
(not demonstrated)
No process to deal
with dilemma.
2
(marginal)
Process to deal with
dilemma based on
personal beliefs only.
3
(meets expectations)
Process to deal with
dilemma based on
personal beliefs and
applicable codes of
ethics.
4
(outstanding)
Process to deal with
dilemma based on
personal beliefs and
applicable codes of ethics
addresses obligations to
multiple parties.
61
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Impact of Engineering and Ethics and Equity
As shown in the ethical dilemma in the previous section, and in the plot below, a significant number of
students were below expectation in applying code of ethics, and making decisions based on their
potential impact on society.
Figure 21
1 - Not Demonstrated
3.09 - FY4: Incorporates
sustainability considerations in No consideration of
decision making (societal and
these factors.
ecological)
3.10 - FY3: Describes ethical
issues and how they affect the No or trivial
individual, the company and the recommendations
public
Threshold
2 - Marginal
Target
3 - Meets Expectations
Incorporated appropriate
Factors mentioned but no
social, environmental, and
clear evidence of impact on
financial factors in decision
decision making.
making
Some important
recommendations and
implications missed
Makes supported
recommendations to prevent
future failures; describes
implications to stakeholders
4 - Outstanding
Well-reasoned analysis of these
factors, with risks mitigated
where possible
Meets expectations PLUS:
Information, model, and
conclusions used to make
recommendations supported by
authoritative information
62
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Economics and Project Management
Generally first year students were observed to plan and manage time and money effectively at the end
of the semester-long design project (3.11-FY1). However, many teams struggled to create an
appropriate project scope and project plan early in the project.
Figure 22
1 - Not Demonstrated
3.11 - FY1: Plans and efficiently
manages time and money
Threshold
2 - Marginal
No useful timeline or
Poor timeline or budget;
budget described;
infrequent meetings; minor
poorly managed
safety problems
project; safety issues
3.11 - FY2: Establishes
appropriate project scope, after Scope of project is not
consultation with client, based defined.
on available resources.
Project scope not
appropriate for available
resources and project
objectives.
3.11 - FY3: Plan maps out project Plan does not refer or Plan has a general outline
with clear milestones and
map out a timeline
of milestones with some
delegation.
for the project
reference to duration
Target
3 - Meets Expectations
Plans and efficiently manages
time and money; team
effectively used meetings;
safety considerations are clear
4 - Outstanding
Efficient, excellent project plan
presented; detailed budget;
potential risks foreseen and
mitigated
Project scope definition clearly
Establishes appropriate project defended by economic analysis,
scope using cost/benefits
available time and skills, and
linked to project timeline.
Plan maps out project with
clear milestones and
delegation appropriate to
project stage.
Detailed layout (e.g. Gantt chart)
with clear plan including
consideration for item
dependencies as well as room for
readjustment and remedial action
63
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Lifelong learning
Indicators for lifelong learning at the first year level focus primarily on information gathering and
evaluation. The most significant issue identified here is critical evaluation of information sources, as has
been observed in previous years.
450
400
350
300
250
1
200
2
150
3
100
4
50
0
FEAS - 3.12-FY1 FEAS - 3.12-FY2 FEAS - 3.12-FY5 FEAS - 3.12-FY6
(T2Scope&Info) (T1ProjectDef) (T1ProjectDef) (T2Scope&Info)
APSC100
APSC100
APSC100
APSC100
Figure 23
1 - Not Demonstrated
3.12 - FY1: Uses information
effectively, ethically, and legally
to accomplish a specific
purpose, including clear
attribution of Information
sources.
3.12 - FY2: Identifies a specific
learning need or knowledge
gap.
Target
3 - Meets Expectations
4 - Outstanding
No way to check
validity of
information.
Inconsistent use of citation
Clear attribution of all sources,
guidelines; some
Outstanding range of sources,
including graphics, using a
information sources
cited properly.
single citation style
ambiguous
No description of
learning need.
Some learning needs
identified but some gaps
remain.
Identifies learning needs or
knowledge gap based on
project requirements
Identifies learning needs
encompassing all phases of
project.
Some identified needs not
appropriately met by
identified information
sources.
Identifies appropriate
technical literature and other
information sources to meet a
need
Identifies multiple authoritative
sources from diverse
perspectives. All source selection
is justified.
Presents information that is
not relevant. Accepts all
information found. Does
not check for timeliness.
Describes professional-grade
Presents information that has
sources and reputable experts in
clearly been critically
technical, regulatory, and social
evaluated based on authority,
aspects, as appropriate.
currency and objectivity
3.12 - FY5: Identifies appropriate
technical literature and other
No useful information
information sources to meet a sources identified.
need
3.12 - FY6: Critically evaluates
the procured information for
authority, currency, and
objectivity.
Threshold
2 - Marginal
Presents information
that is not relevant.
Accepts all
information found.
Does not check for
timeliness.
64
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Overall analysis
The assessment of Knowledge Base in first year focused on introductory calculus, APSC-171. Based on
the data, the focus in the next year will be on helping students develop the ability to create a
mathematical model of a real-world problem, and on integration activities.
At the first year level, the majority of the attributes were assessed in APSC-100. In this course there
were several indicators on which students were scored low multiple times. The list of indicators on
which more than 20% of the students fell below the target expectation (“meets expectations”, or 3 out
of 4) were as follows (the number of times 20% or more of the students fell below the target is shown in
parentheses):







3.02-FY1: Identify problem (twice)
3.02-FY4: Evaluate solution (three times)
3.07-FY3: Summarizes accurately (only for proposal, this significantly improved by the end of the
first year)
3.10-FY3: Describes ethical issues (twice)
3.11-FY2: Project scope and 3.11-FY3: Project plan – these were weak on two deliverables in the
middle of the winter semester
3.12-FY5: Identifies appropriate literature
3.12-FY6: Evaluates information
Some of these were reassessed in other ways at the end of the first year, and the overall class
performance was much stronger at the end. At the end of the first year students were also evaluated on
overall written communication, and 16% of the teams were deemed below target.
Student performance on indicators in APSC-100 was examined over the year to examine how students
were adapting to expectations. The figure below shows the mean score on activities, and the percentage
of the students below target, as a function of the date in which the activity occurred. Early activities
more likely to be below threshold, and by the end of the year the percentage of students below
threshold was much lower than at the beginning. The trend lines suggest that students are adapting to
overall expectations within the course.
65
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
45
4.000
40
3.800
% Below target
3.600
Mean
3.400
30
3.200
25
3.000
20
2.800
15
2.600
10
2.400
5
2010-08
Mean score
Percent below target
35
2.200
2010-09
2010-11
2011-01
2011-02
2.000
2011-04
Approximate deliverable date
Figure 24 - Percent of students below target (red) and mean score on indicators scored out of four (blue) by date
The performance of students across all the indicators was also analyzed. The figure below shows the
number of students scoring below the target level of expectation on the indicators. Note that these
include faculty –wide indicators and course-specific indicators. Approximately half of the students did
not fall below target level on any indicator (this includes group work). Approximately 80% of the
students fell below threshold on 2 or fewer indicators.
66
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Figure 25
When the comparison is restricted only to faculty-wide indicators, the result is shown below. About half
of the students do not fall below the target expectation for any indicator, and 70% fall below for 2 or
fewer indicators. Approximately 80% do not fall below threshold level on any indicator, and 96% fall
below threshold for two or fewer indicators.
67
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
700
Number of students
600
584
500
Below target
400
300
Below threshold
293
200
96 108
61 48 55
100
56
54
37
16
6
0
0 13 0
30
00
00
00
00
00
0
0
1
2
3
4
5
6-10 11-1516-2021-2526-3031-3536-4041-50
Number of indicators
Figure 26
Plans for program improvement
At the first year level, specific plans for improvement based on the data include:




Grader calibration: In analyzing the data it was also observed that the variation between
graders was larger than desirable. In the previous year students were assigned to sections
randomly, and all students are taught by same instructor, but graded by section. An analysis
of average scores assigned by graders show statistically significant differences at the .05
level between graders scores on deliverables. As a result greater effort will go into grader
training and calibration
Problem analysis: greater focus will be placed on making an effective argument, and
comparing the outcome of a solution to the originally defined problem definition.
Design: the focus on safety and risk assessment is going to be expanded in the coming year
(overlaps with attributes Professionalism and Impact on Society), including focus on
Occupational Health and Safety.
Communications: Student feedback and performance indicate that communication skill
development needs to be revised in first year. Additional effort will be provided to support
students with weak communication skills at the beginning of first year. If an individual or
68
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program




group falls below some level of expectation in communications they will be required to
meet with the TA for detailed feedback, and pointed at resources that they can use to
improve the document before resubmitting.
Economics and project management: additional emphasis is being placed on problem
definition, project scoping and planning
Lifelong learning: additional emphasis is being placed on evaluating information and making
an effective argument
On some indicators a large number of students are above expectations, suggesting that the
expectations may be too low
Based on survey results, student view ethics and equity, professionalism, and impact of
engineering as receiving lower priority within program. This is being emphasized in the
engineering design and practice stream being introduced in all programs
Faculty-wide middle year assessment: Communications
These two courses taken by most students in second year provide basic instruction in Engineering
Communications. Full details of their delivery are available in the supplemental materials filed with the
Course Information Sheets. The course activities were loosely tied to technical and design activities in
other courses, varying with program. The rollout of our new common design course APSC 200 in the
2011-2012 academic year will allow tight integration of the communications elements of the new APSC
293 with student design projects in second year across all programs.
The focus of both courses was on student communication activities, practice, and revision in workshops.
There were five modules in each of the 12 AU courses, each equally weighted. Each course included an
individual presentation module, a group presentation module, and three written modules. They were:







Job Application, including resume and cover letter. (291 JA)
One Minute Engineer, a brief extemporaneous oral presentation, followed by written reflection
on video of the performance and means for improvement. (291 OME)
Briefing Note, a three page written analysis of the potential for district heating at Queen’s. (291
BN)
Incident Report, a written report of a fictitious machine shop accident, using mandated forms
and a follow-up memo. (291 IR)
Group Presentation, with supporting slides, on a technical topic chosen by the students. (291
GP)
Executive Summary, from review of a report prepared by 4th year students. (292 ES)
Team Oral Communication, in a design group meeting with the aid of a whiteboard, followed by
written reflection on performance and means for improvement. (292 TOC)
69
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program

Procedures and Systems, detailing a sequence in a process and/or a set of instructions,
complete with block diagram. (292 PS)
 Discussion and Conclusions, communications and reasoning on a technical topic chosen by the
students. (292 DC)
 Group Presentation, with supporting slides, on a technical topic chosen by the students. (292
GP)
To pass, students were required to achieve a satisfactory outcome in all modules. Revision cycles in
workshops and resubmission allowed all students the opportunity to improve their work. (Incorporation
of specific, marked revision cycles represents an improvement over previous offerings with a single final
submission of each deliverable.) Specific communications outcomes from the CDIO Syllabus were
evaluated for each module, as well as a separate outcome for overall performance on the module. The
overall performance was not a simple sum of the other outcomes, thus a submission could be
unsatisfactory and require resubmission even if 3 out of 4 subsidiary outcomes were satisfied. CDIO
Syllabus topics were used in part because of the instructor’s familiarity with them, and in part because
the faculty outcome list had not been finalized when the course design was being completed. The
outcomes used will be harmonized as part of the roll-out of the faculty-wide integrated design and
communications courses APSC 200/293 in Fall 2011. All of the outcomes used are contributors to
meeting Graduate Attribute 7, Communications.
Data were accumulated for each outcome during the marking process and stored in the Moodle course
management system. The charts that follow were completed based on Moodle records at the end of the
academic year. They do not reflect the subsequent resubmission and success of a small number of
students who completed modules after the end of classes. In addition, some subsidiary outcomes were
not recorded when students completed revisions in workshops, resulting in lower numbers of
observations. The data for all students in both courses was downloaded, aligned to provide a complete
record for each student, and then anonymized for privacy. The resulting frequencies were accumulated
over all students and plotted for each outcome in each submission of each module. The overall data set
includes 483 students, the vast majority of whom participated in both courses.
Each of the outcomes were ranked on an Engineering Practice scale of:
 Resubmit: The work is unsatisfactory and must be revised.
 File Only: The work is sufficient to provide a record for the file.
 Internal Circulation: I would show this to my boss.
 Almost Client Ready: Only a few minor tweaks are required.
 Client Ready: The work is ready for submission to an outside client with no revisions.
All of the outcomes are satisfactory except Resubmit. File Only is below expectations, but sufficient to
pass the course. More details accompany the Course Information Sheets.
Students generally performed well on the communications tasks set for them. There is evidence that
they were able to improve their outcomes through a cycle of revisions. Any improvements over the
course of the year are masked by other factors, including rising expectations of academic assistants over
the course of the year, student workloads, course administration glitches, etc. More telling comparisons
70
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
are available between second year and final year outcomes displayed in communications elements of
capstone projects.
Over the course it became evident that students were not expecting the graders to apply the strong
standards we advertised for the first assignment, and the vast majority of students were required to
resubmit their job applications, mostly on matters related to format, spelling, punctuation and
grammar. Most students can meet targets of either Internal Circulation or Almost Client Ready on the
first try. Student performance on each module after their revision process in workshops and
resubmission of their work shows that more than 85% of the students met expectations with scores of
Internal Circulation or above and virtually all students were able to meet a minimum threshold standard
of File Only after multiple revisions.
Several areas can be identified where students show higher frequencies of the marginal File Only
outcome were also identifiable from anecdotal experience. These items should receive more attention
in the APSC 200/293 offering in 2011/12:


Students had more difficulty identifying key elements for a summary and expressing them
clearly and concisely.
More students performed below expectations in informal small group discussions than in a
formal presentation, and they were less able to identify their problems and opportunities to
improve on them.
The chart below shows APSC 291/292 outcomes related to the mechanics of written communication.
The outcomes on the left stand out because the Job Application module required students to resubmit
at least twice to eliminate any errors in format, punctuation, spelling or grammar. The vast majority of
students meet expectations on all assignments and can exceed expectations when required.
71
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
72
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
This chart shows APSC 291/292 outcomes related to critical thinking in communications.
73
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
This chart shows APSC 291/292 outcomes for oral presentations, both extemporaneous, individual presentations and more formal group presentations with
slides.
74
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Faculty-wide Middle year assessment: Economics
The table below shows the percentage of the course assessment on the five indicators. These include all
assignments (5 individual 2, team), and questions on the midterm and the final exam.
Code
3.11-MY1
3.11-MY2
3.11-MY3
3.11-MY4
3.11-MY5
Indicator
Gather appropriate information, categorize it, and determine the economic
attractiveness of an engineering project
Measure the financial impact of risks that are associated with the
engineering project and consider the risk and return relationship as a
component of determining economic attractiveness
Plans a project from scoping to completion, including scheduling
Plans for the management of risks and changes required for project
Describes new enterprise formation, business planning and the economic
attractiveness of the enterprise.
% of course
assessment
41
3
15
8
34
The description under Economics and project management speaks specifically to indicators 1, 2, 3 and 4
while indicator 5 was included as important related material.
Student performance on activities was analyzed after the course. Students were scored on a numeric
scale appropriate to the deliverable, rather than using a rubric as in other courses. Overall student
performance on the five indicators measured on questions given on the midterm and final exams are
shown below for comparison.
Code
MY1
Indicator
Economic
attractiveness
Indicator
Gather appropriate information, categorize it, and
determine the economic attractiveness of an engineering
project
MY3
Project Plan
Plan the project from scoping to completion, including
scheduling
MY4
MY5
Manage Risks
Describe
Enterprise
plan for the management of risks and changes required
Describe new enterprise formulation, business planning
and enterprise economic attractiveness
AVG High Low
59 100
0
63 100
5
79 100
35
95 100
0
70 100
0
82 100
45
74 100
0
35 100
0
69 100
15
78 100
30
MY5, describing enterprise formulation and economic attractiveness, had the lowest average score.
75
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Observations
Slightly less emphasis can be placed on MY1 (economic attractiveness) and a lot less on MY5 (new
enterprise formation). While MY5 is reduced overall the Capital asset Pricing Model (CAPM) reflecting
cost of equity for the Enterprise must be included. Slightly more emphasis will be placed on developing
and assessing MY2,MY 3, and MY4 (risk assessment, project scoping, and risk management).
Approach for 2011-12:
One more indicator will be added to the course activities:
Describe a project's sustainability and broader contribution and impact on the enterprise,
environment and society.
Overall the course learning outcomes will be the six economic indicators. The course delivery and
assessment will be adjusted to provide a balance in assessment.
Code
3.11-MY1
3.11-MY2
3.11-MY3
3.11-MY4
3.11-MY5
3.11-MY6
Indicator
Gather appropriate information, categorize it, and determine the economic
attractiveness of an engineering project
Measure the financial impact of risks that are associated with the
engineering project and consider the risk and return relationship as a
component of determining economic attractiveness
Plans a project from scoping to completion, including scheduling
Plans for the management of risks and changes required for project
Describes new enterprise formation, business planning and the economic
attractiveness of the enterprise.
Describe a project's sustainability and broader contribution and impact on
the enterprise, environment and society.
% of course
assessment
35
5
20
10
20
10
76
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Student input on Graduate attributes
The faculty conducted a survey across all programs to understand student perceptions of the graduate
attribute requirements, and to solicit their input into how the program can be developed to help them
develop the required attributes. This was done by two methods:
1. Web-based surveys of first year and graduating year students. Survey questions are shown in
Appendix 3.1B.
2. Interviews with members of the undergraduate engineering student society.
This study was approved by the Queen’s University General Research Ethics Board.
First year survey
Of the approximately 650 students in the first year program 188 students (29%) responded to the
survey, and 129 of those (20%) completed the entire survey.
Students were asked to report the degree to which they felt they already possessed attributes upon
entry to the program. The top five graduate attributes where students reported a rating of 2 or 3 (yes or
to a great degree) out of three is as follows:
Individual and Team Work
Communication Skills
Professionalism
Problem Analysis
Investigation
89%
78%
69%
61%
61%
Students were then asked to report on the degree to which first year programming supported
development of the attributes. The top five graduate attributes where students reported a rating of 2 or
3 (yes or to a great degree) out of three is as follows:
Individual and Team Work*
Knowledge base in engineering
Problem analysis*
Professionalism*
Investigation*
Design
Impact of Engineering on Society
95%
94%
94%
86%
83%
81%
81%
*Identified as a strength coming in to the program
77
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Knowledge base and problem solving were most frequently listed as the top priority in the program as
shown in the table below. This is not expected in an engineering program, particularly in first year when
the majority of the courses focus on natural sciences and mathematics. Interestingly, teamwork,
communications, and professionalism were also frequently listed.
Attribute
Problem Solving*
Individual & Team Work*
Knowledge Base
Communication*
Professionalism*
# of students who incl it in
top 5
120
97
95
76
47
# of students who selected it
as #1 priority
34
17
51
6
4
The three listed least frequently as being developed by first year programming were:
Ethics and Equity
Economics and Project Management
Lifelong Learning
64%
70%
73%
It is interesting that even the attributes with the lowest scores were perceived by approximately twothirds of the first year class as being developed in the first year of the program.
Attribute
Lifelong learning
Economics and Project
Management
Ethics and Equity
Use of Eng Tools
Impact on Society
# who included it in bottom 3
80
68
45
38
37
# who ranked it lowest
priority
38
25
1
15
6
78
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Fourth year surveys
The graduating year survey was sent to approximately 600 final year students. There were a total of 156
responses, and 102 completed the entire survey.
Students were asked which of the attributes they felt were difficult to demonstrate. Responses, along
with some general comments, are shown in the table below. There were 73 responses to this question.
Attribute
Ethics and Equity
# of students
25
Professionalism
Impact on Society
Communication
13
12
11
Design
9
Lifelong Learning
Economics & Project
Mgmt
8
8
Rationale
Next to no content since 1st year on these issues; student
perception of unethical behaviour in program (plagiarism
etc)
Lack of opportunity in the program (12)
Many assignments are team based so difficult for all
students to demonstrate competence; feedback on
written work takes too long to be able to improve for
next assignment; not always integrated
Design is integral, should be focal point of more courses;
focus on team work means that individual contributions
are difficult to recognize/demonstrate
More of an attitude than a skill
Lack of opportunity in the program; only touched on in
APSC 100 and APSC 221
Four of the students pointed out that, aside from knowledge base for engineering, most attributes are
difficult to demonstrate through traditional exams. There was a general perception that ethics and
equity, professionalism, and impact on society weren’t significant in upper years of programs. This has
was identified as an area of improvement several years ago, and will be discussed under program
development.
Students were asked to what degree did the engineering program support the development of
attributes. The table below shows the percentage of students who responded affirmatively (2) or to a
high degree (3) out of 3. The attribute for which the most students felt the program supported their
development to a high degree was individual and team work, and the lowest was ethics and equity.
Attribute
Knowledge base
Problem analysis
Ind & Team work
Investigation
Design
Use of eng tools
Communication
2+3
98.41
95.25
93.65
91.27
88.10
84.93
80.95
3 (to a high degree)
63.49
69.05
74.60
42.86
50.00
38.10
43.65
79
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Lifelong learning
Professionalism
Economics & pro mgmt
Impact on society
Ethics and equity
79.36
79.20
75.20
68.26
60.00
46.03
36.80
20.00
27.78
17.60
The attributes ranked by students as being given the highest priority most often were:
 Knowledge base for engineering (53)
 Problem analysis (31)
Those most often included in the top 5 were:





Problem analysis (105)
Knowledge base (92)
Individual & team work (80)
Design (79)
Communication (45)
Those ranked by students as being given the lowest priority the most often were:
 Ethics and equity (29)
 Impact on society (20)
Those most often included in the bottom five were:
 Ethics and equity (73)
 Impact on society (51)
 Professionalism (46)
 Lifelong learning (46)
With the exception of professionalism, the same attributes were ranked lowest by first year students.
80
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Individual interviews
Toward the end of the academic year students were asked to participate in focus groups and interviews.
At the time students were in exams and so it was difficult to get significant participation.
In the end two undergraduate engineering students were interviewed, both of whom were members of
the undergraduate engineering society executive elected by their peers. Generally these students are
better informed and more engaged in faculty matters than the average student.
One student had just finished second year, one just finished third year.
The most significant comments and recommendations from the students were:







Communicate graduate attributes and draw attention back to them. E.g. For example by
pointing out that assignments develop and demonstrate particular attributes
Use writing activities to support under-developed attributes in program (impact, ethics, etc.)
Make communications an important part of multiple courses, not only communications-only
courses
Don’t understand “lifelong learning” attribute
Professionalism and ethics and equity should be focused on in upper years
Portfolios would not be well received if they were an add-on to what’s already in the curriculum
Movement toward grad attributes is good. Need better communication about them and stress
that they are important elements of what it means to be an engineer
81
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Program development informed by data
Faculty wide development
Improvements in specific faculty-wide course were addressed in the previous sections on first and
middle year courses. This section will address a proposal for a sequence of four courses on Engineering
Design and Practice that has been in development for two years. This course will target 11 of the 12
graduate attributes; only knowledge base is not specifically targeted, as it is the primary attribute
developed in most engineering courses. As identified in the curriculum mapping exercise and student
surveys, development of these professional skills needs to be enhanced. The sequence will build upon
existing courses, including APSC-100 in first year, capstone design courses in the graduating year, and
some existing design and professional practice courses.
The first new course introduction happens in the 2011-2012 with a new pair of faculty-wide second year
courses, APSC-200 and 293. These will be taught by the same personnel, and are separated into two
courses for the purpose of record-keeping. APSC-200 focuses primarily on design and engineering
practice, and APSC-293 focuses on written and oral communications. The projects in the design course
will serve as the context for the reports and presentations used in the communications course. This will
provide multiple revision cycles to help students develop an effective writing process. The design
activities are also supported by materials from APSC 221/321 on engineering economics and business
practices.
In this project based design course, students will participate constructively on teams to create solutions
to open-ended problems, using standard design methods and tools. The course opens with a discipline
specific project activity in the first week to engage the students in an active learning opportunity.
Instruction will be provided primarily in the first 6 weeks of the semester focusing on problem scoping,
creativity and idea generation, decision making incorporating technical, economic, societal, and
environmental factors, safety, engineering codes and regulations, and engineering ethics. In that first six
weeks student groups complete a simple design/build project to provide a focus for tools they are
learning. The final 6 weeks of the course centers around a more advanced design project delivered by
each discipline. The communication course will include a small number of lectures, online modules, and
working tutorials supporting deliverables in the design course. This ensures the communications
expectations occur in an engineering context.
The course objectives include:



Employ and apply design processes and tools with emphasis on early stages (problem definition,
idea generation and decision making)
Promote creative processes in open ended problem solving
Design (involving analysis and/or simulation and/or prototyping) a solution for presented openended problem
82
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program


Excite students about engineering and the discipline they selected
Apply engineering principles and theories from other disciplinary courses to solve an openended problem
 Employ appropriate technical and non-technical factors in decision making, including some
combination of economic, environmental, safety, and social factors, and consideration of the
public interest
 Awareness of engineering as a regulated profession, including reference to some relevant
engineering regulations/codes/standards
 Awareness of professional/technical associations in engineering and discipline
 Awareness of occupational safety regulations
 Teaming skills
In future years third and fourth year courses will be introduced that build upon these objectives.
Section 3.1 Part B: Future plans for graduate attribute assessment
Programs should describe future plans for graduate attribute assessment in preparation for the next
accreditation cycle when criterion 3.1 will form a basis for accreditation decisions. Also describe the
assessment cycle if attributes will be assessed on a rolling basis.
In future years the program development process will be expanded to include more detailed data
gathering and analysis. Specific improvements include:










All 12 graduate attributes will be assessed in first year and the graduating year, and many will
also be assessed in middle years
Review and revise current indicators based on previous year use, including providing
information about the context in which each is evaluated
Program specific indicators will be developed for many of the attributes, including knowledge
base, engineering tools, and investigation
Faculty-wide and departmental committees will establish regular timelines for attribute
assessment, data analysis, and program improvement. These will include student members
Industrial feedback on the process and outcomes will be incorporated
Mapping of attributes in the existing program will be expanded to ensure that attributes are
being developed throughout the program
Faculty-wide design and professional practice courses will be used allow longitudinal assessment
of students throughout their program
Recently introduced and upcoming provincial and university policies for quality assurance will be
incorporated into the program development process
Policies will be established to deal with students who fall below threshold performance on
indicators. In some cases this will require students revise and resubmit material until threshold
performance is met
Instructors will describe relationship of course materials and deliverables to attributes
83
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program


Student portfolios will be introduced in selected courses to allow students to determine how
they meet expectations
Indicators will be assessed using multiple methods to assess reliability of data
84
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Appendix 3.1A: Levelled indicators
These indicators were developed in the 2009-2010 academic year. A selection of these were assessed in the 2010-2011 year based on timeliness
and the current state of the curriculum.
Sec
Attribute
First year
3.01-FY1
3.01-FY2
3.01
3.02
Knowledge
base
Problem
analysis
Middle years
Graduating year
Create mathematical descriptions or
expressions to model a real-world
problem
Select and describe appropriate tools
to solve mathematical problems that
arise from modeling a real-world
problem
3.01-FY3
Use solution to mathematical
problems to inform the real-world
problem that gave rise to it.
3.01-FY4
FCI score as indicator of basic physics
concepts
3.01-FY5
CCI score as indicator of basic
chemistry concepts
3.02-FY1
Identifies known and unknown
information, uncertainties, and
biases when presented a complex illstructured problem
3.02-GY1
3.02-FY2
Creates process for solving problem
including justified approximations
and assumptions
3.02-GY2
3.02-FY3
Selects and applies appropriate
quantitative model and analysis to
solve problems
3.02-GY3
Selects and applies appropriate
model and analysis to solve problems
3.02-FY4
Evaluates validity of results and
model for error, uncertainty
3.02-GY4
Evaluates validity of results and
model for error, uncertainty
Identifies problem, known and
unknown information, uncertainties,
and biases
Creates process for solving problem
including justified approximations
and assumptions
85
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Sec
3.03
Attribute
First year
3.03-FY1
Generates ideas and working
hypothesis
3.03-FY2
Designs investigations involving
information and data gathering,
analysis, and/or experimentation
Investigation
3.03-FY3
3.03-FY4
3.04
Design
3.04-FY1
3.04-FY2
Middle years
Graduating year
Synthesizes data and information to
reach conclusion
Appraises the validity of conclusion
relative to the degrees of error and
limitations of theory and
measurement
Adapts general design process to
design system, component, or
process to solve open-ended
complex problem.
Accurately identifies significance and
nature of a complex, open-ended
problem
3.04-FY3
3.04-GY1
3.04-GY2
3.04-GY3
Identifies customer and user needs
3.04-FY4
3.04-FY5
3.04-FY6
Gathers and uses information from
appropriate sources, including
applicable standards, patents,
regulations as appropriate.
Produces a variety of potential design
solutions suited to meet functional
specifications
3.04-GY4
3.04-GY5
Identify problem and constraints
including health and safety risks,
applicable standards, economic,
environmental, cultural and societal
considerations
Applies appropriate knowledge,
judgement, and design tools, in
creating and analyzing conceptual
design solutions to select best
concept
Creates and tests simulations,
models, and/or prototypes at various
points in design with complexity
appropriate to design stage
Assesses design performance based
on requirements, yield, reliability,
and/or safety as appropriate
Identifies possibilities for further
improvement and conducts design
review to evaluate performance of
the overall process.
Performs systematic evaluations of
the degree to which several design
concept options meet project criteria
86
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Sec
3.05
3.06
Attribute
Engineering
tools
Teamwork
First year
3.04-FY7
Compares the design solution against
the problem objective
3.05-FY1
Selects appropriate measurement
devices or techniques to accomplish
a task
3.05-GY1
Evaluates techniques, resources, and
tools to identify their limitations with
respect to needs
Applies appropriate techniques,
tools, and processes to accomplish a
task
Evaluates appropriateness of results
from several engineering techniques
and tools
Follows protocols when using
techniques, skills and tools
3.05-GY2
3.05-FY3
Demonstrates correct use of testing
apparatus, databases and models
3.05-GY3
3.06-FY1
Recognizes a variety of working and
learning preferences
3.06-GY1
Demonstrates capacity for initiative
and technical or team leadership
while respecting others' roles
3.06-FY2
Applies principles of conflict
management to resolve team issues
3.06-GY2
Evaluates team effectiveness and
plans for improvements
3.06-FY3
Assumes responsibility for own work
and participates equitably
Describes own temperament
3.07-GY1
Writes and revises documents using
appropriate discipline-specific
conventions
3.06-FY4
3.06-FY5
Critically analyzes results from a
temperament sorter, defending
opinion of how well results apply
3.06-FY6
Analyzes impact of own
temperament on group work
3.06-FY7
Exercises initiative and contributes to
team goal-setting
Writes using standard formats
3.07
Graduating year
3.05-FY2
3.07-FY1
Communications
Middle years
3.07MY1
Adapts format, content,
organization, and tone for
various audiences
3.07-FY2
Writes using standard grammar and
mechanics
3.07-GY2
Demonstrates accurate use of
technical vocabulary
3.07-FY3
Summarizes and paraphrases written
work accurately with appropriate
citations
3.07-GY3
Demonstrates confidence in formal
and informal oral communications
87
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Sec
Attribute
First year
3.09
3.10
Professionalism
Impact of
engineering
Ethics and
equity
Graduating year
3.07-FY4
Delivers clear and organized formal
presentation following established
guidelines
3.07-GY4
Uses appropriate referencing to cite
previous work
3.07-FY5
Creates effective figures, tables, and
drawings employing standard
conventions to compliment text.
3.07-GY5
Uses graphics to explain, interpret,
and assess information
3.08-FY1
3.08
Middle years
3.08-FY2
Describes role of protection of the
public and public interest in decision
making
Demonstrates punctuality,
responsibility and appropriate
communication etiquette
3.08-GY1
3.08-GY2
Demonstrates professional bearing
3.08-FY3
Participates actively in meetings,
helps to generate ideas
3.09-FY1
Describes relationship between
human activity and earth systems
3.09-GY1
3.09-FY2
Describes management techniques
for sustainable development
Describes relevant legal
requirements governing engineering
activities
3.09-GY2
3.09-FY3
3.09-FY4
Incorporates sustainability
considerations in decision making
(societal and ecological)
3.10-FY1
Demonstrates behaviour congruent
with academic integrity expectations
of university and faculty
3.10-FY2
Identifies and iterates items from the
professional codes of conduct
3.10-FY3
Describes ethical issues and how they
affect the individual, the company
and the public
Integrates standards, codes of
practice, and legal and regulatory
factors into decision-making
processes (as appropriate)
3.09-GY3
3.09-GYX
Considers economic, social, and
environmental factors and/or
impacts in decisions
Evaluates trade-offs among goals and
concepts
Explains the societal, enterprise,
and/or technical context of the
system
Evaluates technical, social, and
environmental trade-offs among
goals and concepts
88
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Sec
Attribute
First year
3.10-FY4
Plans and efficiently manages time
and money
3.11-FY2
Establishes appropriate project
scope, after consultation with client,
based on available resources.
Economics
3.11-FY3
Plan maps out project with clear
milestones and delegation.
3.11MY1
3.11MY2
3.11MY3
3.11MY4
3.11MY5
3.11MY6
3.12
Lifelong
learning
Graduating year
Describes consequences of deviating
from professional codes of conduct
and university code of conduct
3.11-FY1
3.11
Middle years
Gather appropriate
information, categorize it,
and determine the economic
attractiveness of an
engineering project
Measure the financial impact
of risks that are associated
with the engineering project
and consider the risk and
return relationship as a
component of determining
economic attractiveness
Plans a project from scoping
to completion, including
scheduling
Plans for the management of
risks and changes required
Describes new enterprise
formation, business planning
and the economic
attractiveness of the
enterprise.
Describe a project's
sustainability and broader
contribution and impact on
the enterprise, environment
and society
3.12-FY1
Uses information effectively,
ethically, and legally to accomplish a
specific purpose, including clear
attribution of Information sources.
3.12-GY1
3.12-FY2
Identifies a specific learning need or
knowledge gap.
3.12-GY2
Critically evaluates procured
information for authority, currency,
and objectivity.
Demonstrate skills of self-education
89
Mechanical Engineering, Queen’s University - Questionnaire for Evaluation of an Engineering Program
Sec
Attribute
First year
3.12-FY3
Makes accurate use of technical
literature and other information
sources.
3.12-FY4
Middle years
3.12-GYX
3.12-GYX
Describes own learning style
3.12-FY5
Identifies appropriate technical
literature and other information
sources to meet a need
3.12-FY6
Critically evaluates the procured
information for authority, currency,
and objectivity.
3.12-FY7
Describes connections between past
experience and future projects
3.12-FY8
Selects project planning strategies
based on past experience
3.12-GYX
Graduating year
Describes professional and academic
societies in the discipline and how
new knowledge enters discipline.
(Later year)
Identifies resources and professional
associations that address own
ongoing professional development
(Later year)
Recognizes the need to keep current
regarding new developments in field
(Later year)
90
Appendix 3.1B: Student surveys
Students in first and graduating years were asked to complete web-based surveys on their
perceptions of the graduate attributes. The questions from the fourth year survey are show
below; the first year survey was similar but with fewer questions.
Continuous Program Improvement to Support Engineering Graduate Attribute
Development: A pilot project
Survey Questions
Fourth-Year Students
The 12 Engineering Graduate Attributes identified and defined by the Canadian
Engineering Accreditation Board will be made available for students' reference
while completing the survey.
3.1.1 A knowledge base for engineering:
Demonstrated competence in university level mathematics, natural sciences,
engineering fundamentals, and specialized engineering knowledge appropriate to the
program.
3.1.2 Problem analysis:
An ability to use appropriate knowledge and skills to identify, formulate, analyze, and
solve complex engineering problems in order to reach substantiated conclusions.
3.1.3 Investigation:
An ability to conduct investigations of complex problems by methods that include
appropriate experiments, analysis and interpretation of data, and synthesis of
information in order to reach valid conclusions.
91
3.1.4 Design:
An ability to design solutions for complex, open-ended engineering problems and to
design systems, components or processes that meet specified needs with appropriate
attention to health and safety risks, applicable standards, economic, environmental,
cultural and societal considerations.
3.1.5 Use of engineering tools:
An ability to create, select, apply, adapt, and extend appropriate techniques, resources,
and modern engineering tools to a range of engineering activities, from simple to
complex, with an understanding of the associated limitations.
3.1.6 Individual and team work:
An ability to work effectively as a member and as a leader in teams, preferably in a multidisciplinary setting.
3.1.7 Communication skills:
An ability to communicate complex engineering concepts within the profession and with
society at large. Such abilities include reading, writing, speaking and listening, and the
ability to comprehend and write effective reports and design documentation, and to give
and effectively respond to clear instructions.
3.1.8 Professionalism:
An understanding of the roles and responsibilities of the professional engineer in society,
especially the primary role of protection of the public and the public interest.
3.1.9 Impact of engineering on society and the environment:
An ability to analyse social and environmental aspects of engineering activities. Such
abilities include an understanding of the interactions that engineering has with the
economic, social, health, safety, legal, and cultural aspects of society; the uncertainties
in the prediction of such interactions; and the concepts of sustainable design and
development and environmental stewardship.
92
3.1.10 Ethics and equity:
An ability to apply professional ethics, accountability, and equity.
3.1.11 Economics and project management:
An ability to appropriately incorporate economics and business practices including
project, risk and change management into the practice of engineering, and to
understand their limitations.
3.1.12 Life-long learning:
An ability to identify and to address their own educational needs in a changing world to
sufficiently maintain their competence and contribute to the advancement of knowledge.
1) With 0 being not at all and 3 being to a great degree, to what extent did the
Engineering Program support your development in:
a. A knowledge base for engineering
b. Problem analysis
c. Investigation
d. Design
e. Use of engineering tools
f. Individual and team work
g. Communication skills
h. Professionalism
i. Impact of engineering on society and the environment
j. Ethics and equity
k. Economics and project management
l. Lifelong learning
2) Please identify the 5 attributes that, in your opinion, are given the highest priority
in the Engineering Program. Use 1 to indicate the highest priority attribute; 2 to
indicate the next highest and 3 to indicate the next:
(all attributes listed; drop down boxes for ranking)
3) Which 3 graduate attributes, in your opinion, are given the least priority in the
Engineering Program? (Use 1 to indicate the least priority attribute; 2 to indicate the
next least and 3 to indicate the next.)
93
(all attributes listed; drop down boxes for ranking)
4) Which instructional/assessment strategies were most beneficial in allowing you to
develop each of the graduate attributes? Please select no more than 2 for each
attribute:
Lecture-based classes
Small group long-term projects
Field placements
Studio
Seminars
Small group, short-term projects
Problem-based learning
Labs
Problem sets
Other (please specify)
5) In your opinion and given current course structures and assessment
strategies, which attributes were the most difficult for you to demonstrate?
Why?
6) It is recognized that many opportunities outside the regular engineering
curriculum support the development of graduate attributes. Please indicate
your involvement in the activities listed below and the degree to which they
contribute to your development of graduate attributes:
N/A
1-3 hrs/wk
4-7 hrs/wk
more than 7
Student Government
Special Interest Clubs
Sports Clubs
Employment
Other
7) For purposes of program improvement we would like to be able to make
inferences about the ways program strengths and limitations affect acquisition of
engineering graduate attributes across levels of academic achievement. Please
94
select the grade range that most accurately reflects your overall academic
average in the Queen’s engineering program.
95
Appendix 3.1C: Definitions and Guidelines
Criterion 3.1
The institution must demonstrate that the graduates of a program possess the attributes under
the following headings. The attributes will be interpreted in the context of candidates at the
time of graduation. It is recognized that graduates will continue to build on the foundations that
their engineering education has provided.
3.1.1 A knowledge base for engineering: Demonstrated competence in university level
mathematics, natural sciences, engineering fundamentals, and specialized engineering
knowledge appropriate to the program.
3.1.2 Problem analysis: An ability to use appropriate knowledge and skills to identify, formulate,
analyze, and solve complex engineering problems in order to reach substantiated
conclusions.
3.1.3 Investigation: An ability to conduct investigations of complex problems by methods that
include appropriate experiments, analysis and interpretation of data, and synthesis of
information in order to reach valid conclusions.
3.1.4 Design: An ability to design solutions for complex, open-ended engineering problems and
to design systems, components or processes that meet specified needs with appropriate
attention to health and safety risks, applicable standards, and economic, environmental,
cultural and societal considerations.
3.1.5 Use of engineering tools: An ability to create, select, apply, adapt, and extend appropriate
techniques, resources, and modern engineering tools to a range of engineering activities,
from simple to complex, with an understanding of the associated limitations.
3.1.6 Individual and team work: An ability to work effectively as a member and leader in teams,
preferably in a multi-disciplinary setting.
3.1.7 Communication skills: An ability to communicate complex engineering concepts within the
profession and with society at large. Such ability includes reading, writing, speaking and
listening, and the ability to comprehend and write effective reports and design
documentation, and to give and effectively respond to clear instructions.
3.1.8 Professionalism: An understanding of the roles and responsibilities of the professional
engineer in society, especially the primary role of protection of the public and the public
interest.
96
3.1.9 Impact of engineering on society and the environment: An ability to analyze social and
environmental aspects of engineering activities. Such ability includes an understanding of
the interactions that engineering has with the economic, social, health, safety, legal, and
cultural aspects of society, the uncertainties in the prediction of such interactions; and the
concepts of sustainable design and development and environmental stewardship.
3.1.10 Ethics and equity: An ability to apply professional ethics, accountability, and equity.
3.1.11 Economics and project management: An ability to appropriately incorporate economics
and business practices including project, risk, and change management into the practice
of engineering and to understand their limitations.
3.1.12 Life-long learning: An ability to identify and to address their own educational needs in a
changing world in ways sufficient to maintain their competence and to allow them to
contribute to the advancement of knowledge.
97