To: Dr. Diane Beaudoin, Director of Assessment for the Office of the Provost at Purdue
From: Dr. Barrett Caldwell and Jeremi London (Masters student)
Subject: Learning Outcomes Assessment Grant Project Summary Report
Date: April 18, 2013
OVERVIEW
The purpose of this report is to summarize the goals, methods and outcomes of Learning Outcomes
Assessment Grant awarded in Spring 2012. The following sections include summaries of Project
Goals, Methods and Results, and Impact and Future Activity. These sections address: what was the
project about (Project Goals); what were results and what did we learn (Methods and Results); and
what was impact on our program / what’s next (Impact and Future Activity).
SUMMARY OF PROJECT GOALS
The purpose of this study was to map, align, and estimate assessment outcome regression in
lower divisions of engineering education. More specifically, the three project goals were:
1. Systematically link three sets of learning outcomes: Purdue Learning Outcomes, Purdue
Engineer of 2020 Outcomes, and ABET Student Learning Outcomes.
2. Determine which outcomes are addressed in the First Year Engineering (FYE) program and
in the sophomore year of the Industrial Engineering program; and empirically evaluate the
extent to which the outcomes are addressed in FYE and in the sophomore year of the IE
program.
3. Generate a process flow simulation model that captures course-grain outcomes
achievements across ABET criteria, Purdue Learning Outcomes, and Purdue Engineer of
2020 outcomes (referred to hereafter as ABET outcomes, PLOs, and PE2020, respectively).
SUMMARY OF METHODS AND RESULTS
Three main tasks were completed addressing the first goal, to systematically link three sets of
learning outcomes. First, the three lists of learning outcomes were compiled. There are 13 Purdue
Learning Outcomes; 20 Purdue Engineer of 2020 outcomes; and 11 ABET Student Learning
Outcomes. Second, a survey was designed, piloted, and administered to allow participants to indicate
the extent to which pairs of learning outcomes were aligned (using an ordinal scale of 0, 1, 2). The
initial survey was emailed to the entire faculty in the Purdue College of Engineering (approx. 350
faculty). A follow-up survey was also emailed to the CoE faculty to address issues that arose when
analyzing the results of the initial survey. Overall, 29 CoE faculty responded to some version of the
survey and completed survey items other than the demographic questions (8.2% response rate). The
survey results were collated into three contingency tables.
Figures 1, 2 and 3 present “modal consensus” responses to questions about relationships
between ABET outcomes and PE2020 outcomes; ABET and the PLOs; and between PE2020 and
PLO outcomes. Because of the restricted ordinal nature of the data, a “modal consensus” technique
was used to report agreement among faculty perceptions of outcome alignment. (Modal consensus
refers to cases where no respondents respond to one anchor (0 or 2), and the modal response is the
opposite anchor (2 or 0). This technique allows demonstration agreement on “no relation” or
“strong relation” with a small sample size.) While there was a consensus among responses on the
relationship between some of the outcome pairs, many important relationships did not indicate a
consensus response. Figures 1, 2, and 3 are presented next.
Figure 1 - Mapping ABET Student Outcomes and Purdue Engineer of 2020 Outcomes
Figure 2 - Mapping ABET Student Outcomes and Purdue Learning Outcomes
Figure 3 - Mapping Purdue Learning Outcomes and Purdue Engineer of 2020 Outcomes
Though exploratory, the findings of this study suggest that all ABET outcomes link to at
least one PE2020 or PLO outcomes. Additionally, all of the campus-wide PLOs map to the ABET
outcomes. Surprisingly, there are several PE2020 outcomes that do not map as expected to ABET
outcomes among faculty perceptions. For example, ABET outcome I (a recognition of the need for,
and an ability to engage in lifelong learning) did not map to PE2020 Curious and Persistent
Continuous Learner. Additionally, there were four instances where the responding faculty came to a
consensus that there was no relation between pairs of outcomes (see Figure 3). Three out of four
instances relate to the alignment between PLO Human Cultures and three of the “hard” skills that
many engineering scholars agree are central to being an engineer (Shuman et al., 2005). These are
indicators of areas of potential misalignment. Additional evaluation and inquiry from Purdue
stakeholders will be required to address these potential concerns. The methods and results for
addressing the second project goal will be discussed next.
An archival analysis was completed to address the goal of determining which learning
outcomes are addressed in the FYE and the sophomore year of IE, and to what extent. Course
syllabi for the 19 courses of interest served as the primary data source for understanding which
outcomes were addressed in each course. Most of the syllabi directly referenced and included
information about ABET outcomes addressed. However, none of the syllabi-based course outcomes
information included references to the PE2020 outcomes or PLO. Such information would be
useful for determining relationships between courses, ABET student outcomes, PE2020 outcomes,
and the PLOs in the future.
The information collected and analyzed from the course syllabi was collated into a graphical
presentation of ABET outcome alignment by first- and second-year courses. Figure 4 (see below)
presents information on the relationships between ABET outcomes and courses in the first two
years of an Industrial Engineering undergraduate curriculum.
First Year Engineering
Industrial
Engineering (IE)
Sophomore Year
Science
2nd
1st
Elec. Semester Semester
A
B
C
D
ABET Student Outcomes
E
F
G
H
I
J
K
MA161/165
CHM 115
ENGR 131
ENGL 106
MA 162/166
ENGR 132
PHYS 172
CHM 116
CS 159
IE 200
IE 230
IE 330
IE 343
MA 261
MA 265
NUCL 273
ME 270
Figure 4 - Relationships between ABET Student Outcomes and Courses in IE Curriculum
Figure 4 shows that almost all of the ABET outcomes are addressed in both of the academic
years that are of interest in this study. ABET outcome D is addressed in FYE, but is not addressed
in the IE sophomore year. The ABET outcomes were reported in the syllabi as being addressed 50
times during the first two years of the IE undergraduate curriculum. Of these 50 instances, a
majority (29) of them are integrated into the first year engineering program. ENGR 131 and ENGR
132 address the most (9) ABET student outcomes in a single course. The fewest ABET outcomes
are incorporated into ENGL 106, a non-science, non-engineering, non-mathematics course.
Figure 5 provides a second view of how often, and in which courses, the ABET outcomes
are integrated into the courses taught during the first two years of an IE curriculum.
Frequency of ABET Outcomes Addressed in Courses
0
2
4
6
8
10
12
14
16
(a) An ability to apply knowledge of mathema cs, science, and
engineering
(e) An ability to iden fy, formulate, and solve engineering problems
(b) An ability to design and conduct experiments, as well as to analyze
and interpret data
ABET Outcomes
(f) An understanding of professional and ethical responsibility
(k) An ability to use the techniques, skills, and modern engineering
tools necessary for engineering prac ce
(d) An ability to func on on mul -disciplinary teams
(g) An ability to communicate effec vely
Technical, NonEngineering Courses
(CHM 115 & 116; CS 159;
MA 161/165, 162/166,
261, 265; PHYS 172)
Engineering Courses
(ENGR 131 & 132; IE 200,
230, 330, 343; ME 270;
NUCL 273)
Non-Technical Course
(ENGL 106)
(h) The broad educa on necessary to understand the impact of
engineering solu ons in a global and societal context
(i) A recogni on of the need for, and an ability to engage in life-long
learning
(j) A knowledge of contemporary issues
(c) An ability to design a system, component, or process to meet
desired needs
Figure 5 - ABET Outcome Alignment to Courses by ABET Outcome
ABET outcome A is introduced into 15 out of 17 courses IE students take during their first
two undergraduate years. No other outcome is referenced more than six times in the 17 courses; the
second most frequently referenced Outcome is Outcome E. Four out of eleven outcomes (i.e. A, B,
D, H) are addressed in the technical, non-engineering courses. ABET outcomes A and B are
introduced more often in technical, non-engineering courses than in engineering courses. ABET
outcomes C and J are only referenced or introduced twice across the 17 courses during this time
period. ABET outcome G is the only outcome integrated in both engineering and non-engineering
courses.
While the first part of this project goal (i.e. determining which outcomes are addressed) was
completed, the data required for the second part (i.e. determining the extent to which outcomes are
addressed) is not available for FYE. ABET assessment data was going to be used to determine this
information. Upon further investigation, the researcher learned that the FYE program is not ABETaccredited since it is not one of the professional engineering schools, and thus, the desired ABET
assessment data does not exist. This data was going to be used to develop a computer simulation to
address the third project goal. The methods for addressing the third goal are discussed next.
In lieu of this, however, the existing data on ABET outcomes addressed in courses (from the
course syllabi) were used to develop a computer software process model capability using ArenaTM
simulation software. Figure 6 is a screenshot of the flow process represented in the main Arena
simulation environment. In this process oriented model, a modal undergraduate IE students is the
entity flowing through the system. The courses in the FYE program and sophomore IE curriculum
are the processes the student undergoes. The 16 courses included in this process model make up the
intended IE undergraduate curriculum. The ABET outcomes addressed in each course are assigned
to each course.
Figure 6 - Screenshot of Arena Process Flow Model for Modeling Learning Outcomes
Transformations within First Two Years of the IE Undergraduate Curriculum
While not shown here, the complete model includes 11 dynamic plots of each ABET
outcome that animates while Arena™ runs. More specifically, as the student moves through each
courses in the curriculum, the value on the plot changes depending on whether the ABET outcome
of interest is addressed in the course or not. Each graph reveals the extent to which the outcomes
are addressed over time. In addition to a dynamic plot for each of the 11 ABET outcomes, there is
another dynamic plot that depicts all of the ABET outcomes on one graph. Figure 7 is a screenshot
of the simulation while it is running.
Figure 7 - Screenshot of Arena Simulation for Modeling Learning Outcomes Transformations
Captured During Process Model Runs
SUMMARY OF IMPACT AND FUTURE ACTIVITY
The development of this capability provides a working demonstration that a software
program can model the dynamics of student exposure to ABET Learning Outcomes at a course-bycourse level with a semester level of analysis. This result helps improve the utility of the primary
results linking course syllabi to ABET Learning Outcomes, and provide additional insights regarding
effects at the curricular level. With the appropriate data, this capability has the potential to be
expanded into a simulation that provides predictive power and aids FYE and IE administrators in
decision-making. Additional simulation developments can examine the effects of differing teaching
styles, stakeholder expectations, and student preparations on curriculum effectiveness and Outcome
achievement. Future ENE or IE research is expected to address these simulation process
capabilities.
Lastly, the results of this study will be shared with FYE and IE administrators directly
involved with curriculum development and in conducting ABET self-studies. The data and analyses
presented here are incorporated as central elements of Jeremi London’s MSIE thesis. Dissemination
of the research findings beyond Purdue has already begun. Two papers, related to project goals 1
and 3, will be presented at the 2013 Industrial and Systems Engineering Research Conference in
May. The hope is to gather additional data, build on the current findings, and publish the findings in
the Journal of Engineering Education.
REFERNCE
Shuman, Larry J., Besterfield-Sacre, Mary, & McGourty, Jack. (2005). The ABET "Professional
Skills" -- Can They Be Taught? Can They Be Assessed? Journal of Engineering Education, 94(1),
41-55.