ETD 435
STIMS – the Sustainability, Technology and Innovation
Concurrent Master's Degree: Assessments and Outcomes
James L. Barnes, Ed.D., Susan K. Barnes, Ph.D., Michael J.
Dyrenfurth, Ph.D., Gareth O’Donnell, Ph.D., Kathryne A.
Newton, Ph.D., Robert J. Herrick, Frank Palisi, M.S.
Barnes Technologies International, LLC/ Barnes Technologies
International, LLC/Purdue University/Dublin Institute of
Technology/ Purdue University/ Purdue University/ Purdue
University
Abstract
This paper describes the assessment model used to document the impact of a European-United
States Atlantis grant project awarded by the Fund for the Improvement of Postsecondary
Education (FIPSE, USA) and the Education, Audiovisual and Culture Executive Agency
(EACEA, EU). The results reported in the paper reflect progress through Year III of a four year
project. The partner institutions included Dublin Institute of Technology (DIT), Universitat
Politècnica de Catalunya (UPC), and Purdue University (PU). Two goals of the project were to
advance sustainable full-semester student exchange between the European project participants
and their US counterparts and to accelerate the development and support of collaborative cross
cultural, multi-disciplinary learning environments focused on innovative Engineering, Design
and Technology. The result was a highly successful concurrent (dual) master’s degree where,
upon successful program completion, each student earns two masters degrees, one from Purdue
University, and the other from the European university of their choice.
To measure the extent to which the project goals were met, a process-outcome evaluation design
was used. Both qualitative and quantitative methods and measures were used to evaluate the
degree to which the team accomplished the proposed goals. Qualitative measures were used to
report outcomes to the project team as to contextual/environmental, input/communication,
process and outcome variables that were assessed, and used to strengthen the program on an
ongoing basis. In addition, quantitative measures were used to conduct performance outcomes
evaluations of the program’s goals and objectives that are required by the US Department of
Education’s FIPSE Program. The findings provided information to the principal investigators on
how well the program’s goals and objectives had been met and what adjustments must be made
to fill gaps in the program. The evaluator also evaluated the project in light of two of the
Government Performance and Results Act of 1993 (GPRA) measures and found strong support
Proceedings of the 2014 Conference for Industry and Education Collaboration Copyright
©2014, American Society for Engineering Education
ETD 435
that the program is making more than adequate progress on the two GPRA performance
measures.
Introduction
Globalization, technological innovation and sustainability are critical issues that continue to
impact the critical infrastructure of most nations in the world. Significant to this concern is the
lack of preparation of leaders with significant capabilities in these areas. In the Americas and in
Europe, advanced programs in leadership development are often configured as master’s degree
programs – both conventional and professional [1]. There exists a wide range of masters
programs that include professional masters, MBAs, online degrees, and conventional campusbased experiences. But, few if any focus specifically on the intersection of Technology –
Globalization – Innovation and Sustainability!
The Sustainability, Technology and Innovation Concurrent (Dual) Master’s Degree (STIMS)
Program incorporates the best of what is known about technological innovation, research and
development, and sustainability [2], [3], [4], [5], [6], [7], [ [8]). The program was also designed
around the best practices of joint, dual, concurrent degree international programs [9], [10], [11],
[12], [13], [14], [15], [16].
To address this critical need Purdue University (USA), the Dublin Institute of Technology
(Ireland), and the Universitat Politècnica de Catalunya (Spain) proposed and won a joint
Education, Audiovisual and Culture Executive Agency (EACEA, EU) and the US Fund for the
Improvement of Postsecondary Education (FIPSE, USA) Atlantis project. This grant provided
funding for the development and implementation of a transatlantic four-semester dual master’s
degree initiative across the three-institution consortium. To highlight the success of this FIPSE
Atlantis Program, key background information, outcomes, and results, and the assessment model
are presented in this paper through Year III of this four year project.
Project Objectives & Outcomes
This project’s chief objective and outcome is the exchange of master’s level graduate students
among its three partners. These students will benefit from an innovative concurrent master’s
degree program that develops leading edge understandings and skills with technology,
innovation and sustainability and that also promotes transatlantic mobility. By design the
outcome will include global perspectives, multiple culture awareness, and sensitivities, as well as
dual professional level language capabilities. Important additional outcomes include:
Scholarship pertaining how to do effective exchanges and promote international
collaboration
Collaborative research and teaching based on increased mutual understanding and faculty
contact due to mobility
Proceedings of the 2014 Conference for Industry and Education Collaboration Copyright
©2014, American Society for Engineering Education
ETD 435
Better administrator, faculty and student understandings of cultures and global
perspectives
Enhanced procedures in place for collaboration and exchange due to increased
transparency, and reciprocal recognition of credits
Increased transatlantic faculty and administrator mobility
MS Degree Outcomes
Upon completion of the Masters of Science in Technology Degree focused on Technology
Innovation and Sustainability, students will demonstrate:
1. Enhanced capability with research and development. They will:
become familiar with research and experimental design sufficient to apply it to real world
problems
be able to apply design and development procedures to real world problems
demonstrate the necessary professional, research and development skills common to
engineering and industrial technology disciplines, as evidenced by successful completion
of either a thesis or a directed project that are required for successful life-long learning
and professional contribution.
2. Global perspectives on technology, technology management and sustainability. They will be
able to:
employ project management, technology assessment, and sustainability analyses
applicable both to their native cultures as well as other cultural settings
establish the appropriateness of technology for specific cultural settings
use knowledge of how industrial technology impacts society and organizations, from both
a technical as well as a leadership and management perspective
demonstrate critical thinking in these arenas.
3. Innovation and related process skills. They will be able to:
employ key creative and innovation generation procedures
sophistically retrieve information from databases and global sources
advance product realization and commercialization skills
4. Awareness of and capability with entrepreneurship procedures. They will be:
capable with entrepreneurial procedures and skills
able to secure business related information from sources around the world
5. Enhanced cross cultural communication and professional effectiveness. They will able to:
become proficient in professional communication (reading, writing and
speaking/presenting) in at least two languages
demonstrate effectiveness in operating in a culture/country other than their native one.
demonstrate ethical leadership and a commitment to their personal professional
development and life-long learning.
6. A graduate level of technological expertise in one or more of the technology fields.
Proceedings of the 2014 Conference for Industry and Education Collaboration Copyright
©2014, American Society for Engineering Education
ETD 435
Approach to Evaluating Program Impact on Student Learning
Barnes Technologies International LLC (BTILLC) evaluated the STIMS Program based on how
the Project Director and STIMS Program leadership team developed their implementation plan.
BTILLC evaluated how the project’s management team planned the system of input,
communications channels, and processes used to establish a baseline. BTILLC combined
Stufflebeam’s Context, Input, Process and Product Evaluation Model (CIPP) [17], [18],
Kirkpatrick’s Four Levels of Evaluation Model (reactions, learning, transfer, and results) [19],
and Wilder’s Collaboration Inventory Factors (environment, membership characteristics, process
and structure, communication, purpose, and resources) (Mattessich, Murray-Close, & Monsey
[20]; Meyerson [21] to conduct the third party program evaluation of the impact of FIPSE
Atlantis Concurrent MS Degree Project. The CIPP Model provides comprehensive approach to
evaluating programs, projects, personnel, products, institutions and systems. This evaluation
approach is built on the assumption that anything that can be evaluated can be successfully
evaluated at various stages of its development, context, input, process, and product. Simply put,
the CIPP Model focuses on “What needs to be done?”, “How should it be done?”, “Is it being
done?”, and “Did it succeed?”
The Kirkpatrick model is based on four levels of evaluation. Each level represents a different
measure of the effectiveness of the program. BTILLC has successfully used this approach on
numerous third party program evaluations of company programs, educational projects, and
agency impact. The data collection methods to be used by BTILLC during this third party
evaluation have been used in similar evaluations. The Wilder model used in this evaluation is
based on five measures.
Level I – Reactions evaluation measured how participants in the STIMS Program reacted
to it. It attempted to answer questions regarding the participants’ perceptions. Did the
team members feel they accomplished the STIMS Program goals and objectives? Is the
STIMS Program relevant to their proposed goals and objectives? How did participants
react to their concurrent MS Degree experiences?
Level II – Learning evaluation assessed the extent to which teachers have advanced their
skills, knowledge, or attitude. Methods of evaluation for Level II included both formal
and informal evaluations, as well as team assessment, and self-assessment.
Level III – Transfer evaluation assessed the change in behavior that has occurred during
the development and implementation processes due to the STIMS Program.
Level IV – Results evaluation measured the final results that occurred because of
processes used by the STIMS Program leadership team [19].
Each of the evaluation measures were analyzed based on short-term, intermediate, and longerterm outcomes. BTILLC has used this model to identify changes that are logically expected as
Proceedings of the 2014 Conference for Industry and Education Collaboration Copyright
©2014, American Society for Engineering Education
ETD 435
result of the project’s activities, are within the sphere of influence of the project, and generally
accepted as valid by the various stakeholders of the STIMS Program.
This integrated program evaluation model assessed to what degree the STIMS Program provides
a value added experience for the participants due to a comprehensive, coherent and sustainable
plan. At the heart of every phase of this external evaluation was assessing the degree to which
the STIMS Program team supported their participants through proven strategies and programs.
Key variables, based on the goals and objectives of the project for context, input, process and
product, were evaluated to assess to what degree the STIMS Program was meeting the goals and
objectives.
BTILLC supported Kirkpatrick’s four levels of evaluation with Wilder’s six factors of evaluation
that were broken down as short-term, intermediate, and longer-term outcomes. This integration
provided a more robust evaluation technique to assess to what degree the STIMS Program is
meeting its goals and objectives.
Short-term outcomes – refers to change in knowledge or awareness that typically
precedes change in behavior or practice.
Intermediate outcomes – refers to behavioral change in knowledge and awareness.
Longer-term outcomes – refers to more global change, either affecting more people than
those directly served or they involve more profound and lasting changes that the program
directly influences [21].
The findings and results of the qualitative and quantitative assessment have provided
recommendations to the principal investigators for how well the program’s goals and objectives
have been met and what adjustments must be made to fill in gaps.
Stock and Flow Model
The stock and flow model is used to convey the relationships and dynamic interactions of
components of a system. It is a modeling technique for designing, interpreting, and discussing
complex problems and the behavior change of a system. BTILLC used stock and flow modeling
to study the relationships and dynamic interactions between Stufflebeam’s CIPP and
Kirkpatrick’s Four Levels models of program evaluation with the program outcomes and
instruments used to assess them, which is presented in Figure 1.
Arrows indicate the direction of the relationships. The stock and flow model highlights the
efficiency of using one instrument to measure more than one outcome. Faculty interviews, for
example, provided data for examining four of the program outcomes. Note that the
Stufflebeam/Kirkpatrick evaluation models are housed in the same block and that the
relationship with outcomes is bidirectional, indicating that all levels of evaluation are used to
Proceedings of the 2014 Conference for Industry and Education Collaboration Copyright
©2014, American Society for Engineering Education
ETD 435
inform the outcomes over the period of the activity. Also, all levels are used to interpret the
meaning of the results.
The Results and Findings
Administration and partnership development
The FIPSE Atlantis Concurrent Degree Program evolved out of the relationship and trust built by
Dublin Institute of Technology (DIT) and Purdue University (PU) through their work on the
previous FIPSE-funded Design, Entrepreneurship, Technology, Engineering, Collaboration,
Transatlantic (DETECT) Exchange Mobility project. Universitat Politècnica de Catalunya
(UPC) was asked to join DIT and PU in this concurrent degree effort because of their strong
background in sustainability. DIT already had an established relationship with UPC that made
establishing them as a partner an easy process. From the onset, the three institutions involved
their executive administration and that led to a strong foundation and commitment being
established for STIMS program.
Outcomes
PreDeparture
Learning
Gain
Evaluation of
Exchange
Innovation
Gain
Faculty
Interviews
Cultural
Sensitivity
Leadership
Interviews
Knowledge
Transfer
European
Qualifications
F
k
Sustainability
Stufflebeam/Kirkpatrick
Context
Reactions
Evaluation
Input
Learning
Evaluation
Process
Transfer
Evaluation
Product
Results
Evaluation
Results/Accomplishments
Instruments
Figure 1. Stock and flow model indicating the relationships between the instruments, outcomes,
and evaluation models.
Proceedings of the 2014 Conference for Industry and Education Collaboration Copyright
©2014, American Society for Engineering Education
ETD 435
Collaboration
Evaluators used the Wilder Collaborative Factors Inventory Areas to establish baseline data for
the level of collaboration with the project leaders. Areas of concern in the first two years of the
project were the scores for collaboration factors measuring pace of the project and the
sufficiency of funds, staff, materials, and time. This year very few project leaders participated in
the Wilder Collaborative Factors Inventory. However, BTILLC did investigate those
collaboration components through other methods, including conducting interviews, making
direct observations, and analyzing other anecdotal data. The results of this investigation
provided evidence that these concerns have been adequately addressed and that the
administration of the STIMS Program continues to work well. The leadership teams from PU,
UPC, and DIT built a strong working relationship through continuous and open communication,
numerous face-to-face and electronic meetings, faculty exchanges, and a high level of senior
administrative involvement. The strengths and expertise of each university complement each
other, resulting in a strong leadership team that continues to sustain the project.
Communication
One of the key success elements of projects such as the STIMS Program is a strong
communication network among the partners that allows for leaders of each institution to have an
opportunity for input into the planning process so that they develop ownership in the project.
Thus, a strong trust relationship is built to assist the leadership team to resolve issues when they
occur. The team established a variety of mechanisms to ensure an open dialogue amongst the
partners to discuss and develop the key components of the STIMS Program. Each institution
established a prime point of contact and they involved university personnel at all necessary levels
of each university in the communication network. A critical element was the establishment of
significant communications at the beginning of the project and during the first few years to help
establish these three graduate level programs in their respective institutions and work together to
solve the various complex issues that arose during the project’s implementation. Key participants
met regularly, either in face-to-face-meetings, e-mail, telephone, or via video conferences using
such technologies as ooVoo video call. PU established a SharePointTM account to store and share
STIMS Program documents. Each institution created a website, linking universities and to other
key links necessary to administer the project.
BTILLC analyzed communications factors using the interviews with the leadership team to gain
an understanding of how well the STIMS Program communication process was working. The
leadership from each institution indicated that academic officers, faculty, and the International
Programs Office have continued to be key university personnel involved in the communication
process. In addition to the academic officers, faculty, and the International Programs Office
being the key university personnel involved in the communication process, the leaders, as in
2011, also indicated more involvement from the president’s/rector’s office and the registrar’s and
Proceedings of the 2014 Conference for Industry and Education Collaboration Copyright
©2014, American Society for Engineering Education
ETD 435
record’s office. The project’s leadership felt that the students also were a major part of the
communication process.
In the Year III interviews, team members concurred that the openness of the communication
process amongst the STIMS Program partners remained a strength of the program. Overall,
members who participated in the interviews felt they were as informed as often as they should be
about the project and that information was exchanged in both formal and informal ways.
Planning vision and process
Implementing an effective and efficient planning process is another key element to the success of
any project of the nature of the STIMS Program. The planning process must follow a logical
pattern towards implementation and must be designed so that all partners, at all necessary levels,
have ample opportunity for input. The process must assure that the project is adequately staffed
and funded and must also include measures for sustainability beyond the life of the grant period.
Finally, the planning process must include the development and signing of all necessary
governing documents by each institution.
When STIMS Program leaders were asked how well the planning process was organized and if it
followed a logical pattern, they agreed the process was working well. They explained that the
planning meetings were scheduled to allow the partnership goals and objectives to be
accomplished on time and that responsible parties were able to keep up with the work necessary
to coordinate all the people, institutions, and activities related to the project. The leaders
responded that the planning process had equal participation from all its partners and that the
project was supported by an adequate number of people with the requisite skills and expertise.
Most of the program leaders responded that the program had adequate funding.
The STIMS Program leadership continues to give credit for the well-articulated design and
implementation plan to the strength of their partnership. The evaluator feels these opinions can
be attributed to the fact that PU and DIT already had a strong working relationship and that they
looked carefully to find an appropriate partner with comparable academic rigor and the ability to
contribute the missing component of a campus that operates totally on sustainability principles.
The evaluator, through his interaction with the partners, has seen significant evidence that the
partnership is stronger in Year III than it was in previous years. Because of the strength of the
partnership and the trust established by the key people from each institution, the leaders
overwhelmingly felt that the design and implementation plan has clear long-term return on
investment and tangible outcomes. These responses are consistent with the leader’s responses
from Years I, II, and III.
Curriculum alignment and approval
Over the past year the three institutions have met repeatedly either face-to-face or through videoconferencing to develop, gain approval, and implement the STIMS Program curriculum. In Year
Proceedings of the 2014 Conference for Industry and Education Collaboration Copyright
©2014, American Society for Engineering Education
ETD 435
I, PU and UPC already had curriculum in place, but DIT needed to develop new courses and
have them approved, which was accomplished. The result of DIT’s work was the development
of six programs of study. The six different programs of study were developed to address the
variations as to when students started the STIMS Program, fall or spring, and on which of the
three campuses they began the program.
Key to the success of implementing new programs, especially programs that have never been
implemented at an institution, can be the considerable task of having the new program approved.
None of the three institutions had previously implemented a concurrent international master’s
degree program, therefore, each institution was embarking on new territory. All three
institutions successfully had their programs approved. This success was due to three key factors:
1) ongoing communications amongst the institutional Atlantis leadership teams and with their
respective administrations; 2) having all the appropriate university administrators and faculty
involved from the program’s onset; and 3) having the Memorandum of Understanding (MOU)
and other governing documents in place.
Each year, new students began their program of study at their home institution. Based on the
reports of the project leaders and students the curriculum implementation in Year III went very
well. Students were aware of when and where they were to exchange and the courses they were
to take when and where. They were also aware of the language fluency requirements and of the
preparation needed to be successful in a country where their native language is not the
predominate language of the community.
Marketing recruitment
The leadership team was asked what they thought about using a variety of the recruitment
techniques to make students aware of and interested in the STIMS Program. The leadership
identified five key recruitment techniques as being the most important for the STIMS Program:
Graduate Program Office announcements, college announcements, student interviews, in-class
announcements, and departmental websites.
Admissions
Agreements have been made on the admissions standards and processes to follow so that
students in the program are admitted to all three partner universities. In addition, the three
partner institutions have finalized the admission criteria to be used between each university for
the admission of students into all three programs.
When thinking about the program outcomes and the selection of students to participate in the
STIMS Program, the leaders identified four key areas for the selection of students: undergraduate
GPA, grades in specific courses, field-specific experience, and student attitude.
In the fall semester 2011, the beginning of Year II, ten students began the STIMS Program − five
from PU, five from UPC and none from DIT. In the spring semester 2012, DIT enrolled an
Proceedings of the 2014 Conference for Industry and Education Collaboration Copyright
©2014, American Society for Engineering Education
ETD 435
additional four students. By the end of Year III (Table 1) 45 students were participating in the
Atlantis Concurrent MS degree program. The leadership team needed to keep the student
enrollment manageable and work hard to recruit students. A breakdown of student mobility by
institution is detailed in the table below. One PU student dropped out of the Atlantis program
during her final semester. BTILLC interviewed the student and concluded it was based on two
factors: 1) giving up family for one year and 2) equipment necessary for her to complete research
was not the same at each institution.
Table 1.
Atlantic Concurrent Degree Program Student Enrollment Years II and III
Institution
PU
DIT
UPC
2010-2011
2011-2012
Completion
Fall
Semester
Spring
Semester
Fall
Semester
Spring
Semester
6
1
1
0
7
0
5
0
6
5
4
0
Fall
Semester 2012
PU/DIT PU/UPC
0
0
0
0
0
0
Spring
Semester 2012
PU/DIT PU/UPC
5
0
0
0
0
1
Language and cultural development
The language component, and eventually language gain of the students, is a crucial matter to
address when establishing an international concurrent degree program, especially when one
partner language is different from the other institutions. PU requires a Test of English as Foreign
Language (TOEFL) score for all international students. DIT and UPC have similar requirements
for their international students. As of Year III, the three institutions are currently exploring
language development and support options to help students during their studies.
The students were asked three questions about their foreign language capability upon entering
the STIMS Program and two questions towards the completion of their first year in the program.
Upon entering the program, BTILLC had students report their language fluency and their ability
to read and explain scientific and technical information in a foreign language. As the students
completed their first year of study, BTILLC asked the students about their level of confidence
while living and studying in a foreign country to see if there was any difference in their
confidence level since entering the program. Most students, when entering the program,
indicated they had studied a foreign language, but did not speak it well. They responded neutral
to being able to read and explain scientific and technical information in a foreign language.
When surveyed at the end of their first year of study, four students agreed, four students were
neutral, and one student disagreed that they felt confident with their foreign language fluency
while living in their exchange country. They also indicated similarly about their confidence
level with their foreign language fluency for studying in the exchange country. In Year III,
language fluency continued to be a concern. While graduating students indicated they managed
to survive language barriers, they, as well as other students participating in the program,
Proceedings of the 2014 Conference for Industry and Education Collaboration Copyright
©2014, American Society for Engineering Education
ETD 435
indicated some problems studying, reading and explaining scientific and technical literature in
their exchange language.
Faculty development
Most of the STIMS Program leadership work has focused on building a strong foundation
through a strong planning process that has included some effort towards faculty development.
The roots of a strong faculty development component need to be established during the initial
start-up year, and they have been established. Faculty development increased as the program
matured and faculties were provided opportunities to collaborate on course development,
research, conference papers, and other scholarly activity.
The leadership of all institutions visited each other’s campuses to explore potential faculty
development opportunities. Faculty development and mobility continues to be an ongoing
discussion during the leadership planning meetings. When the faculty leaders were asked about
faculty mobility activities, the majority of the leaders responded that the partnership had made
very good progress toward establishing faculty mobility activities. Last year, the leadership team
developed a Sustainability Technology and Innovation (STI) Masters Faculty Mobility Proposal,
a STI Masters Summary for Faculty, and an Atlantis Faculty Mobility Application. This year,
the team implemented this mobility program. Evidence that the program is working well
includes extended faculty exchanges and collaborative research papers and presentations.
Sustainability
Sustaining an initiative beyond the grant funding period is an important consideration that needs
to be built into the initial planning process. The STIMS Program leadership team has
successfully established a strong foundation that should lead to the program being sustainable.
During the first two years of the project, the STIMS Program leadership team identified four key
areas where there was opportunity for continuing growth and development. These areas include:
Learning outcomes assessment
Faculty mobility
Directed project/thesis
Language preparation
In the evaluator’s review of project documents, evidence was found to support the conclusion
that the leaders have made adequate progress in addressing these four areas. In addition to the
documents developed in the first years, the evaluator worked with some faculty to develop
strategies to measure student learning through coursework. Last year, project leaders began to
investigate the potential for collaboration with institutions and organizations beyond higher
education. The project leaders continue to expand their list of United States’ companies with
facilities or financial interests in Ireland and Spain. Using this list, project leaders will generate a
plan to network with those companies to gain additional support for the Atlantis project.
Proceedings of the 2014 Conference for Industry and Education Collaboration Copyright
©2014, American Society for Engineering Education
ETD 435
Institutional change and support
Creating institutional changes goes hand-in-hand with developing sustainable programs. This
initiative matches the strategic plans of the three institutions. In addition, this program fits well
within the direction of accreditation.
This year, BTILLC talked to project leaders about their reflections regarding institutional support
and institutional change. The leaders of the STIMS Program agreed that they were receiving
institutional support. When asked to think about the program outcomes for learning gains that
lead to institutional change and to identify factors that would mostly likely lead to institutional
change, the teams expressed the importance of expanding alliances with other institutions and
expanding institutional visibility. When discussing the program outcomes relative to learning
gains for faculty and administrators, the leadership team members agreed that this component
could be accomplished best by sharing successful ways of implementing international student
exchange programs, sharing innovative pedagogical strategies among partner institutions,
developing new courses, and expanding collaborative conference presentations among partner
institutions.
Academic and intellectual achievement
All three institutions have agreed on the academic standards acceptable for the STIMS Program.
They have also agreed to a three-person oversight committee with representatives from each
institution. This three-person committee also has served as the directed project or thesis
committee.
BTILLC asked the leadership to think about the program outcomes relative to learning gains for
graduate students. The leadership team confirmed that the outcomes established early in the
program were still appropriate. These items are:
Developing an ability to speak another language well enough to function in another
country
Developing an ability to read and present in another language
Expanding cross-cultural and global awareness and experiences
Interacting with faculty
Independence/self-reliance
Continuous improvement in coursework
Learning under a different educational system
Developing new logic and problem solving strategies
Expanding ability to design/apply sophisticated methodological techniques
Expanding ability to think critically
Developing stronger oral communication
Developing stronger written communncation
Increasing accountability
Proceedings of the 2014 Conference for Industry and Education Collaboration Copyright
©2014, American Society for Engineering Education
ETD 435
Increasing educational aspirations
The leadership team was also asked to think about the program outcomes relative to the
assessment of learning gains and how that assessment should be accomplished. The STIMS
Program leadership felt most strongly that course-embedded assessments, such as facultydeveloped test/exams, host faculty assessment, student presentations, and faculty assessment, in
addition to language proficiency tests, were the most important factors to the assessment of
learning gains.
BTILLC asked the students about key factors that impacted student learning and academic
achievement. BTILLC used five questions for entering students to gain an understanding the
confidence of being prepared and succeeding in the STIMS Program. BTILLC followed up with
the students toward the end of their first year with nine questions about academic achievement.
Upon entering the STIMS Program, the students were asked about how confident they were
about their STIMS Program. All students either felt very confident or confident about the
educational experience they were beginning. The students were also asked about their
knowledge of global issues, processes, trends, and systems. Seven out of ten students responding
to the survey indicated they were confident with their knowledge of global issues, processes,
trends, and systems. Six out of ten students responded that they were confident with their
knowledge of other cultures. However, when asked about their knowledge about world
geography and conditions, the students were almost evenly divided between being very
confident, confident, or neutral. Finally, students entering the program were asked about
interacting with professors who will teach them during their FIPSE STIMS Program. Eight out
of ten students indicated they felt confident interacting with professors who would teach them
during their STIMS Program.
At the end of their first year in the STIMS Program, the students were asked if the expectations
they had when entering the program were being met. Five out of nine students responding to the
survey indicated they agreed that their expectations were being met, while three students strongly
agreed and one student was neutral as to whether their expectations were being met by the
Atlantis Concurrent MS Degree program.
The students were also asked if the courses they had taken were meeting their educational needs.
All students either agreed or strongly agreed that the courses that they had taken met educational
needs. The students were questioned about their interactions with faculty. They also were asked
about faculty being available for help with coursework. The students responded identically to
these two related questions, four strongly agreeing, four agreeing and one neutral. BTILLC was
also curious if the students understood the requirements for earning a concurrent degree in the
Atlantis Program. The responses puzzled BTILLC, because this material was given to all
students, covered in orientation, and published on the STIMS Program website. Three students
strongly agreed, four agreed, one was neutral and one disagreed that they understood the
Proceedings of the 2014 Conference for Industry and Education Collaboration Copyright
©2014, American Society for Engineering Education
ETD 435
requirements for earning a concurrent degree in the Atlantis Program. BTILLC was interested in
how curious the students were about their exchange city and country. Four students indicated
they strongly agreed that they had been studying about the exchange city and country to better
prepare themselves for the exchange experience. Three students agreed and two students were
neutral about studying about the exchange city and country to better prepare themselves for the
exchange experience (Tables 2, 3, and 4).
Table 2
Student learning outcomes N=16
Question
The expectations that I set for myself before entering the Atlantis Concurrent Degree Program
are being met.
The courses that I have taken are meeting my educational needs.
The professors are effective teachers.
My research activities have been rewarding.
My interaction with faculty has met my expectation.
Faculty have been available for help with coursework.
I understand the requirements for earning a concurrent degree in the Atlantis Program.
I have been studying about my exchange city and country to better prepare me for my exchange
experience.
I feel confident with my foreign language fluency for living in my exchange country.
I feel confident with my foreign language fluency for studying in my exchange country.
SD
D
N
A
SA
0
0
2
13
1
0
0
0
2
1
0
0
0
1
0
2
1
1
0
9
5
6
4
5
1
2
6
8
8
4
6
8
7
1
2
2
4
3
6
7
1
1
1
0
1
3
7
8
6
4
Table 3
Continuing Student Survey results, Fall 2011 – N=16, Spring 2012 N=23
Items
I have changed my expectations that I set for myself before entering the
Atlantis Concurrent Degree Program.
I feel that the concurrent masters degree is my best option for a master
degree.
The orientation information provided me by my home institution about
what I should expect during my exchange was helpful.
My research activities have been rewarding.
I have maintained contact with my advisor during the summer.
Faculty have been available for help with coursework during the summer.
I have been studying about my exchange city and country to better prepare
me for my exchange experience.
I feel confident with my foreign language fluency for living in my
exchange country.
I feel confident with my foreign language fluency for studying in my
exchange country.
I feel confident of being able to read technical and scientific literature in
the language of my exchange country.
I feel confident of being able to explain technical and scientific literature
in the language of my exchange country.
I feel confident about my knowledge of global issues, processes, trends,
and systems.
I feel confident about my knowledge of other cultures.
I am acceptant of other cultures.
I feel confident about interacting with local citizens in another country.
I feel confident about my knowledge of world geography and conditions.
I feel confident about using local transportation systems in another
country.
I feel confident about taking excursions in another country.
SD
0
Fall Semester 2011
D
N
A
2
2
10
SA
2
SD
0
Spring Semester 2012
D
N
A
6
6
8
SA
3
0
0
2
4
10
0
0
3
9
11
1
4
5
5
1
1
5
7
9
1
0
1
0
0
0
4
3
0
1
5
6
1
10
1
4
13
5
5
3
2
0
0
0
0
0
1
1
1
5
12
11
2
14
7
8
11
4
3
3
9
0
3
4
7
2
1
5
3
12
2
0
5
5
5
1
2
6
3
9
3
1
4
4
6
1
2
8
2
6
5
1
6
5
3
1
4
8
4
4
3
0
0
3
8
5
0
1
0
14
8
0
0
0
0
0
0
0
0
0
0
2
0
0
3
0
9
2
8
11
9
5
14
8
2
7
0
0
0
0
0
0
0
0
0
0
3
0
1
1
0
13
4
9
12
7
7
19
13
10
16
0
0
0
5
11
0
0
0
6
17
Proceedings of the 2014 Conference for Industry and Education Collaboration Copyright
©2014, American Society for Engineering Education
ETD 435
I feel safe living in another country.
I know the key historical events of my exchange countries.
I know the key cultural events celebrated annually in my exchange
countries.
I know the types of government in my exchange countries and how they
function.
I know the major products manufactured in my exchange countries.
I know the major multi-national corporations that are based both in my
home and exchange countries.
I know the key technology or engineering careers in my exchange
countries.
I know the key occupations in my exchange countries.
I know the key tourist attractions in my exchange countries.
0
0
0
0
1
1
0
10
5
7
3
8
9
2
2
0
0
0
0
0
1
1
8
8
10
11
11
12
4
3
0
1
5
9
1
0
1
6
14
2
1
1
4
4
4
3
7
7
0
1
0
0
2
3
9
7
9
10
3
3
0
2
9
5
0
0
0
12
9
2
0
0
5
0
7
2
4
13
0
1
0
0
0
0
11
1
10
15
2
7
Table 4
Post Assessment of Completers N=5
Question
SD
D
N
A
SA
The expectations that I set for myself before entering the Atlantis Concurrent Degree Program
were met.
The courses that I took in my masters program met my educational needs.
The professors were effective teachers.
My research was rewarding.
My interaction with faculty has met my expectation.
Faculty were available for help with coursework.
I understood the requirements for earning a concurrent degree in the Atlantis Program.
I began my thesis/capstone research at the appropriate point in my studies.
My thesis/capstone advisor was very helpful to me throughout my thesis/capstone process.
My thesis/capstone research was well coordinated by my thesis/capstone committee.
The differences in laboratory equipment at the different universities hindered my ability to
complete my thesis/capstone research in a timely fashion.
I feel more confident studying in a foreign country.
I feel more confident with my foreign language fluency for studying in my exchange country
than I did when I began studying in my exchange country.
I feel more confident of being able to read technical and scientific literature in the language of
my exchange country than when I began studying in my exchange country.
I feel more confident of being able to explain technical and scientific literature in the language of
my exchange country than when I began studying in my exchange country.
I feel more confident about my knowledge of global issues, processes, trends, and systems than
when I began the Atlantis Concurrent Degree Program.
I feel more confident about my knowledge of other cultures.
I am more acceptant of other cultures.
I feel more confident about interacting with local citizens in another country.
I feel more confident about my knowledge of world geography and conditions.
I feel more confident about using local transportation systems in another country.
I feel more confident about taking excursions in another country.
I feel more comfortable living in another country.
I know more about the key cultural and historical events of my exchange countries.
Earning concurrent degrees, one from my home country and one from an exchange country, has
prepared me better to be successful in the workplace.
I made the correct decision earning my current masters degrees through the Atlantis program.
The transition between the three Atlantis universities worked well.
0
1
0
4
0
1
0
0
0
0
1
1
0
1
0
0
0
0
1
0
0
1
0
1
1
2
0
0
0
1
2
0
4
2
2
2
5
2
3
5
0
2
1
1
0
0
0
3
1
0
2
1
0
0
2
0
0
1
1
1
0
1
2
2
2
1
0
1
2
1
1
0
1
2
1
0
0
0
1
4
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
3
3
2
1
1
1
2
1
4
2
2
3
4
4
4
3
3
1
1
0
0
0
2
1
3
3
0
Equity in cost amongst institutions
Although the current exchange rates between the euro and the U.S. dollar has improved, strong
consideration needs to be made by the funding agency to adjust the amount of money provided to
U.S. students to compensate for the low value of the U.S. dollar in Europe. The allotted $12,000
Proceedings of the 2014 Conference for Industry and Education Collaboration Copyright
©2014, American Society for Engineering Education
ETD 435
in U.S. dollars is not equivalent to €12,000. The cost difference could greatly impact the level
of future students’ interest and their ability to gain a robust cultural and global experience.
Dissemination and reporting
The leadership team continues to actively disseminate the results of the STIMS Program. They
have established a website for the project. They have teamed to present papers at major
professional association conferences, such as the American Society for Engineering Education,
SEFI, and are currently working on several other international conference presentations. They
are exploring disseminating information about the project in scholarly journals. All institutions
have given internal presentations to various university constituencies about the program,
including several presentations to potential students.
Accomplishments
The STIMS Program leadership team has accomplished much through their third year of this
initiative. BTILLC, in their discussions with the leadership team and their analysis of pertinent
documents, find the following accomplishments were made by the STIMS Program team:
graduated their first class of students
continued to build trust in the already strong partnership
continued to develop and align relevant courses
executed and followed a signed MOU and other necessary by all three institutions
recruited an appropriate and manageable number of students to begin the program
strengthened the communication process and network
enhanced the recruiting and marketing materials, including a website
implemented further the faculty development and mobility component of the project
continued to strengthen the foundation for institutional change through this project
continued working on long-term sustainability strategies
continued a strong dissemination effort of information about the program to various
internal and external constituencies
moved in an appropriate pathway toward sustainability
Guided by the Outcomes—Capability Matrix, and evaluation determined how well the project
improved teaching and student achievement and is also addressed the two Government
Performance and Results Act of 1993 (GPRA) performance measures established for FIPSE
Atlantis:
1. the percentage of students pursuing a joint or dual degree who persist from one academic
year to the next (persistence), and
2. the percentage of students who graduate within the project's stated time for completing a
joint or dual degree (graduation).
Proceedings of the 2014 Conference for Industry and Education Collaboration Copyright
©2014, American Society for Engineering Education
ETD 435
A third party evaluation of STIMS provides strong support that the program is making more than
adequate progress towards the two GPRA performance measures for the project. The following
evidence supports that conclusion:
• To date, STIMS performance measures for GPRA 1 and 2 are 100 and 87.5 percent
respectively.
• Wilder’s Collaborative Factors Inventory shows that the STIMS team works well and is
working to sustain the program.
• The students have adapted well to studying under different educational systems.
• Of the eight students who began the program, all but one student completed their
concurrent dual degree on or before schedule.
• Of the other 26 students enrolled in STIMS all students are matriculating through the
program on schedule.
• No student, to date has failed a course or had particularly difficulty with a non-native
language.
• The only shortcoming found is the need for more Purdue students to obtain their second masters
at UPC.
Conclusion
The STIMS Program continues to be on a great track for being sustainable after the completion
of the grant funding cycle. The leadership team works well together, and they have done an
outstanding job building the kind of trust and foundation necessary to make this program highly
successful. The experiences gained by the leadership team and those within each institution that
has been a part of the development and implementation process will greatly help in Year IV.
Having the top university officials involved from the beginning of the initiative and keeping
them in the loop established a solid foundation for institutional change and sustainability, a real
strength of this program.
Therefore, BTILLC sees strong evidence that the right foundation was established for
sustainability once the project funding ends. However, sustainability will only occur if the key
personnel and administration learn from each year’s experiences and modify the upcoming year
based on this knowledge. During the third year of the program, the leadership team successfully
addressed four major tasks identified in earlier evaluations: create a plan to assess student
learning outcomes; implement a proposal to increase faculty mobility; finalize the expectations
of the student directed project/thesis; and explore ways to address foreign language preparation.
Central to becoming sustainable will be getting more departments, faculty, and students involved
in the STIMS Program. The team needs continuing to work on this urgent need. BTILLC is very
confident that the program is moving in the right direction and has successfully accomplished its
third year goals and objectives.
The contents of this paper were developed under an EU-U.S. Atlantis grant from the Fund for the
Improvement of Postsecondary Education, (FIPSE), U.S. Department of Education. However,
Proceedings of the 2014 Conference for Industry and Education Collaboration Copyright
©2014, American Society for Engineering Education
ETD 435
those contents do not necessarily represent the policy of the Department of Education, and you
should not assume endorsement by the Federal Government.
References
[1] Committee on Enhancing the Master’s Degree in the Natural Sciences, the Board on Higher
Education and Workforce Policy and Global Affairs. (2008). Science Professionals: Master's
Education for a Competitive World. Washington, DC: The National Academies Press.
[2] National Center for Best Practices. (2013, January). “Making” Our Future: What States Are
Doing to Encourage Growth in Manufacturing through Innovation, Entrepreneurship, and
Investment. Washington, DC: National Governor’s Association.
[3] Committee on the Engineer of 2020, Phase II, Committee on Engineering Education,
National Academy of Engineering. (2005). Educating the Engineer of 2020: Adapting
Engineering Education to the New Century. Washington, DC: National Academies Press,
http://www.nap.edu/catalog/11338.html
[4] National Academy of Sciences, National Academy of Engineering, Institute of Medicine.
(2007). Rising Above The Gathering Strom: Energizing and Employing America for a
Brighter Economic Future. Washington, DC: National Academies Press.
[5] National Academy of Sciences, National Academy of Engineering, Institute of Medicine,.
(2010). Rising Above The Gathering Strom, Revised: Rapidly Approaching Category 5,
Washington, DC: National Academies Press.
[6] National Research Council. (2005). Decision Making for the Environment, Washington, DC,:
National Academies Press.
[7] National Research Council. (2008). Research on Future Skill Demands: A Workshop
Summary. Washington, DC,: National Academies Press.
[8] Board on Testing and Assessment. (2011). Assessing 21st Century Skills: Summary of a
Workshop. Washington, DC: National Academies Press.
[9] Asgary, N., & Robert, M. (2010). A Cost-Benefit Analysis of an International Dual Degree
Programme, Journal of Higher Education Policy and Management, 32(3) 317-325. DOI:
10.1080/13600801003743513
[10] Bhandari, R. (2009, February). Key Research in U.S. Study Abroad: Findings from the
Institute of International Education's Study Abroad Capacity Series. Paper presented at
Proceedings of the 2014 Conference for Industry and Education Collaboration Copyright
©2014, American Society for Engineering Education
ETD 435
Emerging Directions in Global Education 2009 conference, Feb 9-11, New Delhi, India: lIE
(New York).
[11] Blumenthal, P., & Laughlin, S. (2009, March). Meeting America's Global Education
Challenge: Promoting Study Abroad in Science and Technology Fields. Number 5 in IIE
Study Abroad Series of White Papers on / Expanding Capacity and Diversity in Study
Abroad. New York: Institute of International Education.
[12] Committee on Enhancing the Master's Degree in the Natural Sciences, the Board on Higher
Education and Workforce Policy and Global Affairs. (2008). Science Professionals:
Master's Education for a Competitive World. Washington, DC: The National Academies
Press.
[13] Culver, S.M., Puri, I.K., Spinelli, G., DePauw, K.P.K., & Dooley, J.E. (2011, April 21).
Collaborative dual-degree programs and value added for students: Lessons learned through
the Evaluate-E Project. Journal of Studies in International Education, 20(10) 1-22. DOI:
10.1177/1028315311403934
[14] Marginson, S., & van der Wende, S. (2006, September). Globalisation and higher
education. [draft #2b, prepared for OECD]. Paris, France: OECD.
[15] National Association of International Educators (NAFSA). (n.d.). Washington, DC.
http://www.nafsa.org/index.aspx
[16] Obst, D., & Kuder, M. (eds). (2009). Joint and Double Degree Programs, An Emerging
Model for Transatlantic Exchange. New York: Institute of International Education.
[17] Stufflebeam, D. (2002). CIPP evaluation checklist: A tool for applying the fifth installment
of the CIPP model to assess long-term enterprises. Retrieved from
http://www.wmich.edu/evalctr/checklists/cippchecklist.htm
[18] Stufflebeam, D. (2004). The 21st Century CIPP Model: Origins, Development, and Use. In
M. Akin (Ed.). Evaluation roots: Tracing theorists’ views and thoughts (pp.245-266).
Thousand Oaks, CA: Sage.
[19] Kirkpatrick, D. (1996). Great ideas revisited. Training & Development, 50, 54-60. (2), 227240.
[20] Mattessich, P. W., Murray-Close, M, & Monsey, B.R. (2001). Collaboration: What makes it
work (2nd ed.). St. Paul, MN: Fieldstone Alliance.
[21] Meyerson, J. (2010). Smart measurement: Tools and tips for capturing results. Evaluation
Series Workshop. Retrieved from http://www.wilder.org/download.0.html?report=2299
Proceedings of the 2014 Conference for Industry and Education Collaboration Copyright
©2014, American Society for Engineering Education
ETD 435
JAMES L. BARNES, EDD.
As Director of Research and Business Development for Barnes Technologies International, LLC, Dr. Barnes
oversees all proposal development and research design. He also coordinates all needs assessment and benchmarking
studies. His current research examines naturalistic sustainability, how understanding systems and their interaction
influence solving complex global problems through solution innovation.
SUSAN K. BARNES, PHD.
Dr. Barnes is the Managing Director of Operations of Barnes Technologies International, LLC. She coordinates all
education and assessment and evaluation initiatives. Dr. Barnes specializes in instrument design, scale development,
and data analysis. Her research focuses on assessment and computerized testing.
MICHAEL J. DYRENFURTH, PHD
Dr. Dyrenfurth is professor in the Department of Technology Leadership and Innovation at Purdue University. He is
also Co-PI of two international exchange programs funded by FIPSE (Fund for the Improvement of Postsecondary
Education) and the European Union. Previously he served as the College of Technology' s Assistant Dean during
which he was responsible for the graduate program, securing approval for the College's Ph.D. program and
international initiatives.
GARETH O’DONNELL, PHD
Dr. O’Donnell is a lecturer in the Department of Mechanical and Transport Engineering at the Dublin Institute of
Technology. He is a He is also Co-PI of the Sustainability, Technology and Innovation Concurrent Master's Degree,
an international exchange program funded by FIPSE (Fund for the Improvement of Postsecondary Education) and
the European Union. His research interests include materials, machine design, and compaction and sintering.
KATHRYNE A. NEWTON
Dr. Newton is a professor of Industrial Distribution and Graduate Program Chair in the Department of Technology
Leadership and Innovation at Purdue University. Her research interests include resource development, operations
human, and quality management.
ROBERT J. HERRICK
Professor Herrick is the Robert A. Hoffer Distinguished Professor of Electrical Engineering Technology at Purdue
University. His research interests include electronic switching, the art and technology of teaching, and navigational
aid and communication systems.
FRANK PALISI, MS
Mr. Palisi is a recent graduate of the of the Sustainability, Technology and Innovation Concurrent Master's Degree
earned master’s degrees from Purdue University and the Dublin Institute of Technology.
Proceedings of the 2014 Conference for Industry and Education Collaboration Copyright
©2014, American Society for Engineering Education