Assessment Toolbox for
International Educators
Presenters:
David Comp, Darla K. Deardorff,
Elaine Meyer-Lee,
Lee Sternberger, Victor Savicki
Session Overview



Assessment Overview and Context
Highlights of Specific Assessment
Tools/Methods
Tools/Methods Exploration through
“Roundtable” discussions
Assessing International
Education: An Overview
Dr. Darla K. Deardorff
Duke University
d.deardorff@duke.edu
Why assess international
education?

Inform continuous quality improvement

Advocate for international education

Satisfy regional accrediting bodies


Add to basic understanding of student growth in
this area
Provide feedback to students on their personal
growth
Ways of approaching
assessment
Some key points…
ASSESSING OUTCOMES
Starting point:
Mission Statement
into
Goals
into
Measurable Objectives
Assessment Cycle






Define outcomes (based on
mission/goals) and establish
measurable criteria
Identify appropriate assessment
methods
Collect data
Analyze data
Reflect on needed changes
Design and apply changes
Who to involve?
Set up
Assessment Team
(include students, faculty,
stakeholders….)
Assessment Methods
Tools determined by clearly articulated
assessment questions –
What do we want to measure?
What is the evidence of student
success?
DIRECT METHODS & INDIRECT METHODS
= COMBINATION OF QUANTITATIVE AND
QUALITATIVE
DIRECT METHODS






Embedded course assessment
Portfolios
Performance
Testing
Papers/projects
Capstone
INDIRECT METHODS





Surveys (inc. self-assessments)
Interviews
Focus groups
Curriculum/transcript analysis
Documented data (student numbers)
Assessment – Key Points

Multi-method, multi-perspective

Ongoing

Integrated

Aligned
Questions to consider…



What are the specific goals and
objectives to be measured and why?
What does the tool/method
specifically measure?
Does the tool/method match goals
and objectives?
More questions to consider…

How will assessment data be utilized?
(How will data be used to provide feedback to
students? To improve the program/curriculum,
etc?)

Is assessment multi-method, multiperspective, ongoing, intentional,
integrated, part of strategic plan?
Assessment – Lessons Learned






Collaborate - Put together
Assessment Team
Adapt - Build on what you have
Measure – what is valued (align!)
Plan - Develop assessment plan
Use – use what you assess
Support – from leadership,
stakeholders
Deardorff, 2008
Global Perspective Inventory
(GPI)
Presented by
David Comp
The University of Chicago
dcomp@uchicago.edu
Global Perspectives Inventory (GPI)
Developed by Larry Braskamp, David
Braskamp & Kelly Carter Merrill.
-Larry Braskamp is a senior fellow at the
Association of American Colleges and
Universities (AAC&U). He is professor
emeritus at the School of Education at Loyola
University Chicago where he also served as
senior vice president for academic affairs for
four years. Braskamp also served as the first
executive director of the Council for Higher
Education Accreditation (CHEA).
Global Perspectives Inventory (GPI)
The GPI is a survey of 46 items specifically
designed to provide self-reports of students’
perspectives in three domains of holistic student
development--cognitive, intrapersonal, and
interpersonal.
The GPI can provide evidence of students’ global
perspectives:
• At entry, during and at graduation from college
• At entry and conclusion of an “education/study abroad”
experience
Braskamp, Braskamp & Carter Merrill, 2009
Global Perspectives Inventory (GPI)

Theoretical Background of GPI
• The construction of GPI is based on the developmental
perspective that students are on a journey during
college. In this journey, students are given
opportunities to reflect on three “big questions”:

“How do I know?” reflects the Cognitive dimension.

“Who am I?” reflects the Intrapersonal dimension.

“How do I relate to others?” reflects the
Interpersonal dimension.
Braskamp, Braskamp & Carter Merrill, 2009
Global Perspectives Inventory (GPI)
Description of the six GPI
scales



The Cognitive domain consists of two
scales: Knowing and Knowledge.
The Intrapersonal domain consists of
two scales: Identity and Affect.
The Interpersonal domain consists of
two scales: Social interactions and Social
responsibility.
Global Perspectives Inventory (GPI)
Validity and Reliability
• Reliability. Four GPI scales each loaded on their own independent
factors (some with the exception of one or two items) considering a
combination of both the highest and second highest loading scores:
Cognitive/Knowing, Cognitive/Knowledge, Intrapersonal/Affect, and
Interpersonal/Social Responsibility. The Intrapersonal/Identity and
Interpersonal/Social Interaction scales did not seem to load onto any
independent factor. The overlapping of scales on each of the factors is
not troubling, but rather encouraging. Theoretically, scholars agree
these factors would all contribute to one concept – global perspective.
• Validity. As students progress through college their development is
likely to increase (Baxter Magolda, 1992; Pascarella & Terenzini,
2005). Presumably, therefore, as people get older their global
perspective may increase as well.
Braskamp, Braskamp & Carter Merrill, 2009
Advantages and Disadvantages
Advantages
GPI can be administered at:


entry, during and at graduation from college
entry and conclusion of an “education/study abroad” experience
GPI has only 46 survey questions and brief demographic questions
GPI was developed by a leading expert in higher education
assessment
Disadvantages
GPI is a relatively new assessment tool
GPI has not been used in any research studies
Global Perspectives Inventory (GPI)
https://gpi.central.edu/
New Tools for
Intercultural Learning
Outcomes Assessment
Elaine Meyer-Lee
Director
Joy Evans
Assistant Director for Research and Scholarship
Saint Mary’s College
Center for Women’s InterCultural Leadership
Notre Dame, Indiana
Evaluation & Assessment Goals


Evaluate programs’ effectiveness
Measure Students’:
• Changes in sensitivity to cultural differences
• American identity development
• Own sense of growth toward goals

Explore correlations of change with:
• Intercultural engagement while abroad
(interaction and reflection)
• Program or demographic characteristics
Measure of Intercultural
Engagement
Asks students to quantify the
frequency of their engagement
through:



Interaction with cultural resources
Interaction with local people
Explicit reflection on these
interactions
American Identity Measure


Adaptation of Jean Phinney’s Multigroup Ethnic
Identity Measure (good track record), and her
much newer American Identity Questionnaire
developed for underrepresented groups
Based on Eriksonian identity development theory.
Measures 2 Factors:
• identity search (e.g. I have often talked to other people
about what it means to be an American.)
• affirmation, belonging, and commitment (e.g. Being
American plays an important part in my life.)
American Identity Measure


Adaptation of Jean Phinney’s Multigroup Ethnic
Identity Measure (good track record), and her
much newer American Identity Questionnaire
developed for underrepresented groups
Based on Eriksonian identity development theory.
Measures 2 Factors:
• identity search (e.g. I have often talked to other people
about what it means to be an American.)
• affirmation, belonging, and commitment (e.g. Being
American plays an important part in my life.)
American Identity Measure




Adapted it (with Phinney’s blessing) for study
abroad students because one goal is more
complexity in understanding one’s own culture
Students in returnee courses felt this national
identity version captured an important dimension
10-item Likert-style Self Assessment (5 for each)
Currently refining adaptation’s psychometric
properties, but alpha coefficients good so far
(.79+)
Sources


Phinney, J. (1992). The Multigroup Ethnic
Identity Measure: A new scale for use with
adolescents and young adults from diverse
groups. Journal of Adolescent Research, 7,
156-176.
Phinney, J., & Devich-Navarro, M. (1997).
Variations in bicultural identification
among African American and Mexican
American adolescents. Journal of
Research on Adolescence, 7, 3-32.
Measure of Intercultural
Engagement
Asks students to quantify the
frequency of their engagement
through:



Interaction with cultural resources
Interaction with local people
Explicit reflection on these
interactions

MEI
 Comparable &
program specific data
on students’
intercultural
interactions

AIQ
 Examine identity
dimension, connect to
literature on this
?
MEI
? Labor-intensive
development and
administration
?
AIQ
? Reliability and Validity of
adaptation still being
established
? Needs refining
Intercultural Leadership
Certificate Program




Capture Saint Mary’s strengths and CWIL’s unique location at the
intersection of intercultural (local and global) and leadership
education in the context of a women’s college
Guide students through a developmental process into a deeper
capacity for intercultural leadership
Bring coherence and provide an organizing framework for every
effort (curricular and co-curricular) being undertaken at Saint
Mary’s College to encourage intercultural leadership
Portfolio assessment of students’ proficiency as an intercultural
leader in
6 proficiency areas






Recognize the Leader Within
Articulate Your Ethical/Spiritual Center
Engage With & Value Diversity
Dialogue on Power & Privilege
Create Inclusive & Equitable Community
Make Your Difference in the World
For more handouts or bibliography on
assessing intercultural outcomes of
study abroad programs, go to
http://www.saintmarys.edu/~cwil/php/intercultural.learning/
IILOutcomes.php
or email me: meyerlee@saintmarys.edu
Sociocultural
Adaptation Scale
(SCAS)
Presented by
Vic Savicki, PhD
Sociocultural Adaptation Scale
(SCAS)



Developed by Colleen Ward and
Antony Kennedy
Ward, C. & Kennedy, A. (1999). The
Measurement of Sociocultural
Adaptation. International Journal of
Intercultural Relations, 22, 659-677.
Several subsequent supporting
studies…
ABC Theory of Acculturation


Ward, C. (2001). The A, B, Cs of
acculturation. In D. Matsumoto (Ed.),
Handbook of Culture and Psychology. (pp.
411-446). NY: Oxford University Press.
A Affect based on stress and coping
theory

B Behavior based on learning cultural
knowledge and skills

C Cognition based on social identification
theory..
Sociocultural Adaptation
Definition



SCAS measures the Behavior aspect
of the ABC’s
“The ability to ‘fit in,’ to acquire
culturally appropriate skills and to
negotiate interactive aspects of the
host environment.”
Applied to both foreign nationals
coming to the U.S and U.S. nationals
going abroad..
SCAS Scales


SCAS Total: Sum/average of 29 items
Two sub-scales based on a factor analytic
study
• Cultural Empathy and Relatedness (13 items)
(32.1% of variance)
 Understanding local perspectives, values, world
views, and communication
• Impersonal Endeavors and Perils (7 items) (8.7%
of variance)
 Management of impersonal interactions and/or
awkward situations..
Advantages and Disadvantages







Based on a theory of acculturation
Tracks key intercultural adaptation issues
Gives several scores to interpret
Predicts an overall progression of
adaptation
Can be used formatively and summatively
List may be incomplete
List may have cultural bias..
The Beliefs, Events, and Values Inventory
(BEVI)
Developed by Craig N. Shealy, Ph.D.
International Beliefs and Values Institute
James Madison University
shealycn@jmu.edu
Presented by Lee Sternberger, James Madison University
The Beliefs, Events, and Values Inventory
(BEVI)
What does the BEVI measure?
The BEVI…is designed to assess a number of processes relevant
to international and multicultural learning including (but not
limited to): basic openness; receptivity to different cultures,
religions, and social practices; the tendency (or not) to stereotype
in particular ways; self and emotional awareness; and preferred
but implicit strategies for making sense of why ‘other’ people and
cultures ‘do what they do’ (Shealy, 2005, p. 99).
The BEVI

Has been in development since the early 1990s

Empirically and theoretically grounded (e.g., in
Equilintegration or EI Theory and the EI Self;
Shealy, 2004, 2006, in press)

Strong evidence of good reliability and validity

Is derived from belief-value and life event
statements as well as background information items
The BEVI

Is comprised of two validity scales, eighteen
process scales, and three qualitative “experiential
reflection” items

Does not appraise “correctness” of individual
responses (e.g., whether responses are “right” or
“wrong”) but facilitates 1) comparisons between
individuals and groups to the normative sample
as well as 2) the association of response patterns
to a wide range of processes and outcomes (e.g.,
who learns what and why, and under what
circumstances)
The BEVI

Includes 65 background/demographic items, 334
belief, value, and life event items rated on a fourpoint Likert scale (strongly agree, agree, disagree,
strongly disagree)

Takes 45-55 minutes to complete

Is available online and accessible globally

Allows for a wide range of analyses, reports, and
feedback at the individual, group, and organizational
level
The BEVI

Sample Background Variables (65 on Current BEVI)
Age
Gender
Ethnicity
Citizenship
Countries Visited

Sample BEVI Scale Items (334 on Current BEVI)
There are too many foreigners in our country.
Everyone could benefit from learning more than one language.
Hinduism and Buddhism have much to teach the modern world.
If it comes down to it, the man must act as head of the house.
I worry about the health of our planet.
BEVI Scales
I.
Validity Scales
Consistency
Congruency
II. Formative Variables
Demographic and Background Items
1. Negative Life Events
III. Fulfillment of Core Needs
2. Needs Closure
3. Identity Closure
IV. Tolerance of Disequilibrium
4. Basic Openness
5. Basic Closedness
V. Critical Thinking
6.
7.
8.
9.
Causal Closure
Naïve Determinism
Refined Determinism
Socioreligious Determinism
VI. Self Access
10. Emotional Attunement
11. Cognitive Control
12. Self Awareness
VII. Openness to Others
13. Socioemotional Openness
14. Sociocultural Openness
15. Socioreligious Openness
VIII. Global Worldview
16.
Gender Awareness
17.
18.
Ecological Resonance
Global Engagement
IX. Experiential Reflection Items
The Forum BEVI Project
The Forum BEVI Project
SAMPLE BEVI STUDY
Mean Differences on Significant “Intercultural Openness” BEVI Items Between Participants
Who Have and Have Not Participated in an International Learning Program1
No Participation
(N = 460)
Some Participation
(N = 564)
We should try to understand cultures that are different from our own.
3.642
3.74 *3
Knowledge of other cultures helps me understand my own culture.
3.28
3.45 **
In this country, everyone should be required to speak the same language.
2.18
2.09 (p = .086)
Everyone could benefit from learning more than one language.
3.34
3.50 **
I enjoy learning about other cultures.
3.41
3.57 **
Too many foreigners live in our country.
2.16
1.93**
It is important to be informed about world events.
3.49
3.57 *
_______
_
the BEVI is a copyrighted instrument and these data are preliminary, items or data may not be copied, disseminated, or reported
without the written permission of Craig N. Shealy, Ph.D.
2 Scale Conversions: 1 = Strongly Disagree, 2 = Disagree, 3 = Agree, 4 = Strongly Agree
3 * p < .05, ** p < .001
1 Because
_
The Forum BEVI Project
Sample BEVI Study:
Experiential Reflection Item Responses
Question 1.
Which event or aspect of your international
learning experience had the greatest impact
upon you and why?
Interacting with the people of different cultures in their everyday lives
helped me to understand the motivations behind world affairs.
Everyday interaction with people from another culture and using a language
other than my own – it forced me to look at myself as an ‘other’ and to see
what someone else might see, and to critique my own culture and myself.
The Forum BEVI Project
Sample BEVI Study:
Experiential Reflection Item Responses
Question 2.
Was there some aspect of your own “self” or
“identity” that became especially clear or
relevant to you or others as a result of this
experience?
My feelings about my home country and certain cultural practices became
more clear. Because I had to really think about the origins and motives
behind many cultural practices, I was able to have clearer opinions about
them.
The experienced actually produced more ambiguity about my own identity.
Yes, I learned I want to further my education by helping other people.
The Forum BEVI Project
Sample BEVI Study:
Experiential Reflection Item Responses
Question 3.
What have you learned and how are you different as a
result of this experience?
I learned about what it means for me to be an American in this world of
increasing globalization.
I have learned that there is a global community. We are all people just living
our lives the best we can in different environment. We are required to help
each other enjoy all the benefits of life.
Your Turn….
Join us for
“Roundtable Discussions” on the
specific tools you’ve just heard
about… ask more questions, learn
more in detail about how and
when to use these instruments
Resources

ACE’s Internationalization website
http://www.acenet.edu/Content/NavigationMenu/ProgramsServices/cii/res/assess/in
dex.htm

NCSU’s Assessment Website
http://www2.acs.ncsu.edu/UPA/assmt/resource.htm

SIETAR Europa’s website
http://www.sietar.de/SIETARproject/Assessments&instruments.html#Topi
c26





Fantini’s List www.sit.edu/SITOccasionalPapers/feil_appendix_f.pdf
Forum’s Guide to Outcomes Assessment in Education
Abroad (Forum, 2007) – ed. by Mell Bolen
The SAGE Handbook of Intercultural Competence (Sage,
2009) – ed. by D.K. Deardorff
Deardorff, D.K. “Principles of International Education
Assessment.” In IIENetworker (Spring 2007).
Maki, P. Assessing for Learning (2004, Stylus)