Assessing CrossCultural
Competence:
How Good Are
the Available
Instruments?
Institute for CrossCultural Management
Florida Institute of
Technology
Authors
• William Gabrenya, PhD - Cross-Cultural
Psychology (USA)
• Rana Moukarzel, MS Industrial/Organizational Psychology
(Lebanon)
• Marne Pomerance, MS – I/O (USA)
• Richard Griffith, PhD – I/O (USA)
Florida Tech – Institute for Cross-Cultural Management
2
Support
• DEOMI Contract FA2521-10-T-0087
• Dan McDonald, PhD
• Patrice Reid, PhD
Florida Tech – Institute for Cross-Cultural Management
3
Cross-Cultural Competence
and the U.S. Military
• Cross-Cultural Psychology watches Iraq
disintegrate
– Case study in worst practices
– Sidelined culture experts
– How could this happen?
Florida Tech – Institute for Cross-Cultural Management
4
Culture Expertise
Rediscovered
•
•
•
•
•
Defense Language Office (DLO)
DEOMI
Army Research Institute
Services
DLO project to try to keep track of it all…
Florida Tech – Institute for Cross-Cultural Management
5
Three Issues in CrossCultural Competence
1.
2.
3.
4.
Identify competencies
Measure competencies
Train competencies
Utilize capabilities acquired at
considerable cost
Florida Tech – Institute for Cross-Cultural Management
6
DLO Framework
• “Framework for Cross-Cultural
Competency”
• Competency model
– Organized set of requirements for acceptable
performance
– Higher level generality to lower level, hierarchy
– Expressed behaviorally
• Two parts
– 3C competencies
– Antecedent variables (enablers)
• Revised
Florida Tech – Institute for Cross-Cultural Management
7
Competencies
Name (#
elements)
Example
C1
Culture-General
Concepts and
Knowledge (3)
Comprehends and navigates intercultural
dynamics
C3
Cultural PerspectiveTaking (3)
Understands how one’s own group is
viewed by members of another group
C4
Communication (2)
Acquires and applies knowledge and
concepts of intercultural communication
skills
C5
Interpersonal Skills
(2)
Builds relationships in support of mission
performance
C6
Cultural Adaptability
(2)
Understands the implications of one’s
actions and adjusts approach to maintain
relationships with other groups, or
Florida Tech – Institute for Cross-Cultural Management
cultures
8
Antecdents (Enablers)
Name (#
elements)
Example
E1
Cognitive Bias
Resilience (3)
Accepts, or does not feel threatened by,
ambiguous situations and uncertainty.
E2
Emotional Resilience
(4)
Tolerates emotionally shocking,
frustrating, or exhausting circumstances…
E3
Self-Identity
Resilience (3)
Demonstrates ability to maintain personal
values independent of situational factors
E4
Learning Motivation
(4)
Is motivated to make sense of
inconsistent information about social
rules and norms…
E5
Social Interaction (4)
Actively seeks out and explores
unfamiliar cross-cultural interactions …
Florida Tech – Institute for Cross-Cultural Management
9
Validation* of the Framework
• Content validity
– The right competencies?
• Criterion validity
– Are these competencies related to performance
and adjustment?
• *Analogously
Florida Tech – Institute for Cross-Cultural Management
10
Content Evaluation of the
DLO Framework
• Developed using subject matter experts
(SMEs):
– Returned soldiers
– Experts in expatriate assignments
– Military psychologists
• We compared it to:
– Military models
– Civilian models
– New SME research
• Short answer: content is good
Florida Tech – Institute for Cross-Cultural Management
11
Criterion Evaluation
• Difficult…
– The research is civilian
– The measurement is poor
• Imprecise…
– Mapping a competency model to common
constructs
• Short answer: Middling support
Florida Tech – Institute for Cross-Cultural Management
12
This Report: Evaluate the
quality of instruments that
can assess Framework
competencies
• Using “off the shelf” instruments
• Using non-cultural instruments
– Personality, cognitive, etc.
Florida Tech – Institute for Cross-Cultural Management
13
Overview of the Procedure
1. Deconstruct the competency model to
elemental form
2. Map elements to commonly studied
constructs
3. Find “all” plausibly useful instruments
4. Evaluate the quality of the instruments
5. Map instruments to Framework elements
1. First: All instruments
2. Second: Only the good ones
6. Tally it up
Florida Tech – Institute for Cross-Cultural Management
14
1. Deconstruct the
Framework
•
•
•
•
Element
Aka “component” in competency modeling
C5: Interpersonal Skills
“[C5.1] Develops and maintains rapport.
Builds relationships in support of mission
performance. // [C5.2] Manage and
resolve conflict in support of mission
objectives.”
Florida Tech – Institute for Cross-Cultural Management
15
2. Map the elements to
constructs
• What existing, researched construct does
this competency seem to be related to?
Florida Tech – Institute for Cross-Cultural Management
16
Framework
Constructs
Measures
Competency element
1.1
Personality Construct 1
Instrument/Subscale 1
Competency element
1.2
Personality Construct 2
Instrument/Subscale 2
Attitude Construct 1
Instrument/Subscale 1
Attitude Construct 2
Instrument/Subscale 2
Cognition Construct 1
Instrument/Subscale 1
Cognition Construct 2
Instrument/Subscale 2
Enabler element 2.1
Abilities Construct 1
Instrument/Subscale 1
Enabler element 2.2
Abilities Construct 2
Instrument/Subscale 2
Competency 1
Competency 2
Competency element
2.1
Competency element
2.2
Enabler 1
Enabler element 1.2
Enabler element 1.1
Enabler 2
Competency 3: Cultural Perspective Taking
Competency element 3.1:
Demonstrates an awareness of
one’s own worldview… and how
one’s own group is viewed…
Competency element 3.3:
Takes the cultural context into
consideration when interpreting
situational cues.
• Knowledge of
attributed
stereotypes
• Self-insight
• Perspective
taking
• Metacognition
• Situational
Awareness
• MAKSSAwareness
• ICC-Awareness
• SEE-Empathic
perspective
taking
• SEE-Empathic
Awareness
• BEVISociocultural
Closure
• (Perspective
taking)
• CQSMetacognition
3. Find ALL Instruments
• Hunting and gathering in a vast literature
• Two kinds:
– Single-construct
– Batteries
• Three business models (why this matters)
– Open
– Restricted
– Proprietary
Florida Tech – Institute for Cross-Cultural Management
19
Many Instruments
• Don’t trust the published lists
• We unfairly divided scales into “primary”
and “secondary”
Florida Tech – Institute for Cross-Cultural Management
20
Primary Instruments
Name
Subscales
CCAI
Cross-Cultural
Adaptability Inventory
CQS
Cultural Intelligence Scale Metacognition； Cognition； Motivation； Behavior
ICAPS
Intercultural Adjustment
Potential Scale
IDI
Intercultural
Development Inventory
MPQ
Multicultural Personality
Questionnaire
Cultural Empathy; Openmindedness; Social Initiative;
Emotional Stability; Flexibility
ICC
Intercultural
Communicative
Competence
Complex structure: 3 domains; 5 dimensions; 4
developmental levels
INCA
Flexibility/Openness (FO); Emotional Resilience (ER);
Perceptual Acuity (PAC)； Personal Autonomy (PA)
Emotion Regulation (ER); Openness (OP); Flexibility (FL);
Creativity (CT)
Denial/Defense (DD); Reversal [R]; Minimization (M);
Acceptance/Adaptation (AA); Cultural Disengagement
(CD)
Intercultural Competence
Assessment
6 subscales
Florida Tech – Institute for Cross-Cultural Management
21
Name
Subscales
CCAI
Cross-Cultural
Adaptability Inventory
IES
Intercultural Effectiveness Continuous Learning; Interpersonal Engagement;
Scale
Hardiness
Interaction Engagement; Respect for Cultural
Intercultural Sensitivity
Differences; Interaction Confidence; Interaction
Scale
Enjoyment; Interaction Attentiveness
Empathic Feeling and Expression (EFE); Empathic
Scale of Ethnocultural
Perspective Taking (EP); Acceptance of Cultural
Empathy
Differences (AC); Empathic Awareness (EA)
ISS
SEE
Flexibility/Openness (FO); Emotional Resilience (ER);
Perceptual Acuity (PAC)； Personal Autonomy (PA)
Florida Tech – Institute for Cross-Cultural Management
22
Secondary Instruments
Instrument
Instrument
ADS
Adjustment Difficulties Subscale
IRC
Intercultural Readiness Checklist
AIC
Beliefs, Events, and Values Inventory
IRI
Interpersonal Reactivity Index
ASSIS
Acculturative Stress Scale for International
Students
ISAS
BASIC
Behavioral Assessment Scale for
Intercultural Communication Effectiveness
BEVI
Beliefs, Events, and Values Inventory
CCSI
Cross-Cultural Social Intelligence
MAKS
S
MASQ
UE
MCI
The Inventory of Student Adjustment
Strain
Multicultural Competence Scale
CGAIC
Culture-Generic Approach to Intercultural
Competence
CWQ
The Culture in the Workplace Questionnaire
EMMIC
GAP
Test
ICSI
Munroe Multicultural Attitude Scale
Questionnaire
Multicultural Competence Scale
MCKA
S
PCAT
Multicultural Competence Scale
European Multidimensional Models of
Intercultural Competence
Global Awareness Profile
PCSI
Peterson Cultural Style Indicator
SCS
Social Connectedness Scale
Intercultural Sensitivity Inventory
WDS
Workplace Diversity Survey
Peterson Cultural Awareness Test
Florida Tech – Institute for Cross-Cultural Management
23
Validation of the Instruments
• Primary
– Face
– Construct
– Criterion
• Secondary
– Criterion
Florida Tech – Institute for Cross-Cultural Management
24
Face Validity
• Good: Most instruments
• Bad: ICAPS (Intercultural Adjustment
Potential Scale)
• Moderate: CQS (Cultural Intelligence
Scale)
Florida Tech – Institute for Cross-Cultural Management
25
Construct Validity
• Convergent validity
– Similar to what it should be similar to?
• Divergent validity
– Different than what it should be different than?
• Structure
– If subscales: support for dimensionality?
Florida Tech – Institute for Cross-Cultural Management
26
Construct Validity Findings
•
•
•
•
•
(Primary scales only)
Missing information
Diverse outcomes, moderate validity
Mushy
Method variance not accounted for
Florida Tech – Institute for Cross-Cultural Management
27
Construct Validity Findings
• Good: MPQ (Multicultural Personality
Questionnaire)
• Mixed: most instruments
• Bad: ICAPS, ISS (Intercultural Sensitivity
Scale)
• Unknown: GCI, IES, INCA, ICC (mainly
proprietary)
Florida Tech – Institute for Cross-Cultural Management
28
Criterion Validity
• Does the scale predict something real?
– Adjustment: psychological or sociocultural
– Performance
– Response to training in experiments
• Known-groups
– Do groups differ on the scale the way they
should?
Florida Tech – Institute for Cross-Cultural Management
29
Criterion Validity Findings
• Good: MPQ (Multicultural Personality
Questionnaire)
• Moderate: CCAI, CQS, SEE
• Mixed: ICAPS
• Poor: ISS
• Unknown: GCI, IES, INCA
Florida Tech – Institute for Cross-Cultural Management
30
Conclusion: Validity
• Few scales with overall high validity
• Many highly desirable scales have little
support
Florida Tech – Institute for Cross-Cultural Management
31
5. Map (good) instruments to
Framework elements
• So: can the Framework be assessed?
• Strategy: Estimate extent to which each
element is assessed using high and
moderate validity instruments
Florida Tech – Institute for Cross-Cultural Management
32
Framework
Constructs
Measures
Competency element
1.1
Personality Construct 1
Instrument/Subscale 1
Competency element
1.2
Personality Construct 2
Instrument/Subscale 2
Attitude Construct 1
Instrument/Subscale 1
Attitude Construct 2
Instrument/Subscale 2
Cognition Construct 1
Instrument/Subscale 1
Cognition Construct 2
Instrument/Subscale 2
Enabler element 2.1
Abilities Construct 1
Instrument/Subscale 1
Enabler element 2.2
Abilities Construct 2
Instrument/Subscale 2
Competency 1
Competency 2
Competency element
2.1
Competency element
2.2
Enabler 1
Enabler element 1.2
Enabler element 1.1
Enabler 2
Competency 3: Cultural Perspective Taking
Competency element 3.1:
Demonstrates an awareness of
one’s own worldview… and how
one’s own group is viewed…
Competency element 3.3:
Takes the cultural context into
consideration when interpreting
situational cues.
• Knowledge of
attributed
stereotypes
• Self-insight
• Perspective
taking
• Metacognition
• Situational
Awareness
• MAKSSAwareness
• ICC-Awareness
• SEE-Empathic
perspective
taking
• SEE-Empathic
Awareness
• BEVISociocultural
Closure
• (Perspective
taking)
• CQSMetacognition
Score Card: Competencies
Competency
Name (# elements)
Rating
C1
Knowledge (3)
*
C3
Perspective-Taking (3)
***
C4
Communication (2)
**
C5
Interpersonal Skills (2)
**
C6
Cultural Adaptability (2)
****
Florida Tech – Institute for Cross-Cultural Management
35
Score Card: Enablers
Enabler
Name
Rating
E1.1
Ambiguity tolerance
(****)
E1.2
Need for closure
(****)
E1.3
Suspending judgment
*
E1.4
Inclusiveness
**
E2.1
Stress resilience
**
E2.2
Emotion regulation
****
E3.1
Self-confidence
(****)
E3.2
Self-identity
****
E3.3
Optimism
Florida Tech – Institute for Cross-Cultural Management
(**)
36
More Enablers
Enabler
Name (# elements)
Rating
E4.1
Learning through observation
(3)
0
E4.2
Inquisitiveness
(*)
E5.1
Social flexibility (3)
****
E5.2
Willingness to engage
***
Florida Tech – Institute for Cross-Cultural Management
37
So How Do We Measure a
Competency?
• Plan A: off the shelf self-report measures
– Mixed value
– Various problems with self-report measures
• Faking, response biases, cognitive/declarative
knowledge, etc.
• Plan B: behavior-based assessment
centers
– Designed for each competency
– Difficult, expensive, slow
Florida Tech – Institute for Cross-Cultural Management
38
Another Way…
• 1. Rewrite the Framework as a blended
competency/causal model
• Competency model:
– Just the right competencies for the job
• Causal model:
– Related competencies and enablers in models
– Add moderators; mediation
Florida Tech – Institute for Cross-Cultural Management
39
• 2. Assess the model, not just the
competencies
• For each competency, assess several
variables in the model
– “Triangulate” on the individual’s competency
– Relieves the pressure on assessing the
competency itself
Florida Tech – Institute for Cross-Cultural Management
40
Blended Model
C1.1a: CultureGeneral
Knowledge
Openness
Multicultural
Attitudes
E4.2: Culture
Inquisitiveness
E4.1.2:
Learning
Through
Observation
Need for
Cognition
Situational &
External
Antecedents
Florida Tech – Institute for Cross-Cultural Management
C: Competencies
Requiring
Knowledge
41
fin
Florida Tech – Institute for Cross-Cultural Management
42