Survey Design and
Cross cultural Issues
Avinish Chaturvedi
Carlos Torelli
Agenda



Brief Review of the readings
Methods and Data Analysis for Cross-Cultural
Research
Discussion
Heine et al., (2002)

Reference group effect: people from different cultural groups use
different referents in their self-reported values (i.e., compare with
different others)

Low Individualism -------------------- High Individualism 
------------- Japanese High (7) ------ American High (7) 
An apparent no difference is actually a significant difference

Wong et al., (2003)


Problems with Likert scales that contain a mixture of
positive-worded items (PWI) and reverse-worded
items (RWI) in cross-cultural research.
Why use RWI?:


Reduce acquiescence bias.
General problems with RWI?


May negatively impact internal consistency.
May disrupt dimensionality (RWI loading in a separate
factor).
Brislin chapter:




Why do we need to change/modify the survey
instrument in a different culture
Linking it with previous discussion
What factors change across cultures
Why not to use existing scales
Types of cross cultural
studies: Van de Vijver chapter

Generalizability studies:
The design of a generalizability study is usually a
replication of the original study.
- When the goals of cross cultural studies are
defined as both delineating and explaining cross
cultural differences, generalizability studies
capitalize on first and often ignore the second.
Types of cross cultural
studies: Van de Vijver chapter

Studies of psychological differences
These are often based on a less elaborate
theoretical framework than are generalizability
studies
Types of cross cultural
studies: Van de Vijver chapter

Theory driven studies
- These studies test a theory of cross cultural
differences. Theory driven studies are more
systematic. Such studies are designed and
carried out to critically test a hypothesis. These
studies provide powerful tets of theories of cross
cultural differences, which is one of the main
goals of cross cultural research.
Types of cross cultural
studies: Van de Vijver chapter

External validation studies
- These studies take observed cross cultural
level or structural differences as their starting
point and scrutinize these differences either by
exploring their antecedents or by testing
interpretations of these differences.
Cross cultural issues


Researchers run the risk of missing aspects
of a phenomenon as viewed by people in
other cultures.
They risk imposing conclusions based on
concepts which exist in their cultures but
which are foreign, or at least partially
incorrect, when used in other culture.
Emic and Etic distinction




How to determine what attributes of a
construct are static, i.e., unchangeable
across cultures
What a priori measures can be employed in
this regard
Example from Brislin chapter: Authoritarian
and conservatism.
What kind of questions are more sensitive to
cultural variations?
Method and Data analysis

In cross cultural studies what kind of
sampling measures should be taken

Convenience Sampling:
Studies adopting this sampling scheme often
fall into the category of psychological
differences studies
Method and Data analysis

Systematic Sampling:
Cultures are selected in a systematic, theory guided fashion.
These studies usually fall into the categories of theory driven or
generalizability studies. Cultures are selected in this procedure
because they represent different values on a theoretical
continuum

Random Sampling:
It involves the sampling of a large number of cultures randomly.
This strategy is preferable for generalizability studies, in which a
universal structure or a pan cultural theory is evaluated.
Summary
Generalizability studies
sampling procedures
major weaknesses and
strengths
Studies of psychological
differences
sampling procedures
major weaknesses and
strengths
Theory driven studies
sampling procedures
major weaknesses and
strengths
External validation
studies
sampling procedures
major weaknesses and
strengths
Method and Data analysis

Choosing Cultures:
How does choice of cultures in a research
design changes when:
a. we are looking for similarities
b. we are looking for differences.
Method and Data analysis

Administration of instruments:
tester/interviewer
testee/interviewee
interaction between two
response procedures
stimulus materials
Method and Data analysis

Remedies:
A priori and post priori techniques.
Similarly prior and post hoc remedies can be used to alleviate
problems of sample incomparability.
Application of a monotrait multimethod matrix in order to examine
the influence of response procedures is useful
What are the other ways through which this effect can be
minimized??
Method and Data analysis
Changing the survey instrument
Application
Adaptation
Assembly

In which scenario, each of above three
technique becomes useful?
Method and Data analysis

Validity Enhancement
The major criterion in the choice of application,
adaptation and assembly is the type of bias
expected. If there are serious concerns that
construct bias could play a role, adaptation or
assembly should be chosen.
Indigenization which aims at maximizing the
appropriateness of psychological theories and
instruments to local cultures, will often amount to the
assembly of new instruments
Method and Data analysis

Obtaining linguistically equivalent
instruments:
Translation and back translation
Committee approach
Method and Data analysis

Translation
Translation is more than producing text in
another language. Translators should know
or be made aware of the linguistic and
cultural differences that could influence
responses to translated or adapted
instruments.
Translation


Guidelines for translating and adapting
psychological and educational instruments:
What are the other translation issues involved
Study 1 – Experts’ Ratings

Sample: Japanese specialists living in North America

What were the questions?


Findings:


How much items in Singelis (1994) scale characterize Japanese or
Americans
Experts were in agreement with the common view for all 23 items (high face
validity).
These findings suggest that the use of different reference groups
can obscure cross-cultural differences (Americans use other
Americans, and East Asians other East Asians).
Study 2 – Manipulation of Reference
Groups


Sample: Canadian biculturals with knowledge about Japan and
Canada.
Instruments:




Singelis (1994) in standard form.
Answers in comparison with most Japanese.
Answers in comparison with most Americans.
Findings:



Weak support for the common view using standard format.
When comparing with people from the other culture: strong support for the
common view.
When comparing with people from their own culture: evidence from
acculturation.
Study 3 – Within-culture

Sample: Asian Canadians vs. European Canadians, and returnee
Japanese vs. Japan-bound Japanese
Common referent for each group.

Findings:



European Canadians scored higher in independence and lower on
interdependence than Asian Canadians
Returnee Japanese scored higher on independence and lower on
interdependence than Japan-bound Japanese
Discussion

Likert scales capture one’s feelings relative to a comparison
group or shared norm?



Implications from the reference-group effect:


Is it always the case? The case of self-esteem?
When can Likert scales assess one’s feelings without a referent? When do
people use less social comparison for answering (i.e., introspection)?
Use of subjective Likert scales most valid for identifying differences within
rather than between groups  What do you think?
Remmedial measures:





Measure at the cultural level
Prompt comparisons with an standard (average peer)
Responses to information that is consistent/inconsistent with implicit theories
(new information)
Use of behavioral measures
Items with concrete, objective response options
The MVS

The MVS in cross-cultural settings:



Good reliability and validity in the U. S.
Questionable reliability and validity abroad.
Why the problems outside the U. S.?



Negations and contradictions can be confusing
Languages can employ different ways of making negation or contradiction.
Differences in how PWI and RWI are interpreted (cultural norms regarding
agreeableness?)
MVS
across
cultures
–
Study
1
Countries: the U. S., Singapore, Japan, Korea, and Thailand.


Prediction:



Findings:





Negative correlation with life satisfaction in wealthy countries only.
Positive correlation with individualism across all countries
MVS specified as a 2nd. Order factor with 3 dimensions (happiness, centrality,
and success) showed a poor fit  systematic error?
PWI and RWI correlated in the U.S., but weakly or uncorrelated elsewhere
RWI being responded differently?
Divergent correlations for materialism and life satisfaction
Revised Model: include two method factors  fit comparable to
those obtained in the U.S., but construct validity still an issue
Support for the notion that RWI might have led to the poor initial
performance
MVS across cultures – Study 2

Objectives:




Predictions:




Show conceptual equivalence by using adjectives
Other scales that include RWI should show two-factors structures
Inquiring about a respondent’s position on an issue would minimize
“agreement for the sake of being polite”
What do you think about the way conceptual equivalence was
assessed?


Assess conceptual meaning of materialism.
Show that other scales that use PWI and RWI have the same problems.
Evaluate alternative scales  interrogative format
Does it assess the 3-dimensions?
What does the evidence that other scales show two factors mean?

Does it prove that agreeableness is the underlying issue? What
about the findings in the general discussion?
Discussion

Acquiescence vs. “substantive cultural differences”?




Are the results driven by acquiescence or not?
Are the meanings of the MVS items different across cultures due to religion
and beliefs? If so, what is the explanation for the results with the other 4
scales?
Can there be any referent group effect?
Recommendations:



Use only PWI
Use interrogative formats
Which one is better?