How to evaluate the cross-cultural
equivalence of single items
Melanie Revilla, Willem Saris
RECSM, UPF
Zurich – 15/16 July
Cross-cultural equivalence
• Usually
– Discussed in the frame of cross-national research
– Idea: in different countries people can express themselves in different
ways
– Different cultures can also be defined on other criteria (e.g. language)
– Procedure: can be applied in similar way to all kinds of different groups
• “Equivalence”  measurement equivalence
– 2 persons with the same opinion will give the same answer (whatever their
group)
• Important
– because observed differences may result from non equivalent measures
and not be real differences
• If measurement equivalence does not hold
– cannot make comparison across groups!!
Important distinction (Northrop, 1947)
• Concept by Intuition (CI)
– Simple concepts that can be measured directly
– Single item
– Ex: trust in the parliament
• Concept by Postulation (CP)
–
–
–
–
Complex concepts that cannot be measured directly
Also called “construct”
Need several CI to measure them
Ex: political trust: trust in parliament + legal system + police …
• Classic procedure to test for equivalence for CP but not
for CI
 start with a reminder of the procedure for CP
When we have multiple indicators
CP / complex concepts
Basic Confirmatory Factor Analysis model
λ11
λ21
CP1
Y1
Y2
λ31
Political trust
Y3
intercepts slopes
≈ regression
equation
[Y
i
τ1
Answer Trust in the parliament
e1
τ2
e2
τ3
Answer Trust in the police
e3
error terms
= τi + λij CP1 + ei
Dependent
variable
Answer Trust in the legal system
Independent
variable
i = 1,2,3
Multiple Group CFA approach
Group 1
Group 2
• Multiple group:
– possible to test for equality of the parameters in the different
groups
– constraints across groups
• Can be extended to more groups
Different levels of invariance (Meredith, 1993)
• Configural
Group 1
– Same model holds in all groups
• Metric
– Configural + Slopes (λij) the
same in all groups
– Sufficient for comparison of
relationships
• Scalar
– Metric + Intercepts (τi) the same
in all groups
– Sufficient for comparison of
means
• More: error terms, etc…
Group 2
In practice
• Analyses can be done with standard SEM
softwares
–
–
–
–
–
LISREL/Mplus
based on covariance matrices & means
recommended sample size: >200 in each group
3-step procedure: configural, metric, scalar
syntax quite easy to get estimates
• More tricky but crucial step: testing
Testing the model
• Assessing global fit
– Chi2 test / Fit indices: RMSEA (<.05), CFI (>.9), etc…
– Limits: Depends on sample size / Sensitive to deviations from normality
• Assessing local fit
– Saris & Satorra  should test at the parameter level + take into account
type II errors (H0 not rejected despite being false)
– JRule software (van der Veld, Saris, Satorra) + Jrule for Mplus (Oberski)
• See next presentation!
• Always check if estimates are really different
– Difference may be statistically significant but not substantially meaningful
• Partial invariance
– What if some indicators are equivalent but not all?
– Consistent estimates of the means of the latent variables if at least 2
indicators are scalar invariant (Byrne, Shavelson, Muthén, 1989)
If we have single indicator
CI / simple concepts
Single items
Group 1
CI1
λ11
Y1
Group 2
τ1
CI1
e1
Yi = τi + λij CIi + ei
• Testing equivalence single items
• Testing equivalence for CI
λ11
Y1
τ1
e1
Single  multiple indicators?
• Problem: model just presented not identified
• “Single indicator” = single trait in fact
• But possible to use multiple methods
• So for CI:
– Only one trait, but we can always have more than one
method
– Several indicators = same trait asked using different
methods
Can apply again MGCFA
Group1
Group 2
11 points
6 points
Trust in the
parliament
4 points
• Similar at the previous model (for CP) but now
different methods instead of different traits
measuring a same concept
Same procedure
• Different levels of invariance as for CP
–
–
–
–
Configural
Metric
Scalar
etc
• Same procedure to get the estimates and test the
model
– Multiple group analyses
– Test of the model: global / local fit
– Partial equivalence
Problem for the CI
• Fix the scale?
–
–
–
–
–
–
As before, necessary to fix the scale of the LV
Usually, fix the first loading to 1
Can be done here too
Other loadings are relative to the first one
But need to be done in all groups
If there are differences for the method whose loading
is fixed to 1 across groups, may be problematic
– Should try to use methods that have been shown to
be the most similar across groups: e.g. fixed
reference points
General model
u1
α1
v21
CI1
Y21
v31
c1
Y31
u2
CP1
Y11
1
c2
α2
CI2
1
v22
Y12
Y22
v32
Y32
c3
u3
α3
CI3
1
v23
v33
Y13
Y23
Y33
τ11
e11
τ21
e21
τ31
e31
τ12
e12
τ22
e22
τ32
e32
τ13
e13
τ23
e23
τ33
e33
• Even when working on CP: better to use different methods
Final remarks / CCL
Equivalence single items
• Need to repeat the same item with different methods
– 3 or more repetitions
– Multi Methods (MM)?
• Same persons get the question several times using different methods
• Limit: 20 minutes at least to avoid memory effects (Van Meurs & Saris,
1990)
– Mix with Split-Ballot (SB) design?
• Random assignment of respondents to different versions of the
questionnaire
• “SB-MM” (CI) or SB-MTMM (CP)?
Conclusion
• Measurement equivalence can be assessed both
for CP and CI using Multiple Group Confirmatory
Factor Analysis
– For CP, process already well-known and used a lot
– For CI, possible do similar analyses
 But necessary to repeat questions!!  specific data
• So testing equivalence of simple item can be done
using a (SB)-(MT)MM approach
– Similar to what exists in the ESS for CP: main +
supplementary questionnaires (different versions)
– With extension for concepts by intuition
In summary
• To test single item equivalence
– Use multiple methods
– Do everything as for multiple items equivalence
Thank you for your attention!
Questions?