Cross-cultural surveys and translation
ESRC Question Bank Conference
SURVEY MEASUREMENT: ASSESSING THE
RELIABILITY AND VALIDITY OF
QUESTIONNAIRE ITEMS
April 10
Royal Statistical Society
Janet A Harkness
gesis-ZUMA
1
Outline
I. Framework
II. Progress in survey translation
production and assessment
III. Persistent problems & consequences
IV. Towards solutions
gesis-ZUMA
2
I. Framework
1. Survey translation uses
2. Expectations
3. Current practice
gesis-ZUMA
3
1. Survey translations uses
 Within country research
 Groups may partially share larger
context
 Shared contexts, degree of
interaction, immigrant acculturation
affect translations required
 Across country research
 Different contexts and languages
gesis-ZUMA
4
2. Expectations for survey
translations
Assumption is a good translation
 asks the same question
 maintains semantic & pragmatic meaning
 maintains measurement properties
 retains source design features
 satisfies multiple other requirements
(askable and answerable, burden,
saliency, etc)
gesis-ZUMA
5
3. Common current practice
Depending on discipline
 Do not assess translation process quality
 Assess translation quality through back
translation
 Pretest translated questions
 Assess translated instrument quality on
basis of statistical analysis (dif, IRT)
gesis-ZUMA
6
II. Progress in survey translation
production and assessment
gesis-ZUMA
7
Progress in survey translation
production and assessment
1. Guidelines and know-how
2. Tools
3. Basic research
 Procedures, strategies, outcomes
gesis-ZUMA
8
Progress in survey translation
production and assessment
1.
2.
3.
4.
Guidelines and know-how
Tools
Basic research
Procedures and strategies
gesis-ZUMA
9
1. Guidelines and Know-How
gesis-ZUMA
10
1. Guidelines and Know-How
 ESS, USA Census Bureau, International
Test Commission, QoL research and
publications
 Comprehensive sets of guidelines CSDI
Workshop Guidelines Initiative (lead
groups at ISR, Michigan, UNL and gesisZUMA)
gesis-ZUMA
11
2. Tools
gesis-ZUMA
12
2. Tools
 Process documents
 Note-taking templates
 Queries, decisions, rationales
 Harmonization templates
 Decisions, rationales
 Version records
 Decisions, rationales
gesis-ZUMA
13
2. Tools
 Technological options
 Memory files -- repeated questions, instructions,
answer scales …
 Translation software support
 Parallel presentation of source and target
versions
 Importing modifications to existing text
 Potential do-it-all tools
 Combine questionnaire production-documentation
with translation version production-documentation
gesis-ZUMA
14
3. Basic research
gesis-ZUMA
15
3. Basic research
Survey translation research on
 Assessment procedures & outcomes
 Translation procedures & outcomes
 Answer scale translation
 Oral translation and interpreting
 Tool options
 Impact of source questions
gesis-ZUMA
16
3. Basic research
Survey translation research on
 Assessment procedures & outcomes
 Translation procedures, strategies
& outcomes
 Answer scale translation
 Oral translation and interpreting
 Tool options
 Impact of source questions
gesis-ZUMA
17
Procedures, strategies, outcomes
 Team translation efforts
 Interdisciplinary expertise
 Translators, reviewers, adjudicators,
consultants
 Iterative process
gesis-ZUMA
18
Team Translation
TRAPD model --an iterative cycle
Translation
Review
Adjudication
Pretesting and refinement
Documentation underpins all stages
(cf. frameworks in ESS, SHARE, and US
Bureau of Census, Westat, WMHI)
gesis-ZUMA
19
Why iterative
TRAPD procedures may need to be
repeated at different stages. For
example, pre-testing and debriefing
sessions with fielding staff and
respondents will lead to revisions;
these call for further testing of revised
translations.
gesis-ZUMA
20
Translate
and document
SOURCE
gesis-ZUMA
Review, adjudicate
and document
Pretest and document,
reiterate if necessary
21
Basic Team Players
 Translators: selected, competent, briefed
 Reviewers: selected, competent, briefed
 Adjudicator: (takes final decision) selected as
possible on basis of skills as well as seniority.
 May need to work with Consultant.
gesis-ZUMA
22
Other Team Players
 Translators: Reviewers: Adjudicator
--------------
 (Co-ordinator)
 (Substantive experts)
 (External assessors)
 Copy-editors
 (Programmers)
 Back-ups (illness, vacation, leave)
 (Oral translation and interpreting extra)
gesis-ZUMA
23
Basic Procedures
 Translators translate
 Review session: reviewers discuss and review
each question
 Adjudicator decides/signs off


consults with senior reviewer;
if sensible/possible also joins in review session(s).
gesis-ZUMA
24
Scenes from a review session …
gesis-ZUMA
25
The team






Senior reviewer, co-adjudicator
Translator 1
Translator 2
Project coordinator, survey researcher
Pre-tester, survey researcher
Survey researcher, lay translation
talent
gesis-ZUMA
26
Clip 1: The meeting begins





Four in room discussing the weather
Paul and Margrit enter
Greetings
Seating
Framing the session
 Paul: I’ve got a parking space till 5.06 pm.
 Janet: …Yes, we need to finish by five.
gesis-ZUMA
27
gesis-ZUMA
28
Clip 2: reaching decisions
Team has been trying at length to find a
phrase that includes atheist views on
religion.
Translations proposed so far imply people
do have religious views, rather than just a
view about religion (so the clip begins)
gesis-ZUMA
29
Clip 2 events
 Discussion ongoing
 Janet halts discussion (hand motion)
and identifies the continuing problem
 Peter makes a new proposal
 Team consider it and accept
 Adjustments to other text discussed,
approval re-confirmed
 The core group takes notes
gesis-ZUMA
30
.
JH
PM
JH
Let’s recapitulate. There’s the problem that
some have no faith and no religion … our
discussion … solves perhaps what we
could do for Muslims and Protestants or
among Protestant groups, but does not
really solve the other problem.
Well, there’s an odd word in German
“glaubensvorstellungen” that means other
views from those you /they have
Ah.. And would that cover atheists too?
gesis-ZUMA
31
.
PM
MR
JH
MR
JH
Yes, of course, they have a view about faith,
namely none (= without faith)
And the atheists are covered
Good. Then we’ll take that. Then we have
everything
And Muslims are also covered
Good
(More discussion of how good the solution is and
that earlier text should be modified)
gesis-ZUMA
32
gesis-ZUMA
33
What we learn from teams




Language challenges
Source question issues
Strategies
Task knowledge available and needed
gesis-ZUMA
34
Quality improvements considerable,
but…
gesis-ZUMA
35
III. Persistent problems
gesis-ZUMA
36
III. Persistent problems
1. Views on translation and what it
can do
2. Established common practices
3. Nature of questionnaires
4. Researchers and good questions
gesis-ZUMA
37
1. Views on translation & what it can do
i. Anyone can translate
ii. Focus on words
iii. Neglect/avoidance of adaptation
gesis-ZUMA
38
i) Anyone can translate
 My secretary speaks Czech
 My son studies French
 Harry spent a year in Turkey
 Language ability is not a guarantee for
translation ability
gesis-ZUMA
39
ii) Focus on words
 Have you felt blue or down
recently?
gesis-ZUMA
40
ii) Focus on words
Have you felt blue or down recently?





NOT a matter of colour or idiom
Temporary state
Degree of depressed state
[Disclosure]
[Symptom relevance]
gesis-ZUMA
41
iii) Neglect/avoidance of
Adaptation
gesis-ZUMA
42
Adaptation
 Deliberate modification of a question or
questionnaire to meet new requirements
 Frequently but not necessarily
associated with translation
 Undertaken in source questions and/or
translated questions
 Various forms (cf. Harkness 2008)
gesis-ZUMA
43
Adaptation
 Do you have difficulty
several blocks?
 Do you have difficulty
yards?
 Do you have difficulty
metres?
 Do you have difficulty
metres? (Sweden)
gesis-ZUMA
walking
walking 100
walking 100
walking 200
44
Adaptation and Design are related
Measurement properties should remain
Intended latent construct should remain
Burden or difficulty should not change
Relationship to other questionnaire
elements should not change
 Adaptation can be anticipated in source
instruments




gesis-ZUMA
45
Adaptation and Translation are related
 Translation involves adaptation
 (Adaptation need not involve translation)
 In the context of translation, some general
types of adaptation can be identified
 Blends/entanglement of different
types not unusual
gesis-ZUMA
46
2. Established practices as
persistent problems
gesis-ZUMA
47
2. Established practices as
persistent problems
 Back translation
 False economy
 Horse-and-cart structures in survey
implementation
 Horse and Cart are essentially
different and distinct
 Perceived interdependence minimal
gesis-ZUMA
48
3. Nature of questionnaires as
persistent problem
gesis-ZUMA
49
3. Nature of questionnaires
 Complicated text type
 Leads a double life
 Covert measurement properties
 Surveyspeak and scalespeak
gesis-ZUMA
50
3. Nature of questionnaires
 Complicated text type
 Leads a double life
 Covert measurement properties
 Surveyspeak and scalespeak
gesis-ZUMA
51
Surveyspeak
Dentist When did you last visit a dentist?
Patient About two months
Dentist follow-up
Are you sure?
Uh-huh.
I see.
gesis-ZUMA
52
Surveyspeak (2)
Interviewer: When did you last visit a dentist?
Respondent: About two months ago.
Survey follow-up
Would you say that you are very certain /
somewhat certain / neither certain nor uncertain /
somewhat uncertain or very uncertain about the
date you just gave?
(or attempt to get date)
gesis-ZUMA
53
53
Scalespeak in an “importance” scale
very important
somewhat important
neither important nor unimportant
somewhat unimportant
very unimportant
This is somewhat unimportant
This is very unimportant
This is somewhat important
gesis-ZUMA
54
4. Good questions (and researchers) as
persistent problems
gesis-ZUMA
55
4. Good questions (and researchers) as
persistent problems
What makes a question good?
 Q measures what it should
 Q understood as intended
 Q salient for respondents
 Q answerable and answered
 Demonstrated quality through testing and
use
gesis-ZUMA
56
Good questions "fit" their context
 Validity and reliability are facilitated by
common ground, shared speech
community usage and social norms
 Intended meaning of question and answer
options = perceived meaning
gesis-ZUMA
57
A visual example of a good instrument
gesis-ZUMA
58
Chinese diagnostic doll
Patient remains clothed, doll
is naked
Cultural norms on disclosure
observed to enable response
Note: feet are covered
Cultural norms on sensitive
topics observed
gesis-ZUMA
59
The diagnostic doll reflects and
accommodates the cultural embedding of the
instrument (doll), the researcher (doctor) and
the respondent (patient)
gesis-ZUMA
60
When "good" questions go travelling...
In different contexts, good questions may be poor
cultural fits
 change in "meaning"
 different conceptual coverage
 socially difficult to ask or to answer
 lose or gain saliency
gesis-ZUMA
61
When "good" questions go travelling...
In different contexts, good questions may be poor
cultural fits
Translation may then be an inappropriate
means to "ask the same question"
gesis-ZUMA
62
Consequences: what can go wrong
….and why
gesis-ZUMA
63
TRANSLATION
Source
Communication about my illness at home is poor
Translation (Spanish)
We do not talk much about my illness at
home
Why?
Remedy?
gesis-ZUMA
64
PERCEPTION
Source
Do you have difficulty sitting for 2 hours?
Translation (French)
Do you have difficulty standing for 2 hours?
Why?
Remedy?
gesis-ZUMA
65
CULTURE or INTENDED MEANING
Source
Would you take part in a demonstration?
Translation (German)
Would you take part in a demonstration that
blocks the traffic?
Why?
Remedy?
gesis-ZUMA
66
GLITCH
Source
Have you ever felt like hitting someone
Translation (Turkish)
Have you ever felt like shooting someone
Why?
Remedy?
gesis-ZUMA
67
Translation process highlights design issues
Do you provide financial support for grown-up
children or grandchildren?

How many hours TV do you watch on an average
weekday?

Please give me the initials of your first and last
name


Do you prefer OTC or prescription medicines?
gesis-ZUMA
68
Unavoidable design changes
English
neither satisfied nor dissatisfied
isiZulu and Hebrew
neither nor

dissatisfied

gesis-ZUMA
not ... not
not satisfied
69
Unavoidable design changes
English
neither satisfied nor dissatisfied
isiZulu and Hebrew
"not satisfied and not not satisfied"
cf. Henningsson et al, 1998, Harkness et 2005
gesis-ZUMA
70
IV Towards solutions:concluding remarks
gesis-ZUMA
71
IV Towards solutions
 See the source as the source
 Source of information
 Source of challenges
 Source needs to be appraised and
tested for suitability for new
contexts and languages
gesis-ZUMA
72
IV Towards solutions
 See the target as the target
 Determine aims for target language
questions
 Appraise source question goals and
means for target realization
 Engage in deep processing
 Target may need degrees of freedom
gesis-ZUMA
73
IV Towards solutions: more progress
 Progress will be a process
 Base any change on evidence
 Research and documentation
 Create critical mass
 Collaboration in initiatives
 Record and share lessons learned
gesis-ZUMA
74
Thank you
75
Related Literature
Harkness, J. (2008) “Comparative Survey Research: Goals and
Challenges.” Foundation chapter in: Dillman, D., Hox., J. and de Leeuw,
E. (eds.) International Handbook of Survey Methodology, Hyattsville, VA:
Erlbaum.
Harkness, J., Schoebi, N., Joye, D., Mohler, P., Faass, T. and Behr, D.
(2007) “Oral Translation in Telephone Surveys”. In: J.M. Lepkowski, C.
Tucker, J.M. Brick, E. de Leeuw, L. Japec, P.J. Lavrakas, M.W. Link and
R.L. Sangster. Advances in Telephone Survey Methodology, John Wiley
and Sons, Hoboken, New Jersey.
Harkness, J.; Pennell, B.-E., Schoua-Glusberg, A. (2004): Survey
Questionnaire Translation and Assessment. In: Presser, Stanley,
Rothgeb, Jennifer, Couper, Michael, Lessler, Judith, Martin, Elizabeth,
and Singer, Eleanor (Eds.): Questionnaire Development Evaluation and
Testing Methods, Wiley Series in Survey Methodology. New Jersey: John
Wiley & Sons Inc.
gesis-ZUMA
76
Harkness, J. (2003): Questionnaire Translation. In: Harkness, Janet A., Van
de Vijver, Fons J.R., Mohler, Peter Ph.(Hrsg.): Cross-Cultural Survey
Methods. Wiley Series in Survey Methodology. New Jersey: John Wiley
and Sons Inc
Harkness, J.; Van de Vijver, F. J. R.; Johnson, T. P. (2003): Questionnaire
Design in Comparative Research. In: Harkness, Janet A., Van de Vijver,
Fons J.R., Mohler, Peter Ph.(Hrsg.): Cross-Cultural Survey Methods. Wiley
Series in Survey Methodology. New Jersey: John Wiley and Sons Inc
Harkness, J.; Schoua-Glusberg, A. (1998): Questionnaires in Translation.
In: Harkness, J. (Hrsg.): Cross-Cultural Survey Equivalence. ZUMANachrichten Spezial Band 3. Mannheim: Zentrum für Umfragen, Methoden
und Analysen 1998, S. 87-128
gesis-ZUMA
77