Adapting Credentialing Exams
for Use in Multiple Languages
and Cultures
Ronald K. Hambleton
University of Massachusetts
1
Three Reasons for Translating
and Adapting Exams
1.
Knowledge and skills of interest are
often the same across language
groups--exam adaptation ensures
consistency of content and structure.
2. Often more efficient to translate and
adapt an exam than to build a new exam.
3. Easier to insure exam equivalence and
fairness (than construct new exam in a
second language).
2
Goals of the Presentation
1. To describe several myths
about test adaptation.
2. To review steps in the test
adaptation process.
3. To present the ITC guidelines
for test translation/adaptation.
3
Test Translation vs. Test
Adaptation?
“Test Adaptation” is more
descriptive of the process that
usually takes place—directions,
formats, contexts, etc.
“Test Translation” is sometimes
only a small part of the process.
4
Examples of Test Adaptations
Intelligence, Aptitude, and
Personality Tests
 Quality of Life Surveys
 TIMSS, OECD/PISA, and More
 SAT, GED, NAEP, State
Assessments
 Credentialing Exams

5
Example 1
Out of sight, out of mind
(Back translated from French)
invisible, insane
6
Example 2 (IEA Study in
Reading)
Are these words similar in
meaning?
Pessimistic -- Sanguine
7
Pessimistic -- Sanguine
Adapted to
Pessimistic -- Optimistic
8
Example 3 (1995 TIMMS Pilot)
Alex reads his book for 1 hour and
then used a book mark to keep his
place. How much longer will it take
him to finish the book?
A. ½ hour
B. 2 hours
C. 5 hours
D. 10 hours
9
Example 4
Parker Pen in Mexico: “It won’t
leak in your pocket and
embarrass.”
Became:
“It won’t leak in your pocket and
make you pregnant.”
10
Four Common Myths About
Adapting Credentialing Exams
1. Know two languages and you can
be a translator.
11
Selection and Training of
Translators
Knowledgeable in the languages
 Knowledgeable in the cultures
 Knowledgeable in the subject
matter
 Knowledgeable in the principles of
test development, item writing,
and scoring rubrics, etc.

12
Four Common Myths About
Adapting Credentialing Exams
2. A good translation guarantees
validity.
13
Four Common Myths About
Adapting Credentialing Exams
3. Judgmental reviews are
sufficient to identify
problems in a exam
translation/adaptation.
14
Four Common Myths About
Adapting Credentialing Exams
4. The common strategy of a
back-translations design and the
use of a bilingual design to
compile empirical data is
sufficient to justify exam use.
15
8 Steps for Adapting Exams
1. Checking exam content and
format equivalence.
-content may not be
generalizable
-item formats are not
universally used
16
8 Steps for Adapting Exams
2. Deciding on the desirability of
a translation/adaptation.
-unsuitability of content and/or
the exam format?
-technical expertise to produce
and validate a new exam?
17
8 Steps for Adapting Exams
3. Choosing translators.
-Four qualifications given
earlier are important.
-Some technical expertise may
be especially helpful.
18
8 Steps for Adapting Exams
4. Translating and adapting the
credentialing exam.
-Should often be more than a
single translator.
-Several designs and variations
available.
19
Judgmental Designs
for Assessing Suitability of
a Translation/Adaptation
1. Backward Adaptations
2. Forward Adaptations
20
Backward Translation Design
Advantages
1.
2.
Some problems can be identified.
Researchers do not need to be bilingual.
Disadvantages
1.
No review of target language version!
21
Forward Translation Design
Advantages
1.
Judgments made directly about
language versions of an exam.
Disadvantages
1.
Bilingual judges make assessments
of an exam for monolingual
examinees.
22
8 Steps for Adapting Exams
5. Reviewing the adapted version
of the exam.
-Multiple reviewers are often
very valuable.
-”Smoothing” of language is
possible at this step.
23
8 Steps for Adapting Exams
6. Conducting a small tryout of
the adapted exam
-item analysis, reliability
analysis, candidate review, etc.
24
8 Steps for Adapting Exams
7. Carrying out a more ambitious
study of the adapted exam
(that is, checking for exam
validity).
-construct equivalence,
method equivalence, item
equivalence
25
Statistical Designs
1. Bilingual Designs
2. Monolingual Designs
A. Source Language Examinees
Take Original and Backward
Adapted Versions
B. Monolingual Examinees Take
Versions in Their Own
Languages
26
Checking for Construct Equivalence:
Item
1
2
3
4
5
…
19
20
21
22
Factor 1
Language A Language B
.56
.49
.43
.48
.54
…
-.03
.25
.29
.09
.49
.40
.57
.52
.46
…
.22
.13
.08
.26
Factor 2
Language A Language B
-.07
.59
.11
.63
.38
…
.49
.53
.27
.02
.14
.41
-.04
.68
.45
…
.44
.56
-.13
.22
27
18
16
Delta (ES1)
14
12
10
8
6
4
4
6
8
10
12
14
16
18
Delta (English)
28
Checking for Item Equivalence: Delta
Plots
18
16
Delta (RLO)
14
12
10
8
6
4
4
6
8
10
12
14
16
18
Delta (English)
29
Checking for Item Equivalence:
IRT Methods (change in difficulty)
1.00
Probability of
Correct Response .
1.0
A
B
0.5
0.50
0.0
0.00
0
-3
10
-2
20
-1
30
0
40
1
50
2
60
3
Proficiency
Frequency
Language B
Language A
-0.50
-1.00
0
-3
10
-2
20
-1
30
0
40
1
50
2
60
3
Proficiency
30
IRT Methods (change in
discrimination)
1.00
Probability of
Correct Response .
1.0
A
B
0.5
0.50
0.0
0.00
0
-3
10
-2
20
-1
30
0
40
1
50
2
60
3
Proficiency
Language A
Frequency
Language B
-0.50
-1.00
0
-3
10
-2
20
-1
30
0
40
1
50
2
60
3
Proficiency
31
Figure 1-1 Item characteristic curves for possible DIF patterns
ICCs: No DIF
Figure 1-2
1.0
.9
.9
.8
.8
.7
.7
Probability
Probability
Figure 1-1
1.0
.6
.5
.4
.3
ICCs: DIF in bs
.6
.5
.4
.3
.2
.1
0.0
-3
.2
Ref.
.1
0.0
-3
Foc.
-2
-1
0
1
2
3
Ref.
Foc.
-2
-1
Ability Score
ICCs: DIF in as
Figure 1-4
1.0
1.0
.9
.9
.8
.8
.7
.7
.6
.5
.4
.3
2
3
ICCs: DIF in as, bs, and cs
.6
.5
.4
.3
.2
.1
0.0
-3
1
Ability Score
Probability
Probability
Figure 1-3
0
Ref.
Foc.
-2
-1
0
Ability Score
1
2
3
.2
.1
0.0
-3
Ref.
Foc.
-2
-1
0
1
2
3
Ability Score
32
Checking for Method
Equivalence
1. Any biases due to directions,
exam format, scoring?
2. Preparation of exam
administrators?
3. Equivalence of exam taking
skills?
33
8 Steps for Adapting Exams
8. Documenting the process.
-Steps in exam adaptation,
details, evidence for validity.
34
International Test Commission
Guidelines for Adapting
Educational and Psychological
Instruments
Prepared By
International Test Commission
and
Seven International Organizations
(EAPA, ETP, IACCP, IAAP, IEA, ILTA, IUPsyS)
35
Instrument Adaptation Guideline
A practice which is judged as
important for conducting and
evaluating the adaptation or
parallel development of
psychological and education
instruments for use in different
populations.
36
Organization
of the 22 ITC Guidelines
Context (2)
 Instrument Development and
Adaptation (10)
 Administration (6)
 Documentation/Score
Interpretations (4)

37
Organization of Each of the 22
ITC Guidelines
Description of the Guideline
 Rationale/Explanation
 Steps to Meet the Guideline
 Common Errors
 References for Additional Study

38
D.3
Test developers/publishers
should provide evidence that
the choice of testing
techniques, item formats, test
conventions, and procedures
are familiar to all intended
populations.
39
D.8
Test developers/publishers should
provide information on the
evaluation of validity in all
target populations for whom the
adapted versions are intended.
40
D.9
Test developers/publishers should
provide statistical evidence of
the equivalence of questions for
all intended populations.
41
Three Main Conclusions
1. Has progress been made in
test adaptation methodology?
-Definitely!
-Moved from single unqualified
translator and bilingual design
to considerably more
sophisticated methodologies.
42
Conclusions, cont.
-Emergence of ITC guidelines;
methodological advances in
judgmental designs and
statistical methods (e.g., DIF,
SEM);more resources and time
being allocated.
43
Conclusions, Cont.
2. What lies ahead for the field
of test adaptations?
-I’m very optimistic about the
future.
-Needed is commitment of
resources and time, and
implementation of existing
methods.
44
Conclusions, Cont.
3. What are the most important
areas for improvement?
-choose multiple qualified
translators and give them
sufficient time
-aggressively implement empirical
methods
-build on past experiences and
knowledge base
45
For more information about exam
translation and adaptation,
contact Ron Hambleton at
Rkh@educ.umass.edu
46