Translation assessment
Dr. Constanza Gerding Salas
Based on Sonia Colina’s Translation Teaching (2003)
Universität Leipzig -Universidad de Concepción, May 2012
Assessment today
Evaluation vs. assessment (Colina, 2003)
Evaluation refers to the quality of the translation product
regardless of the method and procedures used.
Assessment uses the translation product to determine the
translator’s competence.
Leipzig -Concepción, 2012
Quality assessment
Qualitative statements can be made on the basis of a
comparison between TT and ST, bearing in mind the brief and
the situational context (House, 1997).
A communicative approach to the evaluation of translations
should consider error assessment focused on the
communicative function of the word, phrase or sentence in
question (Kussmaul, 1995).
Evaluators should restrict themselves to the effect the error
has on the target reader (Kussmaul, 1995).
Leipzig -Concepción, 2012
Purpose of evaluation
Summative vs. formative assessment (Hatim & Mason, 1997).
Summative assessment: to establish whether a translator
(student) is fit to be accepted in a translation program, move on
to the next level, practice as a professional translator, etc.
Formative assessment serves as an aid to the learning process by
providing continuous feedback on the development of translator
competence.
Leipzig -Concepción, 2012
Leipzig -Concepción, 2012
Achievement vs. proficiency testing
Achievement testing (formative) determines whether the
goals and requirements specified in the syllabus have been
met.
Proficiency testing (summative) measures ability in relation
to that required for a particular course of action, e.g. work
as a professional translator (Colina, 2003).
Leipzig -Concepción, 2012
Testing translation
Testers should consider the following questions:
1. What is translation competence?
2. What am I looking for in assessing translation
competence?
3. Can multiple translation sub-competences be
distinguished?
A working definition of competence must be established
prior to test elaboration.
Teaching and testing materials must be designed on the
basis of such definition (Colina, 2003).
Leipzig -Concepción, 2012
Componential model
Separate componential scoring criteria could be used for
testing multi-componential translation skills (Bachman, 1990; Cao,
1996).
E.g. language competence criterion, knowledge structures
criterion, translation strategies criterion, etc.
Assessment frequently shows these weaknesses:
• No brief
• No relationship between test and class practice
• Test selection based on text difficulty only
Leipzig -Concepción, 2012
Test format
A good test should meet at least these four criteria (Carroll, 1980):
1. Economy: obtaining the maximum amount of information
about student’s competence with a minimum of effort on
part of the test designer.
2. Relevance: a match between the test and the curriculum
and course goals.
3. Acceptability: the learners’ satisfaction that the test
evaluates their progress.
4. Comparability: similar scores across learners for different
tests for the same group of learners .
Leipzig -Concepción, 2012
Checklist for a quality translation test





Does it provide a brief and/or pragmatic information?
Does it include format specifications?
Does the test have a stated purpose?
Has it been designed accordingly?
Does it specify what competences are to be tested? (Not
necessarily on the test itself).
 Is it economical, relevant, acceptable and comparable?
 Does it state clear grading criteria?
 Does it test what was taught and how it was taught?
Leipzig -Concepción, 2012
Errors in translation
Nord (2009: 236) proposes a definition of error in translation
based upon the principles of translation difficulties and
translation problems.
There are no standards for every translation case. In
functional translation, the normative framework to assess
quality is given by the translation brief.
Errors are identified by comparing the student’s TT with the
profile of an ideal translatum derived from a brief and
considering specific translation problems (Nord, 2009).
Leipzig -Concepción, 2012
Defining translation error
• A translation error is any violation of the standards
stipulated in the translation brief with respect to specific
functional aspects (Nord, 2009).
• A translated text can only be evaluated in relation to a given
translation aim. Translators must know that aim.
• A translation task must be feasible, i.e. in accordance with
student’s knowledge level.
• Time restrictions and quality standards must correspond to
task’s degree of difficulty (Nord, 2009).
Leipzig -Concepción, 2012
Identifying errors (Nord, 2009)
• Most translation problems are identified during pre-translation
analysis, having situation, pragmatics, culturemes, and
linguistic structures in mind.
• The horizon of expectations should consider only those
translation problems whose solution is to be assessed.
• Each adequate solution is given a number of positive points.
The addition of these points makes up the score that should
cover the solution to all translation problems included in the
horizon of expectations.
• The student’s mark should be calculated on the basis of the
percentage of adequate solutions given (Nord, 2009).
Leipzig -Concepción, 2012
Factors determining difficulties (errors)
According to Nord (2009), the degree of difficulty of a
translation task is determined by:
• Textual difficulties: complexity y specificity of ST
• Technical difficulties: number and quality documentation
sources available
• Professional difficulties: translation brief (TT functions,
readers, means, perfection degree demanded for the TT
• Competential difficulties: students’ a) linguistic, cultural
and translational level and b) general and specific
knowledge
Leipzig -Concepción, 2012
Nord’s error categories
Nord (2009) proposes a broad error taxonomy based on the
translation problem categories and the inadequate solution or
the absence of a solution in relation to a given brief. The
categories are listed in order of importance:
1.
2.
3.
4.
pragmatic
cultural
linguistic
extraordinary
PROBLEM
PTP =
CTP =
LTP =
ETP =
Leipzig -Concepción, 2012
ERROR
PTE
CTE
LTE
ETE
Error weight
The weight of an error will depend on how much it affects
the intelligibility of the TT in relation to the brief and the
communicative situation.
According to Nord (2009), cultural errors do not always
impede understanding but they only “hinder”
communication.
Leipzig -Concepción, 2012
Factors affecting grading (Kussmaul, 1995)
•
•
•
•
•
Grading is not simple
Should pragmatic errors weigh more heavily?
How far-reaching is the mistake?
Does it distort meaning?
Does it hinder/destroy communication?
Pym (1992: 279) proposes ‘binary’ and ‘non-binary’ errors:
• Binary error: clearly wrong choices
• Non-binary errors: based on the choice from potential TT
variants (‘It’s correct, but…)
Both authors recommend the non-binary error approach
Leipzig -Concepción, 2012
Error category for evaluation (Kussmaul, 1995)
1.
2.
3.
4.
5.
Cultural adequacy
Situational adequacy
Speech acts (what was said & implied, & its effect)
Meaning of words (misinterpretation of text)
Language errors: Do they impede comprehension or
bring discredit on the author?
6. ST defects. Can meaning be inferred?
Leipzig -Concepción, 2012
Grading criteria
Colina (2003: 136) proposes a descriptive profile, i.e. a
numerical score that is described according to rating criteria:
• 5 = TT reveals careful consideration of brief and very
appropriate translation solutions.
• 3 = TT reveals consideration of brief and acceptable
translation solutions.
• 1 = TT reveals no consideration of brief and poor or
inconsistent translation solutions.
(Model does not provide explicit description of 2 and 4).
In componential rating, each skill tested is graded separately
and final mark is obtained from weighted average.
Leipzig -Concepción, 2012
Multi-componential grading criteria
Colina (2003) proposes a 100-point grading criteria rubric
based on five components:
For component descriptions, see Colina 2003: 137-138.
Leipzig -Concepción, 2012
Translation competence grading criteria
Hatim & Mason (2007) based their model on Bachman’s
paradigm (1990).
It consists of two types of skills:
1. ST & TT processing skills (40 points)
• Intertextuality & rhetorical structure (genre, text type)
• Texture & structure (lexical choice, syntax, cohesion)
• Situationality (context)
2. Transfer skills (60 points)
• Effectiveness
• Efficiency
• Relevance
Leipzig -Concepción, 2012
Translation competence grading criteria:
Cao’s model (1996)
A. Translation Language competence (30 pts.)
1. Organizational: grammatical/textual
2. Pragmatic
B. Knowledge structures (20 points)
1. Cultural
2. Topical
C. Strategic competence (50 points)
Criticism: this model assigns between 25% and 30% of the
score per category even if competence is assessed as
‘deficient’ or ‘lacking’.
Leipzig -Concepción, 2012
Colina’s scale for evaluating translation
competence (2003: 143-144)
1.
2.
3.
4.
5.
ST dependence/adequacy: 25 points
TT adequacy: 25 points
Vocabulary, specialized knowledge, research (15 points)
Functional & textual aspects (25 points)
Revision process (10 points)
Description of all categories given by author.
Leipzig -Concepción, 2012
Useful strategies (Kussmaul, 1995: 150-152)
1. Find out the exact function or purpose of your translation
2. Classify your translation problems
3. Use the parallel-activity technique to remove mental blocks
4. Remember there is divergent thinking and flexibility
5. Verify if SC & TC coincide or diverge about that problem
6. Use parallel texts to find out about TT conventions
7. Do not stick to formal equivalents; be creative
8. Use notions, pictures and experiences in your memory
9. Make use of situational analysis
* Informants are the best dictionaries
Leipzig -Concepción, 2012
Elaborating tests for the CTC
Student’s checklist
 Translation brief and text formalities
 Text types and genres
 Pragmatic factors in ST & TT
 Decision-making
 Research and documentation
 Parallel text analysis
 Translation
• ST analysis and comprehension
• Reformulation
• Revision
Leipzig -Concepción, 2012
Sample Test
A. Instructions
• Read the whole text before translating.
• Type your name in full on the top right corner of a new blank document of
Word.
• Save the document at once under your name, not “Test 5”.
• Format: margins, font type and size, spacing and page size similar to those of the
original.
• Fit the target text on one page only.
• Time assigned for the whole task: 90 minutes (589-word text).
• Draft translation should be finished in half the period so as to have time to revise
TT.
• When ready, close the file and send it to cgerding@udec.cl .
• Do not include source text, instructions or brief in your document.
Leipzig -Concepción, 2012
Test text commission
B. Commission
• This editorial is a complete text unit
• Translate the whole text into Spanish
• The target text is intended to be published in El Mercurio, Santiago,
Chile, in the form of an opinion column. The text should be sourceoriented.
C. Pragmatic information
• Source: The Opinion Pages of The New York Times
• Means: http://www.nytimes.com/2012/01/09/opinion/haitis-slowrecovery.html?scp=2&sq=editorial%20haiti&st=cse
•
•
•
•
Target readership: mainly Chilean adults
Type of readership: conservative, educated adult readers
Register: in accordance with readership and publication
Text function: informative, right-wing focus
Leipzig -Concepción, 2012
Other possible test items
• Cloze passages (open or multiple-choice) to supply the
missing word(s)
• Brief writing
• Précis writing
• Short answer, multiple-choice or T/F questions on content
discussed in class, e.g.:
 What is an editorial? Who writes the editorial in a newspaper?
(Text type and function)
 What is the difference between ‘un editorial’ and ‘una
editorial’? (SL culture; language issue)
 Are there any differences between newspaper editorials in the
USA and in Chile? (Cross-culture)
 What are the main translation problems of editorials?
(Translation competence)
Leipzig -Concepción, 2012
Thank you!
Leipzig -Concepción, 2012