Holistic vs Analytical Assessment
in Legal Translation
Carmen Valero-Garcés & Francisco Vigier – University of Alcalá
Mary Phelan – Dublin City University
• Assessment in Translation Studies and
Professional Practice
• Introduction to HA
– Research Study on HA in Legal Translation
• Introduction to AA
– Research Study on AA in Legal Translation
• Conclusions
2
Assessment in Translation Studies and
Professional Practice
• Underresearched area
• Common problems in TQA (Williams 2009)
– The evaluator
– Level of target language rigour
– Seriousness of errors
– Sampling vs full-text assessment
– Quantification of quality
– TQA purpose
3
What is Holistic Assessment?
• The evaluator gives a TT a rating (0-10) or evaluative letter (e.g. A
= excellent, B = very good) based on an overall impression
• Frequently used in both academia and industry
• Advantages  less time-consuming and assessment of
translations at the discourse/text level not at the sentence/word
level (Garant 2009)
• Some attempts of systematization (Waddington, 2001)
• Disadvantages  subjective, hence arbitrary, intuitive,
unscientific, unsystematic and unreliable; does not provide a clear
justification of the result (Waddington 2001)
4
Research Study on HA in Legal
Translation
• Analyse strengths and weaknesses of holistic methods for the
assessment of legal translation ( interrater reliability)
• One of the WS1 essential documents translated into SP by a
student on MA in Translation
• That translation assessed numerically (0-10) by ten evaluators
• Evaluators surveyed on their assessment method
5
6
Results
• Numeric assessment
7
Results (2)
• Survey
– Most evaluators ranked pragmatic errors as those with highest
relevance and linguistic errors as those with lowest relevance
– Very different opinions expressed by respondents as to the
translation’s strengths and weaknesses (i.e. “The message is
appropriately conveyed. It fulfills its communicative function”
vs. “Errors regarding sense, coherence, punctuation... A poor
quality translation”  assessment is based on personal
criteria, thus subjective and variable
8
Analytical Assessment
ANALYTICAL ASSESSMENT
9
ATA
• ATA system – (a) grid, (b) flowchart and (c)
Explanation of Error Categories
10
ATA Grid
ATA CERTIFICATION PROGRAM
Exam Number:
FRAMEWORK FOR STANDARDIZED ERROR MARKING
Exam Passage:
Version 2009
1
2
Check here if for Review 
4
8
16
Code
Reason
UNF
Unfinished (if a passage is substantially unfinished, do not grade the
Errors that concern the form of the exam
Treat missing material within the passage as an omission
exam)
ILL
Illegibility
IND
Indecision, gave more than one option
Translation/strategic/transfer errors: Negative impact on understanding/use of target text
MT
Mistranslation (use a subcategory if possible)
MU
-
Misunderstanding of source text (if identifiable)
A
-
Addition
O
-
Omission
T
-
Terminology, word choice
R
-
Register
F
-
Faithfulness
L
-
Literalness
FA
-
Faux ami (false friend)
COH
-
Cohesion
AMB
-
Ambiguity
ST
-
Style (inappropriate for specified type of text)
OTH
-
Other (describe)
Mechanical errors: Negative impact on overall quality of target text. Points may vary by language. Maximum 4 points
G
Grammar
SYN
-
Syntax (phrase/clause/sentence structure)
P
Punctuation
SP/CH
Spelling/Character (usually 1 point, maximum 2, if more than 2 points,
another category must apply)
0
0 x 2
0 x 4 = 0
0 x 8 = 0
D
Diacritical marks/ Accents
C
Capitalization
WF/PS
Word form/ Part of speech
U
Usage
OTH
Other (describe)
0
Column totals
A grader may stop marking errors when the
A grader may award a quality point for each of
Quality points are subtracted from the error point total to yield a final
score reaches 46 error points
up to three specific instances of exceptional
score. A passage with a score of 18 or more points receives a grade of
translation
Fail.
Quality points (maximum 3) 0
Final passage score (subtract quality points from error points) 0
=0
Total error points (add column totals): 0
11
ATA flowchart
12
• UAH text – holistic - 532 words in ST
• DCU text – analytical – 256 words in ST
• 5 assessors – three in Europe plus two ATA
assessors
13
Evaluators’ Verdicts
Assessor
Score
Verdict
Spanish evaluator 2
9
Pass
Spanish evaluator 3
16
Pass
Spanish evaluator 1
23
Would accept it with
reservations
ATA evaluator 1
45+
Fail
ATA evaluator 2
43
Fail
14
Conclusions
• HA: subjective method with a low degree of
inter-rater reliability
• Cost and time efficiency  HA as
supplementary method for LT assessment?
• AA: even though the system appears selfexplanatory, there is a lot of variation in the
overall result.
• AA: The ATA evaluators have years of
experience of using this method.
15
References
Garant, M. (2009). A case for holistic assessment. AFinLAe Soveltavan kielitieteen tutkimuksia 2009, 1, 5-17.
Waddington, C. (2001b). Should translations be assessed
holistically or through error analysis? Hermes, Journal
of Linguistics, 26, 15-38.
Williams, M. (2009). Translation Quality Assurance.
Mutatis Mutandis, Vol 2, No 1., 3-23
16
Thank you!
Carmen Valero-Garcés
Francisco Vigier
Mary Phelan
carmen.valero@uah.es
francisco.vigier@uah.es
mary.phelan@dcu.ie