Multimodal feedback mechanisms in interpreter-mediated interaction
Jelena Vranjes, Hanneke Bot, Kurt Feyaerts, Geert Brône
In interpreter-mediated dialogues, the interpreter’s overall task is to create mutual
understanding of the subject matter and establish communicative contact and rapport
between the primary participants who have no access to each other’s language (Linell,
Wadensjö & Jönsson 1992, Pasquandrea 2011). However, the presence of the interpreter
may result in difficulties for the primary speakers to address each other directly and to
display signs of acceptance or mutual understanding (Linell, Wadensjö & Jönsson 1992,
Davidson 2002). It is therefore assumed that multimodal resources such as nods, hand
gestures and specific gaze patterns are of particular importance in this kind of complex
interaction (cf. Pasquandrea 2011), especially for the collaborative process of grounding as
described by Clark (1996). This grounding process, in which interlocutors try to establish
mutual understanding, becomes most evident in the acceptance phase, when the addressee
provides some kind of (verbal or non-verbal) feedback to the speaker’s utterance (cf. Clark
1996). Previous studies on grounding mechanisms in interpreter-mediated interaction have
mainly paid attention to verbal feedback (Linell, Wadensjö & Jönsson 1992; Davidson 2002).
Davidson (2002) provides a collaborative model of interpreted discourse based on the
stipulation that there is not one, but two separate sets of common ground constructed
between the interpreter and each of the two interpretees (Davidson 2002). However, what is
missing in this model is the process of grounding between the two primary participants,
which, we argue, may be explored on the non-verbal level.
In the present study, we focus on the role of eye gaze in relation to other (non-)verbal signals
in providing and eliciting feedback signals in the specific institutionalized setting of
interpreter-mediated dialogue. Our study builds on existing research on grounding in
monolingual face-to-face conversation through gaze and other multimodal resources
(Bavelas 2002, McNeill 2006, Allwood & Cerrato 2003). In order to determine how much
communicative contact is displayed by both interpreter and primary participants on verbal
and non-verbal level, the present study provides a detailed account of temporal dynamics
and multimodal packaging (McNeill 2006) of feedback signals. We also take into account
how specific feedback patterns are linked to the communicative context of the interaction and
to the specific interactional roles (cf. Goffman 1981, Wadensjö 1998, Roy 1999) in the
interpreter-mediated interaction.
We recorded a set of interpreted encounters by making use of unobtrusive mobile eyetracking glasses (Gullberg & Kita 2009, Jokinen 2010, Brône & Oben 2013). Eye-tracking
allows for the detailed study of interlocutors’ gaze patterns and at the same time captures
other semiotic channels, such as gesture and posture.
The analysis of our data reveals specific patterns of gaze distribution related to grounding
signals in the interpreted dialogue. We also find indications of a strong temporal correlation
between eye gaze, speech and gesture in the establishment of common ground. Our study
not only reveals multimodal feedback patterns between the interpreter and the two primary
speakers, but also between the primary participants themselves. The results are thus
indicative of the establishment of common ground at different levels of communication in
interpreter-mediated interaction.
REFERENCES
Allwood, J. & Cerrato, L. 2003. A study of gestural feedback expressions. In: Paggio, P., et al. (eds.)
Proceedings of the First Nordic Symposium on Multimodal Communication: 7-22.
Bavelas, J., Coates, L. & Johnson, T. 2002. Listener responses as a collaborative process: The role of
gaze. Journal of Communication, 566-580.
Brône, G. & Oben, B. 2013. InSight Interaction. A multimodal and multifocal dialogue corpus.
Language Resources and Evaluation.
Clark, H. H. 1996. Using Language. Cambridge: Cambridge University Press.
Davidson, G. 2002. A model for the construction of conversational common ground in interpreted
discourse. Journal of Pragmatics 34: 1273-1300.
Goffman, E. 1981. Forms of Talk. Philadelphia: University of Pennsylvania Press.
Gullberg, M. & Kita, S. 2009. Attention to speech-accompanying gestures: Eye movements and
information uptake. Journal of Nonverbal Behaviour 33: 251-277.
Jokinen, K. 2010. Non-verbal signals for turn-taking & feedback. Proc. of 7th Int. Conf. on Language
Resources & Evaluation (LREC) International Universal Communication Symposium.
Linell, P., Wadensjö, C. & Jönsson, L. (1992): Establishing Communicative Contact through
a Dialogue Interpreter. In: Grindsted, A., Wagner, J. (eds.) Communication for Specific
Purposes - Fachsprachliche Kommunikation. Tübingen: Gunter Narr Verlag, 125-142.
McNeill, D. 2006. Gesture, gaze and ground. In: Renals, S., Bengio, S. (eds.) Proceedings of Machine
learning for multimodal interaction: Second international workshop 2005, Berlin/Heidelberg:
Springer Verlag, 1-14.
Oben, B. & Brône, G. forthc. What you see is what you do. On the relationship between gaze and
gesture in multimodal alignment. Language & Cognition.
Pasquandrea, S. 2011. Managing multiple actions through multimodality: Doctors’ involvement in
interpreter-mediated interactions. Language in Society 40(4): 455-481.
Roy, Cynthia, 1999. Interpreting as a discourse process. Oxford Studies in Sociolinguistics. Oxford
University Press, New York.
Wadensjö, C. 1998. Interpreting as interaction. London and New York: Longman.