TITLE:
Methodological issues in automatic recognition of pointing signs in Langue des signes
Québécoise using a 3D motion capture system
LANGUAGE: English
LONG ABSTRACT:
Sign languages use space to identify and track referents within a discourse. For example, established loci can be pointed to in order to refer anaphorically to previously introduced referents. Prior works on the use of space in sign languages are mainly based on qualitative analysis. With respect to the articulatory possibilities in the production of sign discourse, there is an increasing interest among sign language researchers in understanding and quantitatively describing the use of space, especially for sign recognition and animation purposes. Jantunen et al . (2012) point out that most motion capture studies on sign language have consisted of relatively small sets of isolated expressions (single signs (Wilbur, 1990), phrases (Tyrone et al .,
2010)), and that the collection of biomechanical discourse-type data has been marginal (cf.
Duarte and Gibet, 2010). Even though recording, processing and analyzing biomechanical data is time consuming, we believe that the understanding of the interaction of signs in space in discourse has to be analyzed using discourse data. This requires a robust marker set in order to automatically recognize and identify specific signs such as pointing signs.
The aim of this paper is to discuss the methodological issues we have encountered collecting 3D discourse data in Langue des signes québécoise (LSQ) for morphosyntactic, semantic and pragmatic analysis of the use of four strategies of spatial association (pointing sign, localization, eye gaze, body shift). We will focus on one of the issues that we believe to be fundamental in biomechanical data collection, i.e. marker placement on the hand to ensure the possibility of automatic recognition of pointing signs. Due to the great variability of hand shapes and types of contact in sign languages, the movements of the fingers involved are quite less regular than those in handling activities (e.g. prehension task). Thus, the post-processing (manual identification of markers) becomes a really complex task, especially if the number of markers is important. One of the problems that we face as researchers is the lack in the literature of a described procedure for collecting such data, i.e. the description of a minimal marker set that can be used without affecting the reliability of the data.
We will discuss, using LSQ discourse data, the evolution of our marker set (limits of the former marker set, strengths of the actual one) and the biomechanical criteria to automatically isolate pointing signs in Visual-3D (a powerful biomechanical analysis software). Those criterion are: 1) the variation of the distance between markers on the index and middle fingers, 2) the variation of the angle based on markers on the index and the marker on the protrusion of the radius on the internal side of the wrist, 3) the extended position of the index finger with respect to the position of the other fingers. We will show that both the marker set and the biomechanical criteria allow the automatic recognition of pointing signs in LSQ without false positives (signs incorrectly identified as pointing signs). This paper will benefit researchers interested in automatic recognition of signs.

References:
Duarte, K. and S. Gibet (2010). “Reading Between the Signs: How are Transitions Built in
Signed Languages?” Paper presented at TISLR 10, West Lafayette, IN, September 2010.
Jantunen, T., B. Burger, D. De Weerdt, I. Seilola and T. Wainio. (2012) “Experiences from collecting motion capture data on continuous signing”, 5th Workshop on the Representation and Processing of Sign Languages (Interactions Between Corpus and Lexicon) , organized as a part of LREC 2012 in Istanbul, Turkey, 27 May.
Tyrone, M., H. Nam, E. Saltzman, G. Mathur, and L. Goldstein. (2010). “Prosody and Movement in American Sign Language: A Task-Dynamics Approach”. Speech Prosody , pp. 1-4.
Wilbur, R. (1990) “An Experimental Investigation of Stressed Sign Production”. International
Journal of Sign Language , 1, pp. 41-59.