Modeling the semantics of motion verbs based on sensorimotor measurements
We combine linguistic knowledge from corpus data with sensorimotor data obtained experimentally in
an effort to better specify the minimum conceptual representation of a motion event that distinguishes
it from all other events. Sensorimotor data are collected by measuring the performance of speakers of
Modern Greek. As sensorimotor data allow for optimal linking between embodied experience and
language (Johnson and Lakoff, 1999), we aim to (i) perceptually ground abstract notions, such as
agentivity, typically used in linguistics to study and classify verbal semantic and syntactic properties
and, (ii) shed some light to long standing syntactic questions, eg. the troublesome minimal pair
argument/adjunct (FrameNet; Galen, Grenager & Manning, 2004 ).
We draw from the extensive prior work related to Cognitive Science (Jackendoff, 1990;
Feldman, 2006), Mirror Neurons and their impact on language (Fadiga et al, 2006; Arbib, 2008;
Kemmerer, 2006) and Computer Vision (Santello, 1998). Here, we focus on the clustering of motor
actions and its correspondence to a previous linguistic classification of Modern Greek verbs of motion
and position (MGMV) according to the prototype theory (Antonopoulou, 1987). The detailed sensor
data are analyzed to identify latent factors that represent stable patterns across the many dimensions of
low level data. These factors appear as discrete sets (synergies) of joint angles and orientations
associated with each action. They are correlated with the classificatory linguistic properties, both the
minor, e.g. path and directionality, and the major ones, e.g. related/graded quality in transitivity.
The transitive and intransitive verbs selected for the experiments denote actions with one
human participant. Performers were asked to keep to literal meanings mainly due to lab limitations. In
this pilot study, three subjects performed 22 verbs (29 actions). Data were projected on a twodimensional PCA plot, which visualizes the groups formed by the captured actions. So far, two main
clusters of actions have formed that significantly overlap with the linguistic ones (Antonopoulou,
1987). The first cluster comprises walking-like actions, denoted by verbs linguistically classified under
the prototypical walk, run, jump, ascend and descend. Characteristically, ascend and descend are
virtually projected on the same coordinate on PC2 that strongly weights flexion and extension of knees
and elbows equally throughout the duration of the action. Flexion and extension of knees are
indicators of movement along the vertical axis and the revolution around the y axis in a 3D feature
space. Consequently, the two verbs are clustered together because of the common feature of vertical
direction that appropriately generalizes upwards and downward motion usually employed in prototype
theory classification.
The second major cluster comprises actions involving the manipulation of an object or self that
are denoted by transitive verbs, such as turn or lift an object or self. PC2 separates the two groups due
to the distinct use of elbows and the emphasized shoulders when raising and lowering objects. We
conclude with directions for our ongoing work, especially concerning the repetition of the same
procedure for English and the comparison between the two languages.
References
Antonopoulou, E. (1987). Prototype theory and the meaning of verbs, with special reference to
Modern Greek verbs of motion. Unpublished PhD thesis. University of London
Arbib, M. A. (2008) From Grasp to Language: Embodied Concepts and the Challenge of Abstraction.
Journal of Physiology Paris 102: pp. 4-20
Fadiga, L., et al. (2006). Language in shadow. Social Neuroscience, 1(2): pp.77-89.
Feldman, J.(2006). From Molecule to Metaphor. Cambridge, MA: MIT Press.
Jackendoff, R. (1990). Semantics Structures. Cambridge, MA: MIT Press.
Lakoff, G.& Johnson, M. (1999). Philosophy in the Flesh: The Embodied Mind and Its Challenge to
Western Thought. , Basic Books, New York
Kemmerer, D. (2006). Action verbs, argument structure constructions, and the mirror neuron system.
In M. Arbib (Ed.), Action to language via the mirror neuron system. Cambridge University Press.
Galen, A, Grenager. T. & Manning C. (2004). Verb Sense and Subcategorization: Using Joint
Inference to Improve Performance on Complementary Tasks. EMNLP 2004: pp. 150-157.
Santello, M., Flanders, M., & Soechting, J.F. (1998). Postural hand synergies for tool use. Journal of
Neuroscience, 18.