An Analysis-By-Synthesis Approach to Multisensory Object Shape Perception Ilker Yildirim Brain & Cognitive Sciences Massachusetts Institute of Technology Cambridge, MA ilkery@mit.edu Goker Erdogan Brain & Cognitive Sciences University of Rochester Rochester, NY gerdogan@bcs.rochester.edu Robert A. Jacobs Brain & Cognitive Sciences University of Rochester Rochester, NY robbie@bcs.rochester.edu 1 Introduction The world is multimodal.1 We sense our environments using inputs from multiple sensory modalities. Similarly, digital information is increasingly available through multiple media. In this extended abstract, we present a general computational framework for understanding multimodal learning and perception that builds on the analysis-by-synthesis approach [2, 3]. The analysis-by-synthesis approach makes use of generative models describing how causes in the environment give rise to percepts. It achieves an analysis of the input as a synthesis of these causes by inverting this generative model. This approach has been successfully applied in a wide range of fields from vision[2] to language[3]. Our framework is a natural extension of this approach to multimodal perception. From the perspective of Marr’s tri-level hypothesis2 , our framework can be understood as a computational level analysis of multimodal perception. It offers a conceptual analysis of multimodal perception that enables a unified treatment of a diverse set of questions. As a proof of concept of our framework, we apply it to object shape perception via visual and haptic modalities, and show that it captures people’s behavior in this task with high accuracy. We argue that any cognitive agent capable of multimodal perception can be analyzed in terms of three crucial components [1]. The first component is a representational language for characterizing modality-independent representations. For example, in the case of multimodal object recognition, this representational language characterizes the object shape information necessary for recognition extracted from the sensory inputs. These modality-independent representations need to be mapped to sensory inputs for the purposes of learning and inference. This is where the second component, sensory-specific forward models [5], come into play. For example, a vision-specific forward model maps object shape representations to visual inputs, i.e., images. However, perception goes in the other direction, from sensory inputs to multimodal representations. Therefore, the last component is an inference procedure for inverting forward models. Taking a probabilistic perspective, we think of forward models as specifying a probability distribution over sensory inputs D given a multimodal 1 This extended abstract is a short version of our manuscript titled “From Sensory Signals to ModalityIndependent Conceptual Representations: A Probabilistic Language of Thought Approach” published in PLOS Computational Biology[1]. 2 Marr[4] distinguished between three different levels at which one can understand a cognitive capability. The computational level characterizes what is the computational problem that needs to be solved. The algorithm/representation level specifies the representations and the algorithms used to solve the problem. Lastly, the implementation level is concerned with how the solution is implemented in physical medium. 1 representation H, i.e., P (D|H). One can use the calculus of probability to invert this forward model via Bayes’ rule. A similar framework has been successfully applied to problems in biological and computer vision, and our framework is a natural application of this analysis-by-synthesis [2] approach to multimodal perception. We believe that our framework presents a promising direction for understanding multimodal perception as it presents a unified treatment of various questions. For example, how is knowledge from one modality transferred to another? In our framework, this simply becomes estimating the posterior of observing some sensory input from one modality given another sensory input from another modality. Or, how is information from multiple modalities combined [6]? Again, this simply becomes estimating the posterior over multimodal representations given inputs from multiple modalities. 2 Our framework applied to multimodal perception of object shape In the rest of the abstract, we focus on the application of our framework to multimodal shape perception in visual and haptic settings. We present a specific instantiation of our model for this problem and show that it provides a highly accurate account of human subjects’ shape similarity judgments. We used a set of 16 objects in our experimental and computational work (see Figure 1). Each object consists of five parts at five fixed locations. One part is common to all objects. At each of the remaining four locations, one of two possible parts is chosen. We characterize these objects using a shape grammar that captures the identity and spatial configuration of the parts that make up an object. Production rules for this probabilistic context-free grammar can be seen in Figure 2. Note that this grammar characterizes only the coarse spatial structure of an object, and it needs to be combined with a spatial model, which will be explained in more detail shortly. There are two non-terminal symbols in our shape grammar. Non-terminal symbol S corresponds to spatial nodes in the grammar that determine the positions of parts in 3D space. The spatial model associates positions in 3D space with each one of these S nodes. P nodes are placeholder nodes for parts which are denoted by terminal nodes P 0 through P 8 (Figure 3). The grammatical derivation for an object can be represented using a parse tree. However, as mentioned above, this parse tree does not characterize the spatial structure of an object completely. Hence, a parse tree is extended to a spatial tree by combining it with a spatial model as follows (see Figure 4). Parts are positioned in a 3D space using a multiresolution representation. At the coarsest resolution in this representation, a “voxel” corresponds to the entire space. The center location of this voxel is the origin of the space, denoted (0, 0, 0). The root S node of a grammatical derivation of an object is always associated with this origin. At a finer resolution, this voxel is divided into 27 equal sized subvoxels arranged to form a 3 × 3 × 3 grid. Using a Cartesian coordinate system, a coordinate of a subvoxel’s location along an axis is either -1, 0, or 1. Even finer resolutions are created by dividing subvoxels into subsubvoxels (again, using a 3 × 3 × 3 grid), subsubvoxels into subsubsubvoxels, and so on. S nodes in an object’s grammatical derivation (other than the root S node) are associated with these finer resolution voxels with the constraint that the position of an S node must lie within its parent S node’s voxel (see Figure 4). In other words, each level in the parse tree is associated with a specific level of resolution. For example, for the child S nodes of the root S node, the whole space is split into 27 voxels, 3 along each axis, and each S nodes is assigned one of these voxels. Similarly, for the next level in the tree, each voxel is further split into 27 subvoxels, and each S node at this level can be associated with any of the 27 subvoxels of its parent voxel (hence, there are 27 × 27 voxels in total at this level.). The position of a part node P is simply its parent S node’s position, i.e., the center of the voxel associated with its parent. This assignment of spatial positions to S nodes extends a parse tree to a spatial tree by specifying the positions of the parts that make up an object. Let T denote the parse tree and S denote the spatial model, i.e., spatial assignments for each S node, for an object. The prior probability of a shape representation (T , S) is defined as: P (T , S|G) = P (T |G)P (S|T ) (1) where G denotes the shape grammar. The probability for a parse tree is defined based on the grammatical derivation associated with it: Y P (T |G, ρ) = P (n → ch(n)|G, ρ) (2) n∈Nnt 2 where Nnt is the set of non-terminal nodes in the tree, ch(n) is the set of node n’s children nodes, and P (n → ch(n)|G, ρ) is the probability for production rule n → ch(n). In this equation, ρ denotes the set of probability assignments to production rules and can be integrated out to get P (T |G) (see [9] for details). Letting V denote the set of voxels, the probability of spatial model S can simply be defined as follows since each S node in the parse tree is assigned one voxel: Y 1 1 P (S|T ) = . (3) = |NS | |V| |V| n∈N S Note that the set V simply consists of the voxels that can be assigned to a S node; this set always contains 27 elements since each S node is assigned one of 27 subvoxels of its parent’s voxel. Figure 1: Stimuli used in the experiment and by the computational model. S P → → S | SS | SSS | SSSS | P | P S | P SS | P SSS P0 | P1 | P2 | P3 | P4 | P5 | P6 | P7 | P8 Figure 2: Production rules of the shape grammar in Backus-Naur form. To map these modality-independent shape representations to sensory features, we make use of two forward models, one for each modality. For the vision-specific forward model, we use the Visualization Toolkit (VTK; www.vtk.org) graphics library to render images of objects. Given an object’s parse and spatial trees (T , S), VTK places the object’s parts at their specified locations and renders the object from three orthogonal viewpoints. These images are concatenated to form a single vector of visual inputs. For the haptic-specific forward model, we use a grasp simulator known as “GraspIt!” [7] (see Figure 5). GraspIt! enables us to measure the joint angles of a simulated human hand grasping a given object. We perform multiple grasps on an object as we rotate it around x, y, and z axes in increments of 45◦ (24 grasps in total), and use the joint angles (again, concatenating the joint angles for all grasps into a single haptic input vector) as our representation of sensory input for the haptic modality. The use of multiple grasps can be regarded as an approximation to active haptic exploration. Our haptic forward model is no doubt a crude approximation. We use joint angles as our sole haptic input and assume that an object can be grasped exactly in the same way across multiple trials. We believe these are necessary simplifications made partly in order to make the problem less computationally demanding, and partly due to lack of realistic haptic forward models in the literature. We define a probability distribution over sensory inputs given modality-independent shape representations by assuming Gaussian noise on sensory inputs. Hence, P (D|(T , S)) is defined as follows: ||D − F (T , S)||22 (4) P (D|T , S) ∝ exp − σ2 3 P0 P1 P2 P3 P4 P5 P6 P7 P8 Figure 3: Library of possible object parts. (a) (b) S(0,0,0) P S(0,-1,0) S(1,0,0) P0 P P P P1 P3 P5 S(-1,1,0) S(0,1,0) P P7 (c) Figure 4: (a) Image of an object. (b) Spatial tree representing the parts and spatial relations among parts for the object in (a). (c) Illustration of how the spatial tree uses a multi-resolution representation to represent the locations of object parts. where F denotes the forward model and σ 2 is a variance parameter. To estimate an object’s shape from visual and/or haptic inputs, we invert the forward models via Bayes’ rule to compute a posterior distribution over multimodal representations: P (T , S|D, G) ∝ P (T |G)P (S|T )P (D|T , S) (5) where the first term on the right-hand side is given by Equation 4 in [1], and rest are given here in Equations 3, and 4, respectively. Given the intractability of this distribution, we use a Metropolis4 Figure 5: GraspIt! simulates a human hand. Here the hand is grasping an object at three different orientations. Hastings algorithm, a popular Markov Chain Monte Carlo (MCMC) technique, to collect samples from the posterior. We make use of two different proposal distributions, one used on even-numbered and the other on odd-numbered iterations [8]. Our first proposal distribution, originally developed by [9], picks one of the nodes in the parse tree randomly, deletes all its descendants, and regenerates a random subtree from that node according to the rules of the grammar. The spatial model is also updated accordingly by removing the deleted nodes and picking random positions for newly added S nodes (see [1] for the acceptance ratio for this proposal). Our simulations showed that this “subtree regeneration” proposal by itself is not efficient, largely because it takes quite large steps in the representation space. Our second proposal distribution is a more local one that adds/removes a single part to/from an object. The acceptance probability for an add move3 is: P (D|T 0 , S 0 ) P (T 0 |G) |A| 0 0 A(T , S ; T , S) = min 1, |Gt | (6) P (D|T , S) P (T |G) |R0 | where R0 is the set of S nodes in tree T 0 that can be removed, A is the set of S nodes in tree T to which a new child S node can be added, and Gt is the set of terminal symbols in the grammar. Subtree regeneration and add/remove part proposals, when combined, enabled fast convergence to the posterior distribution. In our simulations, we ran each MCMC chain for 10,000 iterations (burn-in: 6,000 iterations). Samples from a representative run of the model are shown in Figure 6. Figure 6a shows the visual input to the model for this particular run. Samples from the posterior distribution, ordered according to posterior probability, are shown in Figure 6b. The MAP representation (leftmost sample) recovers the part structure and the spatial configuration of the object perfectly. This result is general—that is, our model always recovers the “true” representation for each object regardless of the modality through which the object is sensed. This demonstrates that the model shows modality invariance, an important type of perceptual constancy. 3 Behavioral evaluation of the model In our behavioral experiment, we investigated how people transfer shape knowledge across modalities. This can also be understood as a problem of crossmodal shape retrieval where one needs to measure how likely it is to observe some sensory input from one modality given sensory observations from another modality. In order to understand how well our model captures this aspect of human multimodal perception, we carried out the following behavioral experiment. We asked subjects to rate the similarity of shape between objects under two settings: crossmodal and multimodal.4 In the crossmodal condition, on each trial, subjects first viewed the image of one of the objects on a computer screen, and then haptically explored the second object (physical copies of the objects were created via 3-D printing) by reaching into a compartment where the second object was placed. The compartment prevented subjects from seeing the object. Subjects were asked to judge the shape similarity between the two objects on a scale of 1 to 7. In the multimodal condition, the procedure was the same except that subjects received both visual and haptic input for both of the objects. The experiment consisted of 4 blocks where in each block all possible pairwise comparisons of 16 objects were presented. We collected data from 7 subjects. 3 The acceptance probability for the remove move can be derived analogously and is given in [1]. Here we focus only on two conditions in our experiment; please refer to [1] for the full experiment, details on the experimental procedure, and a detailed analysis of our experimental results. 4 5 (a) (b) S(0,0,0) S(0,0,0) P S(0,-1,0) S(1,0,0) P0 P P P S(0,1,0) P1 P4 P6 P S(1,1,0) S(0,0,0) S(0,0,0) P S(0,-1,0) S(1,0,0) S(1,1,0) P S(0,-1,0) S(1,0,0) P0 P P P P0 P P P S(0,1,0) P1 P4 P6 P1 P4 P6 P P8 S(1,1,0) P S(0,-1,0) P0 P S(1,0,0) P P1 P4 P7 Figure 6: Samples from a representative run of our model. (a) Input to the model. (b) Samples from the posterior distribution over shape representations. Samples are ordered according to posterior probability from left to right. We evaluated how well our model captures subjects’ similarity judgments by comparing its similarity judgments to behavioral data. To obtain similarity ratings from our model for the crossmodal condition, we proceeded as follows. For each trial, we run two MCMC chains, one for each object. We call these the visual and haptic chains referring to the modality through which each object is presented. For the visual chain, the input is images of the object from three orthogonal viewpoints. For the haptic chain, we use the joint angles (for 24 grasps) obtained from GraspIt! as the sensory input. For the multimodal condition, we ran again two chains, one for each object, but provided both visual and haptic input. Here, we focus only on the MAP samples from these chains, and take the similarity between the MAP samples from the visual and haptic chains to be the crossmodal similarity judgment of our model. Similarly, the similarity between samples from the visual-haptic chains for the multimodal condition forms our model’s multimodal similarity judgments. Because our multimodal shape representations are trees, we used tree-edit distance [10] to measure the similarity of two MAP representations. We compare our model’s judgments to subjects’ judgments as follows. After making sure that all 7 subjects provided highly consistent similarity ratings (the average correlation between subjects’ similarity ratings are 0.8±0.097 and 0.86±0.065 for the crossmodal and multimodal conditions respectively), we average all 7 subjects’ similarity ratings to get a single similarity matrix for each condition. We measure the correlation between the average subject similarity matrices and the similarity matrices we get from our model. These matrices are highly correlated (crossmodal: r = 0.978, p < 0.001, multimodal: r = 0.980, p < 0.001), i.e., our model captures human’s performance in this task extremely well. 4 Conclusion We have presented a computational framework for understanding multimodal perception and showed that an instantiation of our model for object shape recognition captures human’s judgments on a shape similarity task very well. We believe this work constitutes a promising first step, and intend to evaluate our framework in more detail and more extensively in future work. 6 References [1] Goker Erdogan, Ilker Yildirim, and Robert A. Jacobs. From Sensory Signals to ModalityIndependent Conceptual Representations: A Probabilistic Language of Thought Approach. PLoS Comput Biol, 11(11):e1004610, November 2015. [2] Alan Yuille and Daniel Kersten. Vision as Bayesian inference: analysis by synthesis? Trends in cognitive sciences, 10(7):301–8, 2006. [3] Thomas G. Bever and David Poeppel. Analysis by Synthesis: A (Re-)Emerging Program of Research for Language and Vision. BIOLINGUISTICS, 4(2-3):174–200, 2010. [4] David Marr. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. MIT Press, Cambridge, Massachusetts, 1982. [5] Daniel M Wolpert and J Randall Flanagan. Forward Models. In T Bayne, A Cleermans, and P Wilken, editors, The Oxford Companion to Consciousness, pages 295–296. Oxford University Press, New York, 2009. [6] Marc O Ernst and Martin S Banks. Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415(6870):429–33, January 2002. [7] B Y Andrew T Miller and Peter K Allen. Graspit! a versatile simulator for robotic grasping. IEEE Robotics Automation Magazine, 11(December):110–122, 2004. [8] L Tierney. Markov Chains for Exploring Posterior Distributions. The Annals of Statistics, 22(4):1701–1728, 1994. [9] Noah D Goodman, Joshua B Tenenbaum, Jacob Feldman, and Thomas L Griffiths. A rational analysis of rule-based concept learning. Cognitive science, 32(1):108–54, January 2008. [10] Kaizhong Zhang and Dennis Shasha. Simple Fast Algorithms for the Editing Distance between Trees and Related Problems. SIAM Journal on Computing, 18(6):1245–1262, December 1989. 7

Download
# An Analysis-By-Synthesis Approach to Multisensory Object