Assigning Gesture Subtlety using an Accelerometer Himanshu Sahni and Shray Bansal Introduction We consider a gesture as being subtle if it's not seen as being out of the norm by other people. So it should not be disruptive to the flow of any social event. In the context of wearable computing, subtle hand gestures play a very important role. They could be used to easily perform tasks such as looking at past notes, checking to see if you've got a new message, or even look up the meaning of a word which you just heard somebody say. It is important that such gestures must also be socially acceptable and comfortable to use. Loud and obtrusive gestures have the obvious flaw of being socially inappropriate and uncomfortable for the user. This problem is addressed in the literature from an HCI point of view where it is found that small, quiet gestures are most acceptable and comfortable. Our problem is to address this from an Artificial Intelligence standpoint. Our goal is to design a system which would be able to assign a subtlety score to any given hand gesture based upon a training set of gestures voted on subtlety by humans. More quantitative comparisons can be made from the error values and the histogram in Figure 1. Score assignment is better predicted by Euclidean kNN on an average. Dynamic time warping seems to display good results (prediction error of 0.09) for ‘Swing Phone’ on which the Euclidean approach performed relatively worse (prediction error 0.17). We took the average of the scores obtained over all the query vectors as the predicted score for that gesture. One challenge in this approach was to handle feature vectors of different sizes. For this we took the smaller of the query or training feature vector and slid it over the other to find the best fit. Here we’re assuming that the gestures were performed in approximately the same length of time. One of our test examples, ‘Window Close’ performed poorly across the board. This might be due to two reasons. Firstly, this may point to a weakness in our methodology. It is worthy to notice that ‘Window Close’ gets consistently scored next to ‘Window Open’ by all the techniques. This might be due to the fact that these two gestures have very similar net acceleration profiles and thus our NN techniques will assign it as such. In fact, we observed that the maximum number of instances appearing were of ‘Window Open’ while our algorithm searched for neighbors for ‘Window Close’. Another plausible reason is due to lack of data. ‘Window Close’ was one of the gestures we had the least data on and thus most machine learning algorithms cannot be expected to perform well. Results User Study Gesture Name Euclidean Distance One NN kNN (k = 4) Dynamic Time Warping One NN k N N ( k = 4 ) Tap Phone (0.162) Tap Phone (0.162) Tap Phone (0.162) Tap Phone (0.162) Fire On (0.312) Fire On (0.312) Fire On (0.312) Fire On (0.312) Draw ‘W’ (0.598) Draw ‘W’ (0.598) WindowClose Draw ‘W’ (0.598) (0.382) Figure 1. Also compared are the scores assigned by the best implementations of the Euclidean Distance and Dynamic Time Warping approach with the user generated scores. The histogram below shows this comparison. Draw ‘W’ (0.598) Triangle (0.632) Triangle (0.632) Door Close (0.625) Analysis We contrast two approaches for assigning subtlety score. Broadly, they can be classified as being either time-dependent or time-independent depending on whether we used dynamic time warping to compute the similarity between gesture examples. Both approaches involve using one nearest neighbor and weighted k nearest neighbor techniques. In the first one, we evaluate the Euclidean distance between a query vector and the entire training set. In the one NN approach, we pick the closest feature vector in the training set and assign the query its subtlety score. In the k NN approach, we weigh the scores of the k neighbors by their distance to the query and assign the query a weighted sum of their scores. In each case, a query gesture will contain more than one query feature vector. Triangle (0.632) Fire Off (0.657) Fire Off (0.657) Triangle (0.632) 0.54 Fire off (0.657) Window Close (0.685) Window Close Fire Off (0.685) (0.657) 0.53 Door Close (0.678) Flick Air (0.688) Door Close (0.686) Flick Air (0.688) 0.52 Flick Air (0.688) Window Open (0.694) Flick Air (0.688) Window Open (0.694) Window Open Orient Phone (0.703) (0.694) Absolute Error Materials and methods Window Open Orient Phone (0.694) (0.703) 0.5 Swing Phone (0.721) Orient Phone (0.703) Window Close (0.749) Multi-Finger Snap (0.776) Door Close (0.746) Swing Phone (0.722) Multi-Finger Snap (0.776) 0.48 Throw Money (0.805) Multi-Finger Snap Multi-Finger (0.776) Snap (0.776) Swing Phone (0.787) 0.47 Swing Phone (0.885) Throw Money (0.805) Throw Money (0.805) Throw Money (0.805) Slap Phone (0.891) Slap Phone (0.891) Slap Phone (0.891) Slap Phone (0.891) Door Open (1.000) Door Open (1.000) Door Open (1.000) Door Open (1.000) Literature cited 0.51 Orient Phone (0.703) 0.49 0 2 4 6 k Values 8 10 12 Figure 2 Shows the effect of increasing the number of neighbors on the sum of absolute errors in the prediction scores. We can see a clear minima around k = 2 to 4. Conclusions Figure 1. The authors partake in a conversation where one of them is performing a gesture to control a wearable device. Table 1. The first column lists all the gestures in the data set, both training and test, in ascending order. Next to the gesture name is the reference to the paper it was mentioned in for a gesture recognition study. Below each gesture is the normalized score the gesture received from the user study. The next four columns contrast the Dynamic Time Warping and Euclidean distance approach for characterizing distance between feature vectors. Below each gesture name is the subtlety score predicted by our system. Given the limited training set, the algorithms perform reasonably well. From Table 1, it is clear that kNNoutperforms the other approaches. In terms of qualitative results, Euclidean kNN with k set equal to 4 ranks two of the three test gestures fairly accurately but fails to rank the ‘Window Close’ gesture properly. It has a total displacement of 5 in the rankings compared to the 11 in the Dynamic Time Warping implementation. . We need to improve our implementation and try using better algorithms. One obvious limitation is just using 1 NN and we can improve that by using more nearest neighbor points. We can also improve on our method of handling time series as time signals do not behave like normal features. Scaling in time (same gesture taking longer to be performed by different users) is a severe limitation to nearest neighbors. We can try to implement more complex algorithms such as SVMs to fit our data better. Kern N., Schiele B., Junker H., Lukowicz P., &Troster G. (2003). Wearable sensing to annotate meeting recordings. Journal: Personal and Ubiquitous Computing.Lyons K., Skeels C., Starner T., Snoeck C. M., Wong B. A., &Ashbrook D. (2004). Augmenting conversations using dual-purpose speech. Proceedings of the 17th annual ACM symposium on User interface software and technology - UIST '04. Julie R. & Stephen B. (2010). Usable gestures for mobile interfaces: evaluating social acceptability. Proceedings of the 28th international conference on Human factors in computing systems (CHI '10), 887-896. Montero C. S., Alexander J., Subramanian S., Marshall M. T (2010). Would You Do That? – Understanding Social Acceptance Of Gestural Interfaces. Proceedings of the 12th international conference on Human computer interaction with mobile devices and services 275278, 2010.Starner T., Auxier J., Ashbrook D., and Gandy M (2000). The Gesture Pendant: A Self-illuminating, Wearable, Infrared Computer Vision System for Home Automation Control and Medical Monitoring. In IEEE International Symposium on Wearable Computers, 87-94Bailly G., Müller J., Rohs M., Wigdor D., Kratz S. (2012). ShoeSense: A New Perspective on Hand Gestures and Wearable Applications. Proceedings of the 30th international conference on Human factors in computing systems (CHI '10). Deyle T., Palinko S., Poole, E.S. , Starner T.(2007). Hambone: A BioAcoustic Gesture Interface. Proceedings of the 11th IEEE International Symposium on Wearable Computers.http://funf.org/inaboxLementec J. &Bajcsy P (2004). Recognition of arm gestures using multiple orientation sensors: gesture classification. Proceedings of the 7th International IEEE Conference on Intelligent Transportation Systems, p 965–970.Ashbrook D., Starner T. (2010). MAGIC:a motion gesture design tool. Proceedings of the 28th international conference on Human factors in computing systems(CHI ’10).