2012 2nd International Conference on Information Communication and Management (ICICM 2012)
IPCSIT vol. 55 (2012) © (2012) IACSIT Press, Singapore
DOI: 10.7763/IPCSIT.2012.V55.11
Implementation of a Motion Recognition Module for Motion-based
User Interface of Smart phones
Tae-Eun Yoon, Kyung-Min Park, Eun-Ji Youand Hoon Choi+
Department of Computer Science and Engineering, Chungnam National University, Daejeon, Korea
Abstract. A mechanism of recognizing the movement of a mobile device and executing a specific event is
proposed in this paper as a part of user-defined and motion-based user interface. As a user grabs a mobile
device and makes a certain motion with his arm, the motion recognition module detects the value changes of
the sensors such as accelerometerand recognizes the motion. The played motion is then compared with
motion patterns which have been previously registered by user. Motion recognition module used the DTW
algorithm and experiment result using the accelerometer sensor showed thatthe module worked well to
distinguish the similarity of motion patterns.
Keywords: Dynamic Time Warping, motion, user interface, pattern recognition,s ensor.
1. Introduction
Motion UI(User Interface)is becoming more popular recently. Motion UI allows users of a mobile device
such as a smart phone to make a certain motion with the device in their hand, and then maps this motion to a
certain input or event occurrence in the device[1]. When the user grabs his device and makes a certain
motion with his arm, the values of sensors inside the device change. The device detects the change of the
sensor values and maps this motion to a specific[2][3], predefined inputor executes a specific event which
has been previously mapped with motion. An example of a motion UI is that a user can zoom in or out a
camera screen by pulling a camera toward or pushing it away from the body. Another example is executing a
specific application by shaking smart phone.
Motion UI mechanism is already offered by many mobile device manufacturers. However, users can
only use this mechanism prepared by the manufacturer. Hence, a new mechanism is suggested in this paper
in order to improve this limitationby providing a user a tool to add his own motion. The mechanism includes
a motion registration module, a motion recognition module and an event execution module. The motion
registration module allows users to register the patterns of motion in the mobile device for personal
preference along with the mapped events. The motion recognition module detects the changes in the sensor
value and recognizes the motion pattern. Finally, the event execution module executes the specific events
corresponding to the motion. To the authors’ knowledge, this mechanism has not been reported before.
This paper focuses on the motion recognition module and describes the detailed mechanism. Even
though a human moves his arm in the same pattern, the motion may vary with respect to the length oftime in
performing the motion. For instance, a motion may be played slowly or quickly. Therefore, motion
recognition module needs to recognize the same motion during different time span. An algorithm to compare
two data sets from two identical movement but different time-span patterns is required. Hence, this study
used the DTW(Dynamic Time Warping)algorithm for this purpose[4].
2. Background
+
Corresponding author. Tel.: +82-42-821-6652; fax: +82-42-822-4997.
E-mail address:hc@cnu.ac.kr.
62
2.1.
Senssors built in
n Android--powered Devices
D
Androidd-powered mobile
m
devices have manny internal seensors in general.Gingerrbread,versio
on 2.3 of thee
Android plaatform, has 11 sensors. Such sensoors measure motion, orieentation, andd various en
nvironmentall
conditions. Depending on the natuure of sensors, all senso
ors except thhe light, preessure, temp
perature, andd
proximity use
u a standardd 3-axis coorrdinate system[5].
Some of
o these sensoors are hardw
ware-based and
a some aree software-baased. Hardw
ware-based seensors derivee
their data by
b directly measuring
m
phhysical component. Softw
ware-based sensors derivve their data from virtuall
sensors thatt is a combinnation of onee or more of the hardware-based senssors and softtware-based sensors.
s
Thee
linear acceleeration sensoor and the grravity sensor are examplees of the softw
ware-based ssensors.
Applicaations and sensors are cloosely related.. For examplle,a proximitty sensor prevvents malfun
nction due too
a touch of the
t face to thhe capacitivee seen duringg a phone caall applicatioon. Also, a light sensor au
utomaticallyy
controls thee backlight off the device to
t provide a clearer screeen.
2.2.
And
droid Sensoor Subsysteem
Androidd sensor subbsystems prrovide data derived from
m the sensoors to Andrroid applicattion. Sensorr
subsystem operates
o
in thhe followingg steps. Whenn the Androiid system is booted, Senssor Service is
i initialized..
An applicattion that usess a specific sensor createss Sensor Maanager objectt and allocatees that sensor. After that,,
Sensor Servvice writes sensor
s
built in the devicces to queue and receivees the data ffrom queue in
i sequence..
Then Sensoor Manager reads
r
the datta in sequencce. Application that creatted the Sensoor Manager registers thee
listener to get
g the data and
a read the data
d using thhe on Sensor Changed meethod.
3. Recoggnition Module
M
Recognnition modulle recognizees the user's motion thrrough the seensors in thhe mobile device. Afterr
recognizingg the user's motion
m
to be
b one of the registered motions, thhe event exeecution modu
ule runs thee
internal eveents which arre mapped with
w the registtered motion
n. Some of thhe internal evvents are as follows:
fo
- executting applicattions / terminnating applications
- keypad and touchppad inputs, allready defineed by users
- modify
fying the connfiguration onn the device or applicatio
ons
For exeecutingeventss which matches with a motion, the event execuution modulee should be in
i the kernell
for dispatchhing events and providde that userss can definee events.A way
w to applly the patterrn matchingg
algorithm annd implemenntation resultt is proposedd in this papeer.Recognitioonmodule coompares the user-defined
u
d
pattern withh the playedd pattern for executingevvent. Patterns are distingguished throuugh the DTW
W algorithm
m
after pre-prrocessingacceelerometer sensor
s
data of
o mobile deevice.DTW algorithm
a
is used for diistinguishingg
similarity between two dynamic
d
pattterns.
3.1.
Pre--processingg of Sensor Data
Fig. 1 shows
s
the grraph drawn with
w raw sennsor data of the
t linearacccelerometersensor.Since the range off
values is wiide, the senssed data musst have pre-pprocessing off two steps so
s that DTW
W works with
h the highestt
accuracy annd lower proccessing time.
F 1: Data from
Fig.
fr
the lineaaracceleromeeter before pre-processin
p
ng.
First, thhe values of sensor
s
shouldd have any number
n
betweeen 0 and 1thhrough the m
min-max norm
malization.
Second, the range of
o data shouldd be determiinedsince thee unchanged range doesnnot need the DTW. If thee
continuous gradient bassed on vertexx at changingg values on either
e
side iss over the thrreshold, a ran
nge betweenn
i determinedd to apply thee DTW.
either side is
63
The Figg. 2 shows thhe graph afteer pre-processsing. The ho
orizontal axiss is number of generated
d data duringg
5 seconds.
Fig. 2: Datta from the liinearaccelero
ometer after pre-processiing.
3.2.
App
plying the DTW
D
DTW iss an algorithm
m for measurring similarity between two
t patterns which may vvary in time or speed.
The Figg. 3 shows thhe matrix forr calculating similarity beetween the tw
wo data afterr pre-processsing throughh
the DTW. N and M aree the lengthss of the two data. The matrix
m
has a size of NxM
M. Each matrix cells hass
value that iss the sum of Euclidean diistance between the two sensed data and
a a smalleest value of adjacent
a
cellss.
The value of
o final cell(N
NxM) is calleed total cost. If two senseed data are thhe same, the total cost is 0.
Fiig. 3: DTW matrix.
m
3.3.
Experiment
Samsunng's Galaxy Tab is used forthis expeeriment.The Android Giingerbread pprovides 11 sensors
s
withh
motion, envvironmental, position cateegory.
In this experiment,,android appplication takkes two mottion patternss for 5 secoondsand distinguishesthee
similarity of the two pattterns.Recognition module is applied to a x-axis value
v
of acceelerometer seensed data inn
ment.
the experim
3.4.
Resu
ults
3.4.1. Same Motions
M
Whencoomparing thhe same mottion with thhe same inteensity(Experiiment 1)andd the same motion
m
withh
different inttensity(Expeeriment 2), tootal cost abouut each of ex
xperiments is calculated.
Fig.4 annd Fig.5show
wpre-processsed sensed data
d of x-axiss ofaccelerom
meter when a user shakees the devicee
three times with similar intensity.
mes-a.
Fig. 5:
5 Shaking thhree times-b..
Fig. 4: Shakkingthree tim
milar wavefo
orms. Althouugh the two ddata waveforrms are veryy
Fig.6 reeveals that Fiig.4 and Fig..5 present sim
similar to each
e
other, the total cost is calculaated as 1.86
642168,whichh is greater than 0, beccause of thee
differences of amplitudee and frequenncy.
64
F 6:Matchhing between
Fig.
n two sensed data.
Fig.7 annd Fig.8 is a pre-processsed x-axis data of acceleerometer whhen the user drawscircless three timess
with differeent intensity.
Fig. 7:Draawing circless-a.
Figg. 8: Drawingg circles-b.
Fig.7 annd Fig.8 show
wthat the waaveforms aree similar to eaach other, buut the large ddifferences arre amplitudee
and frequenncy. In Fig.77,the data caannot be measured moree accurately.Therefore,inn comparison
n to the firstt
experiment,,Fig.9 does not
n similarly match with two sensed data.
d
The tottal cost of 2.884857 is an increase
i
of 1
from total cost
c of the firrst experimennt.
F 9:Matchhing between
Fig.
n two sensed data.
Despitee difference of the datta, when thhe same mo
otion with the same inntensities an
nd differentt
intensitiesarre compared, two patternns are recognnized as the same due to the
t lowest tottal cost.
3.4.2. Differeent Motionss
The saame experim
ment has been perfoormed for the two different
d
mootions with the samee
intensity(Exxperiment 3)).Fig.10 and Fig.11show the pre-proccessed x-axiss of sensed ddata of the acccelerometerr
whena user draws circleesand shakes the device thhree times reespectively. The
T total cosst of Fig.10 and
a Fig. 11iss
a 6.53193666.
calculated as
Fig. 10:Drawing the circcle-c.
Fig. 11:Shaking
1
thhree times-c.
3.4.3. Experim
mental Resuults
total cost
c
Exxperiment 1
Experiment 2
Experim
ment 3
1.8642168
2.8
84857
6.5319366
Table.1:Tootal cost each
h of experimeent.
The tottal costof exxperiment 3iis significantly larger th
han those offexperiment11 and 2.Con
nsidering thee
range of norrmalizeddataa(0 to 1), the total cost deerived from the
t original data
d will be m
much higher..
4. Concllusion
This papper proposedd a motion reecognition module
m
using the DTW alggorithm as a part of user--defined andd
motion-baseed user interrface mechannism for moobile devices. The similaarity of patteerns wasdistinguished byy
thetotal cosst that is an indicator
i
of the similarityy of two pattterns.The reesult of the eexperimentprroves thatthee
pattern recoognition moddule with DT
TW algorithm
m works weell in distinguuishing the ssimilarity off two motionn
patterns.
The tim
me complexitty of DTW algorithm ussed in the motion
m
recognnition module is O(mn)[[6].The timee
overheadneeds to be im
mproved by adopting ann effective method
m
for the mobile environmen
nt which hass
relatively innferior compputational caapability[7].IIn order to improve
i
the time compllexity, matriix D will bee
added to thee global pathh constraintstthat reduces the
t seek timee with the matrix[8].
65
5. Acknowledgement
This research was supported by the MKE(The Ministry of Knowledge Economy), Korea, under the
ITRC(Information Technology Research Center) support program supervised by the NIPA(National IT
Industry Promotion Agency) (NIPA-2012-H0301-12-3005)
6. References
[1] S.-J. Cho, E. Choi, W.-C. Bang, J. Yang, J. Cho, E. Ki, J. Sohn, D. Y. Kim and S. Kim. Motion-Understanding
Cell Phones for Intelligent User Interaction and Entertainment. HCI. 2006, pp. 684-691.
[2] B. Coley, B. M. Jolles, A. Farron, and K. Aminian. Arm position during daily activity. Gait & Posture.2008, vol.
28, no. 4, pp. 581-587.
[3] J. Liu, L. Zhong, J. Wickramasuriya, V. Vasudevan.uWave: Accelerometer-based personalized gesture recognition
and its applications. Pervasive and Mobile Computing. 2009, vol. 5, no. 6, pp. 657-675.
[4] P. Capitani and P. Ciaccia. Warping the time on data streams. Data & Knowledge Engineering. 2007, vol. 62, no.
3, pp. 438-458.
[5] Android sensors overview, http://developer.android.com/guide/topics/sensors.
[6] M. Mueller. Information Retrieval for Music and Motion. Springer. 2007, pp. 69-84.
[7] B. W. Choe, J.-H. Hong, S.-B. Cho. Gesture interface with 3D accelerometer for mobile users. HCI. 2009, pp.
378-383.
[8] L. Rabiner, B. H. Juang. Fundamentals of Speech Recognition. Prentice Hall. 1993.
Tae-Eun Yoon was born on January 27, 1990, in Cheongju, Korea. She received the B.S.
degree in department of computer science and engineering, Chungnam National
University, in 2012. She is currently M.S. in department of computer science and
engineering, Chungnam National University. Her interests include mobile computing and
sensor network.
Kyung-Min Park was born on October 24, 1984, in Daejeon, Korea. He received the B.S.
degree in department of computer science and engineering, Chungnam National University,
in 2010. He is currently M.S. in department of computer science and engineering,
Chungnam National University. His research area is the mobile computing, wireless
network and embedded software.
Eun-Ji You was born on March 10, 1988, in Daejeon, Korea. She received the B.S. degree
in department of computer science and engineering from Chungnam National University,
Daejeon, Korea in 2011. She currently works as research staff at Mobile Distributed
Computing Laboratory in Chungnam National University. Her research interests include
Ubiquitous Sensor Network and Mobile computing.
Hoon Choi received the B.S.E.E degree in computing engineering from the Seoul
National University, Korea in 1983 and the M.S. and Ph.D. degrees in computer science
from Duke University in 1990 and 1993, respectively. Before joining the Chungnam
National University in 1996, he was a senior member of technical staff of Electronics and
Telecommunication Research Institute, Korea since 1983, where he worked on LAN,
broadband ISDN and high-speed network system, His research area is the mobile,
distributed computing. He has been involved in several research projects on middleware
for distributed computing, mobile computing and a software platform of the cellular phone
for wireless applications. He is currently interested in mobile computing and the system software for
wearable computers.
66