PACE UNIVERSITY; CAPSTONE PROJECT - BIOMETRIC AUTHENTICATION & ACCELEROMETER SENSOR 1 Biometric User Authentication on Smartphone Accelerometer Sensor Data Noufal Kunnathu, Seidenberg School of CSIS, Pace University Abstract — This study attempts to build a statistical model to identify a user, based on how the user picks up the phone and how he/she holds the phone to the ear. Phone-pickup and phoneholding methods are quantified using accelerometer readings on the three axes. This study is based on the theory that every person has a unique way of handling the phone during a phone call. The minute variations in the angle of tilt and the micro movements made during the phone call can be quite different between two individuals. The Accelerometer gives a convenient method of quantifying the phone movements and the change in orientation of the phone on a three dimensional space. This study involves data gathered from seven participants who made five test phone calls each lasting for a little over 5 seconds. The study produced a model that can predict the user with high accuracy using a smartphone accelerometer. Index Terms— Accelerometer, Biometrics, Authentication, Weka. I. INTRODUCTION A is a device that can detect the static acceleration caused by the pull of gravity and the dynamic acceleration caused by displacement or vibration of the device [7]. The static acceleration gives useful insights into the orientation of the device at different phases of the experiment. The dynamic acceleration quantifies customer movements while picking up the phone and gives insights into the micro movements during the call. Most smartphones available in the market today, have three-axis accelerometer included. Biometric authentication methods often provide secure alternatives to traditional authentication methods. In fact several biometric authentication methods can be used in tandem to build a secure and an easy-to-use system that provide continuous authentication capabilities. This paper describes a method of accelerometer-based authentication that can be combined along with other means of authentication to improve the accuracy of biometric-based authentication systems. CCLEROMETER A. SMARTPHONE ACCELEROMETER Fig-1 shows the working principle of a smartphone accelerometer. Seismic mass is fixed on the housing of the accelerometer unit that is attached to the phone circuit. The gravitational pull or user movements will change the orientation This study is conducted as part of Capstone Program for the completion of Masters Degree in Information Systems at Pace University during the Fall Semester of 2014. of the seismic mass with in the sensor, generating changes in the capacitance [27]. Fig. 1. Smartphone Accelerometer – picture credit Techulator.com[27] B. WEKA Weka [12] is a product of University of Waikato. It is among the leading open-source tools used for building Data Mining Systems. It is released under GNU General Public License (GPL). Weka requires data in a specific format called AttributeRelation File Format (ARFF). It provides a convenient utility for converting Comma-Separated Values (CSV) format to ARFF format. Weka allows preprocessing of the data before applying the classifier algorithms to build the model. Weka also supports a number of attribute evaluator algorithms to be used for selecting or ranking attributes in the dataset. The evaluator can be used for filtering noisy attributes or ranking the attributes based on their relevance. Weka supports a large collection of classifier algorithms to be used for building statistical models in different contexts. In this study Weka classifiers are used for achieving two things; for finding a threshold value for splitting the data sessions into three phases – picking-up, on-ear and placing-down. More importantly, Weka is used for building the statistical models for the four experiments conducted in this study. After evaluating many of the available classifier algorithms, the highest performance is observed for a particular classifier algorithm called Multilayer Perceptron. This is a classifier PACE UNIVERSITY; CAPSTONE PROJECT - BIOMETRIC AUTHENTICATION & ACCELEROMETER SENSOR based or Artificial Neural Network (ANN) model. This algorithm is discussed further in the next section. C. Multilayer Perceptron Algorithm Multilayer Perceptron is a feed-forward artificial neural network model. It maps a set of input data to a set onto a set of appropriate outputs [8]. Fig. 2. Multilayer Perceptron Model. As shown in Fig. 2, MLP model contains three layers with each layer fully connected to the others – the input layer, the hidden layer and the output layer. II. LITERATURE REVIEW Sensor-based, biometric analysis is a relatively recent field of research. There is an increased interest in this area due to the proliferation of mobile devices and the wide availability of sensors. There were many early studies that used accelerometer-based biometric analysis for gait recognition [2][10][14][22]. The studies produced promising results in identifying user gait based on accelerometer devices worn by the users. There is an interesting study that proposed authentication via electronic pets [4][26] that imprints the user characteristics and continuously nurture the characteristics to be used as an authoritative token for securely identifying a user. A recent study focuses on predicting user traits such as height, weight, sex etc. from the data collected with the accelerometer sensor with the volunteers taking short walks carrying an Android phone [29]. There have been interesting studies proposing gesture-based user authentication using accelerometer sensor [20][11]. There have been researches in mining complex activities and performing activity recognition based on the data collected from the Accelerometer [23][21][19]. There are studies for building Accelerometer-based gestural text-entry systems [29]. Accelerometer and gyroscope is combined in many experiments. The combined data can be used for motion gesture recognition using classifiers with dimensionality constraints [18]. There is a recent study investigating mobile device picking-up motion [9]. Also, there is a study conducted to transparently identify the user of a smartphone when he/she is on a phone call [5]. These last two references are very relevant to the scope of this study. Those studies and this paper look at the same problem but in different perspectives. 2 III. EXPERIMENT DESCRIPTION The aim of this study is to identify a user from the accelerometer readings. For the purpose of this study, data was collected from the accelerometer from volunteers who agreed to participate in the study. There are four experiments conducted in this study. All four experiments are based on different aspects of the same dataset. The first two experiments investigate the phone holding-to-ear pattern. The third experiment investigates the phone picking up activity and the last experiment combines all the features to derive a highperforming continuous authentication system. A. Data Collection The experiments required data corresponding to the phone picking up action and the data for phone holding-to-ear activity. Seven volunteers were selected for the data collection. The participants included five males and one female with ages ranging from 25 to 40, and a thirteen-year-old male. Each participant is requested to repeat the following steps five times, resulting in a total of 35 data samples. Pick up the phone from a table Hold it up to the ear for approximately 5 seconds Place the phone back on the table The Accelerometer data is recorded continuously while the above steps are executed. Users are asked to hold the phone naturally as if making a phone call. It is observed that 5 of the 7 users held the phone to their left ear, one to the right ear, and one to both ears - sometimes the left ear and sometimes the right ear. The selection of the ear resulted in variation of the accelerometer readings but it did not appear to be a significant factor in identifying the user. The accelerometer data gives a unique perspective on the way phone moves on all three axes while the user makes the phone call. Each experiment session lasted a little over 5 seconds. Sampling rate of 25Hz is used for the data collection. A data vector contains following attributes. Time Session-id Ear-used X-acceleration Y-acceleration Z-acceleration and User-id The Session-id, Ear-used and User-id are manually recorded and added to the dataset. Other attributes are directly extracted from the smartphone accelerometer sensor. B. System Design Android operating system was chosen for conducting the study and a Nexus 5 device was used for all data collection. An android application called “Sensor Kinetics Pro” [24] is used for recording the accelerometer data. This app can record readings from many sensors including accelerometer, gyroscope, magnetometer, etc. However, the scope of this study is limited to the accelerometer. So, the readings from other sensors were not processed. PACE UNIVERSITY; CAPSTONE PROJECT - BIOMETRIC AUTHENTICATION & ACCELEROMETER SENSOR Fig. 1 is a screenshot from the app that shows the readings for Accelerometer on X, Y and Z axes. 3 In this study, a semi-automatic procedure is used for tagging the data into stages. The first part is to build a model to correctly tag the three stages. For this purpose, a movement rate metric is created based on the rate of movement on three axes. The training data set is built manually for a few representative sessions. The training data and the computed value for metric were loaded into Weka. J48 decision tree is used to build a simple model to derive a threshold value for the movement metric for the stages. The classifier returned a threshold value for the movement as 0.776. Any vectors that have a lower than the threshold value is tagged as “on-ear” stage. The rest of the data is split into either “picking up” or “placing down” based on the time of recording. After the automated tagging is applied, the data is manually inspected to make sure that the three phases are continuous. For example, any vector tagged as “picking up” in the middle of an “on-ear” phase is manually overridden to “on-ear”. Fig. 1. Screenshot from Sensor Kinetic Pro App. Weka is used to analyze the sensor data. A number of other tools were evaluated as part of the study. Weka was selected for its intuitiveness, ease of use, and the rich feature set. C. Data Pre-processing As stated earlier, there are three phases for each experiment session – picking phone up, holding up to the ear and placing the phone down. Initial pre-processing task is to tag the data to separate the data vectors for these three stages. This can be done by plotting each session in Excel and visually inspecting the graph. For example, see fig-2 for the line chart of a typical session. 15 10 5 -5 time 0.313 0.63 0.948 1.265 1.582 1.9 2.216 2.533 2.851 3.316 3.862 4.406 4.953 5.498 5.993 6.538 7.083 7.628 0 -10 -15 X_value Y_value D. Experiments Four experiments were conducted to build statistical models for identifying users based on the Accelerometer data associated with the phone call sessions. 1) EXPERIMENT-1: PHONE ON-EAR. This experiment uses the average values for the X, Y and Z readings from the accelerometer sensor. The aim is to build a statistical model that can correctly predict the user from the average values. The model will be built with the help of a classifier algorithm available in Weka. For this experiment, only the data vectors tagged as “on-ear” is used. The average values are computed for X, Y and Z readings for each session. The averages were computed by building a pivot table in Microsoft Excel. We are left with 35 feature vectors with following attributes. Session-id Ear-used Average-X-acceleration Average-Y- acceleration Average-Z-acceleration and User-id In order to select the optimum attributes, an Attribute evaluator called Chi-squared Attribute Evaluator [25] was used. The evaluator ranked the attributes as shown in Table-1 Z_value Fig. 2. Line Diagram for a typical experiment session. Large variances can be observed at the beginning and end of each session with relatively stable values for the middle part of the experiment. The beginning part with large variances corresponds to the “picking up” phase, the middle part corresponds to the “on-ear” phase and the end part with variances corresponds to the “placing down” phase. There can also be 0 values very early and end of each session. These zero values will be trimmed out as these correspond to the stationary phone on the table. Table 1. Attribute Ranking It appears that averages on Z and X axes have higher significance over other attributes in predicting the user id. The attributes that have low significance can be eliminated one-byone as long as the removal doesn’t reduce the performance of the system. PACE UNIVERSITY; CAPSTONE PROJECT - BIOMETRIC AUTHENTICATION & ACCELEROMETER SENSOR Figure-3 shows the clusters of instances when the values for the two most significant attributes are plotted on X and Y-axes. 4 User-5 was confused as User-7. This can indicate some similarity in the data for these two users. Table 3. Confusion matrix for Experiment-1 Fig. 3. Data visualization – Average Z value on horizontal and X value on vertical axes Each point in the graph represents a feature subset containing average values for Z and X accelerometer readings. Different colors are used for the feature vectors corresponding to distinct user. The rectangles show the clustering. As observed from the diagram, there is minimal overlap between the rectangles. Only one user has the data spread-out across the graph. This user is the one who used both ears for the experiments. As discussed earlier, in this study Weka is used with the Multilayer perceptron algorithm as the primary classifier for building the model. The entire dataset was used as both training set and the test set with a 35-fold cross-validation. Since there were 35 sessions altogether, the 35-fold cross-validation will test each session individually with the remaining 34 sessions used for training. Table-2 shows the results from the classifier. Out of the 35 sessions, 28 were correctly classified with an 80% success rate. Each row corresponds to a user. All the sessions from a user indicated by the row label is placed on that row based on how the session was classified with the column label indicating the classification. The classifier was executed multiple times to reduce the feature set to the lowest number of attributes that produce the best results. Eliminating Session-Id and subsequently the EarUsed attributes, improved the system performance. Removing any other attributes deteriorated the performance. The remaining attributes constitute the feature set for this Experiment. Following are the attributes that are identified as part of the feature set for this Experiment. Table 4. Feature-set for Experiment-1 with the chi-squared ranks 2) EXPERIMENT-2: PHONE ON-EAR (VARIANCES ADDED) This experiment is very similar to Experiment-1. Addition of variances into the list of attributes is the only change from Experiment-1. Similar to the earlier experiment, attributes are evaluated using Chi-squared Attribute Evaluator. Table-5 shows the attributes with chi-squared ranks. From the rankings, it appears that the variances are not as significant as the averages in identifying the user. The Z and X averages are still the most significant attributes. Table 2. Classifier Results Summary – Experiment-1 Kappa statistic [28] with a value 0.7667 shows a moderate agreement with in the experiment data. Root-mean-squarederror, root-absolute-error and root-relative-squared-error provide metrics for comparing the rate of error in making the correct prediction. These metrics can be used for comparison between the experiments described in this paper in terms of probability of errors with in the models. Table-3 shows the confusion matrix that helps to visualize how the measurement from each user was classified. The diagonal shows the matches and the numbers marked outside of the diagonal are the “confusion”. For example, two sessions from User-7 was confused as User-5 and one session from Table 5. Attribute Ranking The experiment is performed multiple times by eliminating lowest significant attribute each time. It was observed that the following set of attributes provided the best performance. PACE UNIVERSITY; CAPSTONE PROJECT - BIOMETRIC AUTHENTICATION & ACCELEROMETER SENSOR 5 Where - gravity on X, Y or Z-axis for ith measurement. - threshold which can be found by the following formula. For this experiment a constant threshold of 0.8 is assumed. Table 6. Feature-set for Experiment-2 with the chi-squared ranks In terms of number of instances classified correctly, both Experiment-1 and Experiment-2 performed equally, by correctly identifying 28 of the 35 instances. However, Experiment-2 had lower error values compared to Experiment1 signifying an improvement in the model performance. With the 35-fold cross-validation, the model produced 80% success rate. Table-6 shows the result summary from Weka. is the time-constant. is the sampling rate. - ith reading from the accelerometer. As done earlier, the attributes are initially evaluated using chi-squared attribute evaluator resulting in following rankings. Table 7. Classifier Results Summary for Experiment-2 Table 9. Attribute Ranking As observed on Table-9, the gravity acceleration on Z-axis and session id have the lowest ranks in identifying the user instance. As discussed earlier, the feature set need to be reduced by eliminating low ranking attributes with out reducing the system performance. By following this reduction, following attributes are determined as the feature set for this Experiment. Table 8. Confusion matrix for Experiment-2 Confusion matrix has a slight difference from Experiment-1. An instance belonging to User-5 was earlier classified as User1; now it is classified as User-7. Either way, both models got that particular instance wrongly classified. 3) EXPERIMENT 3: PHONE PICKING-UP Phone picking-up phase is relatively more dynamic. There are high variances in accelerometer readings during this phase. There are two components that contribute to the acceleration – gravity and the user movement. The first task in this experiment is to separate the two components. Even though it is impossible to completely separate the two components, a reasonable approximation can be made using a Low-pass Filter [17]. Following formula is used for finding the gravity component with the low-pass filter [1]. Table 10. Feature-set for Experiment-2 with the chi-squared ranks Only "Session Id” was removed for this experiment. All other attributes including Z_gvty contribute to the system performance. Table-10 shows the summary of results for experiment-3. PACE UNIVERSITY; CAPSTONE PROJECT - BIOMETRIC AUTHENTICATION & ACCELEROMETER SENSOR 6 Table 11. Classifier Results Summary The experiment identified 29 of the 35 users correctly, with a success rate of 82.86%. It can be deduced that phone pickingup method is slightly better in identifying the user compared to the phone holding position. Table-12 shows the details of how each session was classified. Table 13. Attribute Ranking Attribute reduction logic is performed again and found that the following subset provided the best performance. Table 12. Confusion matrix for Experiment-3 It can be noted from the confusion matrix that this experiment identified some users (for example, User-7) more reliably compared to the previous experiments and vice versa. 4) EXPERIMENT 4: COMBINATION OF PHONE ON-EAR AND PICKING-UP PHASES In this experiment, the feature sets extracted from experiment-2 and experiment-3 are combined. Following shows a quick visualization of all the available attributes. Each color indicates a distinct user. For example, the Ear-Used attribute is same for all users except for User-3 and for a portion of samples for User-4. Table 14. Feature-set for Experiment-4 with the chi-squared ranks Table-14 shows a summary of results from the classifier. Table 15. Classifier Results Summary Fig. 3. Data visualization of how individual attributes stack-up for each user represented by a distinct color. The combined feature set is evaluated using the chi-squared attribute evaluator. The results are provided in Table-13 The combined experiment has the highest performance compared all previous experiments. This experiment identified 32 of 35 samples correctly with a success rate of 91.43%. Table-16 is shows the classification. The darker diagonal indicates the improvement in performance. 4 of the 7 users are PACE UNIVERSITY; CAPSTONE PROJECT - BIOMETRIC AUTHENTICATION & ACCELEROMETER SENSOR identified with 100% accuracy. The rest of the users were identified with 80% accuracy. Table 16. Confusion matrix for Experiment-4 [4] P. Briggs and P. L. Olivier. Biometric daemons: authentication via electronic pets. In CHI '08 Extended Abstracts on Human Factors in Computing Systems (CHI EA '08). ACM. 2008 [5] M Conti, I. Zachia-Zlatea, and B. Crispo. “Mind how you answer me!: transparently authenticating the user of a smartphone when answering or placing a call,” In Proceedings of the 6th ACM Symposium on Information, Computer and Communications Security (ASIACCS '11). ACM, 2011. [6] J. Demsar et al. “Orange: Data Mining Toolbox in Python,” University of Ljubljana, 2013. [7] Dimension Engineering, “A beginner’s guide to accelerometers,” http://www.dimensionengineering.com/info/accelerometers, Accessed, November, 2014. [8] DTREG, “Multilayer Perceptron Neural Networks,” http://www.dtreg.com/mlfn.htm, Accessed on November 2014. [9] T. Feng, X. Zhao and W. Shi. “Investigating Mobile Device Picking-up Motion as a Novel Biometric Modality,” Computer Science Department, University of Houston. 2013. IV. CONCLUSIONS AND FUTURE WORK The result of this study is promising. The final model built by combining feature sets has a high performance. It shows 91.4% success rate in correctly predicting the user. It is exciting to see that this soft biometric analysis can be used in identifying a user with such accuracy. This experiment could be improved in a number of ways. The sample size selected is relatively small. The experiment could be repeated with a larger sample size to confirm the findings. The data collection method can be improved. Instead of conducting all 5 sessions in a single stretch, the data collection could be done in phases with different sessions from each user collected on different days. The direction were the user was facing was neither random nor supervised. Hence some of the users were choosing a random direction and recording all samples facing that same direction. There were four users who turned towards approximately the same direction. Also, the participants were standing still during the phone call. It would be more realistic to have the users walk or make other movements while making the call. Making such changes can potentially bring down the performance of the model; but such changes will result in a robust model that can be used in realworld applications. This study focused purely on Accelerometer sensor. Combining this sensor data with other available sensors like Gyroscope, Magnetometer, etc. can improve the performance of the model further. This will be another area to explore in the future. This model can be used in combination with other biometric systems like voice recognition, face recognitions, etc. Such a system that combines multiple biometric techniques for continuous authentication could become a foundation for high security systems in future. REFERENCES [1] Android Developers, http://developer.android.com/reference/android/ hardware/SensorEvent.html, accessed December 2014. [2] A. Annadhorai, E. Guenterberg, J. Barnes, K. Haraga, and R. Jafari. “Human identification by gait analysis,” In Proceedings of the 2nd International Workshop on Systems and Networking Support for Health Care and Assisted Living Environments (HealthNet '08). ACM, Article 11, 3 pages. 2008. [3] S. Block and A Popescu, “DeviceOrientation Event Specification,” W3C Working Draft 1 December 2011 http://www.w3.org/TR/orientationevent/ 2 7 [10] D. Gafurov, K. Helkala, and T. Søndrol. “Biometric Gait Authentication Using Accelerometer Sensor,” Journal of computers, vol.1, October/November, 2006. [11] D. Guse. “Gesture-based User Authentication on Mobile Devices using Accelerometer and Gyroscope”, M.S. Thesis, 08 Aug 2011, Berlin Institute of Technology. [12] M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. Witten. “The WEKA Data Mining Software: An Update,” SIGKDD Explorations, Volume 11, Issue 1, June 2009 [13] T. Hastie and T. Tibshirani, “K-Fold Cross-Validation,” February 25, 2009. [14] C. C. Ho, C. Eswaran, Kok-Why Ng, and June-Yee Leow. “An unobtrusive Android person verification using accelerometer based gait,” In Proceedings of the 10th International Conference on Advances in Mobile Computing & Multimedia (MoMM '12), Ismail Khalil (Ed.). ACM, 2012. [15] IBTIMES, http://www.ibtimes.com/how-does-accelerometer-worksmartphone-bill-hammack-engineer-guy-explains-full-text-699762, Accessed on November, 2014. 19 [16] E. Jones, J. Alexander, A. Andreou, P. Irani, and S. Subramanian. “GesText: accelerometer-based gestural text-entry systems,” In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, 2010. [17] K Kircher, “Android Accelerometer: Low-Pass Filter Estimated Linear Acceleration”, http://www.kircherelectronics.com/blog/index.php/11android/sensors/10-low-pass-filter-linear-acceleration, accessed December 2014 [18] S. Kratz, M. Rohs, and G. Essl. “Combining acceleration and gyroscope data for motion gesture recognition using classifiers with dimensionality constraints,” In Proceedings of the 2013 international conference on Intelligent user interfaces (IUI '13). ACM, 2013. [19] J. R. Kwapisz, G. M. Weiss, and S. A. Moore. “Activity recognition using cell phone accelerometers,” SIGKDD Explorer Newsletter 12, 2, 74-82. March 2011. [20] J. Liu, L. Zhong, J. Wickramasuriya, and V. Vasudevan “User evaluation of lightweight user authentication with a single tri-axis accelerometer,” In Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '09). ACM, 2009. PACE UNIVERSITY; CAPSTONE PROJECT - BIOMETRIC AUTHENTICATION & ACCELEROMETER SENSOR [21] T. Maekawa and Shinji Watanabe.. Training data selection with user's physical characteristics data for acceleration-based activity modeling. Personal Ubiquitous Comput. 17, 3, 451-463. March 2013. [22] J. Mäntyjärvi, M. Lindholm, E. Vildjiounaite, S. Mäkelä, H. Ailisto. “identifying users of portable devices from gait pattern with accelerometers,” IEEE International Conference on Acoustics, Speech, and Signal Processing, March 2005. 8 [26] M. Tamviruzzaman, S. I. Ahamed, C. S. Hasan, and C. O'brien. “ePet: when cellular phone learns to recognize its owner,” In Proceedings of the 2nd ACM workshop on Assurable and usable security configuration (SafeConfig '09). ACM, 2009. [27] Techulator, http://www.techulator.com/resources/8930-How-doessmart-phone-accelerometer-work.aspx, accessed December 2014 [23] A. Rai, Z. Yan, D. Chakraborty, T. K. Wijaya, and K. Aberer. “Mining complex activities in the wild via a single smartphone accelerometer,” In Proceedings of the Sixth International Workshop on Knowledge Discovery from Sensor Data (SensorKDD '12). ACM, 2012. [28] A. Viera and J. Garret, “Understanding Interobserver Agreement: The Kappa Statistic”, http://virtualhost.cs.columbia.edu/~julia/courses/ CS6998/Interrater_agreement.Kappa_statistic.pdf, accessed, December 2014 [24] Rotoview, “Sensor Kinetics Pro App”, http://www.rotoview.com/ sensor_kinetics_pro.htm, Accessed on November, 2014 3 [29] G. M. Weiss and J. W. Lockhart. 2011. “Identifying user traits by mining smart phone accelerometer data,” In Proceedings of the Fifth International Workshop on Knowledge Discovery from Sensor Data (SensorKDD '11). ACM, 2011. [25] Star Trek, http://stattrek.com/chi-square-test/independence.aspx, accessed December 2014