Smart Poniting & Tracking Algorithms
Using Accelerometer & Gyroscope
Chapter 1
System Summary
System Summary
X,Y
Or
Smart Phone
(Client)
Button Click
1. Android Application for MEMS sensing
2. MEMS-sensor based pointing & compensation
algorithm
☞ Pointing algorithms
1. Quaternion
2. Rodrigues
☞ Compensation algorithms
1. Kalman filter
2. Tilt compensation
PC (Server)
1. Implementation of Window application for controlling
mouse cursor
2. Implementation of Bluetooth socket communication
algorithm
☞ Mouse cursor control using WinAPI function
☞ Bluetooth SPP(Serial Port Profile)-based socket
program
Chapter 2
Pointing algorithm
(Quaternion, Rodrigues)
Quaternion - multiply
Quaternion – axis transform
,
,

Pointing Algorithm

Pointing Algorithm

HID(Human Interface Device)
Control channel
Interrupt channel
HID – Search & Analysis
 HID PROFILE with Bluetooth wireless communications
 Android open source & Android platform data
 http://www.hackchina.com/en/cont/16692
 HID-Mouse C++ source code
 AndroMouse Desktop 2.4
 Bluetooth / Wifi HID Application
 Keyboard, Mouse only touch use
 http://www.youtube.com/watch?v=X0ixrshL708&feature=player_embedded
Pointing algorithm – Quaternion (1/2)
Quaternion rotation – Axis conversion
If above p and q are multiplied
- Quaternion multiply formula
Pointing algorithm – Quaternion (2/2)
User-frame : Axis that a user look at a target
Body-frame : Basic axis of a device
Assume rotation  occurs around z-axis. Then
the followings are driven.
Z-axis is toward the direction to the display of
a device, the above equation is based on z-axis.
If the equation is rewritten,
Once we apply Quaternion the axis-conversion
equation to the above equation, we can obtain
Rotation matrix from axis and angle
y
p’(a’,b’)
p(a,b)
x
Trigonometrical function:
Addition theorem
Rotation matrix from axis and angle
Rodrigues’ rotation formula
Rodrigues’ rotation formula
Rodrigues’ rotation formula
y
w
x
Rodrigues’ rotation formula
Rodrigues’ rotation formula
Pointing Algorithm 02
Pointing algorithm – Rodrigues (1/3)
1번 식의 유도 과정을 살펴보면 아래와 같다.
Pointing algoritms – Rodrigues (2/3)
Once we solve the above equation, we can obtain
Pointing algorithm – Rodrigues (3/3)
Since the Rodrigues rotation-conversion matrix is
for x, y, z axis, removing the part of z-axis in the
matrix can be simplified to the following 2x2 matrix.
Using line acceleration values taken from an
accelerometer, the following x, y axis vector
can be obtained.
Using values for x, y axis vectors, the following
 ' can be obtained.
The variations of x and y corresponding to
 ' in display can be driven as a following
equation using the Euler’s equation.
Comparison in processing sped(Quaternion / Rodrigues)
- Comparison in required computing time
The number of loop
1 - 100
Quaternion
(Adopted mapping process)
382,125 ns
- x, y variation after mapping
x(rodrigues)
Rodrigues rotation
-5
-0.7
x 10
x(quaternion)
y(rodrigues)
y(quaternion)
411,118 ns
101 - 200
300,005 ns
442,801 ns
201 - 300
258,423 ns
423,145 ns
301 - 400
196,343 ns
406,271 ns
401 - 500
225,620 ns
435,145 ns
501 - 600
227,691 ns
680,828 ns
601 - 700
279,900 ns
919,970 ns
701 - 800
480,468 ns
838,399 ns
801 - 900
533,706 ns
919,483 ns
901 - 1000
374,154 ns
646,239 ns
Average
325,844 ns
612,340 ns
⇒ Quaternion is twice faster than Rodrigues
rotation in average.
-0.8
-0.9
-1
-1.1
-1.2
-1.3
0
10
20
30
40
50
60
70
80
90
100
⇒ Quaternion provides more stable in x, y
variation in case of no movement of a smart
phone.
Chapter 3
Compensation algorithm
(Kalman filter, Tilt compensation)
Kalman Filter
Kalman filter predicts variations in states and estimates current state for measured current noise.
It can also estimate the state values similar to real states by updating covariance.
Effects of Kalman filter
1. Minimize noise in data from accelerometer
2. Compensation in noise caused by hand shaking
or vibration
Kalman Filter
1. Minimize noise in sensor data
- Smart phone ⇒ No movement
- Smart phone ⇒ moved
0.2
6
accel valuex
accel valuex (befor kalman)
accel valuey
accel valuey (befor kalman)
accel valuez
accel valuez (befor kalman)
0.1
4
0
2
-0.1
Before
application
0
-0.2
-0.3
-2
-0.4
-4
-0.5
-0.6
0
100
200
300
400
500
600
700
800
900
1000
0.05
-6
0
100
200
300
400
500
600
700
800
900
1000
2
accel valuex (after kalman)
accel valuey (after kalman)
accel valuez (after kalman)
1.5
0
1
accel valuex (kalman)
-0.05
accel valuey (kalman)
0.5
accel valuez (kalman)
-0.1
0
-0.15
-0.5
After
application
-1
-0.2
-1.5
-0.25
-2
-0.3
-2.5
-0.35
0
100
200
300
400
500
600
700
800
900
1000
-3
0
100
200
300
400
500
600
700
800
900
1000
Kalman Filter
2. Noise compensation caused by hand shaking or vibration
☞ generated when you touch the touch pad of a phone
- Before and after in x axis
- Before and after in y axis
2
x variation - before kalman(1:1000,1)
x variation - after kalman(1:1000,2)
4
y variation - before kalman(1:1000,4)
y variation - after kalman(1:1000,5)
1.5
3
1
2
0.5
0
1
-0.5
0
-1
-1
-1.5
-2
0
100
200
300
400
500
600
700
800
900
1000
-2
0
100
200
300
400
500
600
700
800
900
1000
Tilt compensation
No compensation information in Quaternion mapping
- Key point:
Calculating unit vector based on periodic characteristic
according to tilt of a device
1
0.8
0.6
0.4
0.2
0
-0.2
-0.4
9.8
Accel_Value[0]
0
- 9.8
Accel_Value[1]
0
9.8
Accel_Value[2]
90’
45’
0’
-45’
90’
0
-0.6
-0.8
-1
-8
-6
-4
-2
0
2
4
6
8
Chapter 4
Wireless Mouse for remote TV control
Using Smart Phone
Application development and implementation
X,Y
Or
Button Click
< Tray Icon >
Bluetooth Based Algorithm Development Enviroment
• Overall structure
Sender
Receiver
Screen
Accel, Gyro
Sensor
User’s
Input
OS
Windows 7
플랫폼
Android
통신방법
Bluetooth 3.0
개발도구
Android SDK
Tool
Eclipse
Pointing Algorithm
Calibration Algorithm
센서
Bluetooth
안드로이드 버전
Accelerometer,
Gyro
2.3.3
 Test Environment : Android OS Base smart devices with Bluetooth
support
 APP base algorithm processing and data display
Development Enviromment Based on Bluetooth
* GUI for Transmitter
* GUI for Receiver
- Main display
Search for Bluetooth
devices
Bluetooth switching
Display switching for test
After Bluetooth connection
Bluetooth switching
Waiting for connection
Sensor data transmission/Motion recog.
command
Increase sensitivity
- Test display
Display after algorithm
Processing on input data
Decrease sensitivity
Algorithm selection
Application of tilt variation with Euler formula
ACC
Data
GYRO
Data
ACC
Data
GYRO
Data
Integration
Integration
Mapping
Mapping
Matrix
multiply
Matrix
multiply
Display
Tilt
compensation
Display
Application of tilt variation with Euler formula
•
Solution of the tilt problem in right rotation
•
In case of right rotation, y output data is in the
opposite direction.
3. Results & Experiments
 Algorithm flow of the proposed pointing algorithm using
sensor fusion
ACC
Data
GYRO
Data
Integration
Mapping
Matrix
multiply
The number of loop
Quaternion
Rodrigues rotation
1 - 100
382,125 ns
411,118 ns
101 - 200
300,005 ns
442,801 ns
201 - 300
258,423 ns
423,145 ns
301 - 400
196,343 ns
406,271 ns
401 - 500
225,620 ns
435,145 ns
501 - 600
227,691 ns
680,828 ns
601 - 700
279,900 ns
919,970 ns
701 - 800
480,468 ns
838,399 ns
801 - 900
533,706 ns
919,483 ns
901 - 1000
374,154 ns
646,239 ns
Average
325,844 ns
612,340 ns
< Mapping process 연산 속도 비교 >
x(rodrigues)
-5
-0.7
x 10
x(quaternion)
y(rodrigues)
y(quaternion)
-0.8
Tilt
compensation
-0.9
-1
-1.1
Display
-1.2
-1.3
Page  36
0
10
20
30
40
50
60
70
80
90
< Mapping 처리 후 x, y 변화량 비교 >
100
3. Results & Experiments
 Tilt compensation & Hand tremor
Begin
Prediction on state variation
Current state estimation on current input noise
Continuous update on two uncertainty
Estimation similar to real measured data
False
True
End
< Tilt compensation algorithm >
Page  37
< Algorithm flow when Kalman filters are applied>
3. Results & Experiments
 Bluetooth based App-form algorithm development environment
Device(Client)
Android Application
X,Y
Host(Server)
Windows Application
Or
Button Click
<Transmission part >
Page  38
<Bluetooth Socket Communication >
3. Results & Experiments
 Mouse cursor control in PC using Smast
< Demonstration >
Page  39
< Simulation of Windows application >
Chapter 5
Bluetooth Socket Program
Bluetooth SPP (Serial Port Profile)
-
SPP is a profile to connect applications to serial port.
LMP, L2CAP, SDP, RFCOMM implemented on Baseband layer
Connect application through serial port with loading Serial Port Emulation(or other API)
on firmware layer.
Bluetooth SPP UUID : 00001101-0000-1000-8000-00805F9B34FB
Max. 30-port generation possible in RFCOMM protocol.
Bluetooth socket programming
-
send, recv / sendto, recvfrom function for data transmission/receiving in socket communication
(1) Function resources
send(int sockfd, const void *buf, size_t len, int flags)
recv(int sockfd, const void *buf, size_t len, int flags)
sendto(int sockfd, const void *buf, size_t len, int flags, const struct sockaddr *dest_addr, socklen_t addrlen)
recvfrom(int sockfd, void *buf, size_t len, int flags, struct sockaddr *src_addr, socklen_t *addrlen)
(2) TCP data transmission/receiving
send() / recv()함수는 address를 지정할 수 없기 때문에, 연결지향 방식인 TCP 통신에서 사용된다.
(물론 sendto / recvfrom도 사용 가능)
○ send(int sockfd, const void *buf, size_t len, int flags)
- sockfd : 목적지의 주소 정보를 갖는 파일 디스크립터
- buf : 전송하기 위한 데이터의 포인터
- len : 데이터의 길이
- flags : 함수의 호출이 어떤 일을 할지 나타내는 플래그
- return value : 전송 성공 시 전송한 바이트 수 반환, 실패 시 –1 반환
○ recv(int sockfd, const void *buf, size_t len, int flags)
- sockfd : 접속된 소켓의 파일 디스크립터
- buf : 수신 데이터를 저장할 버퍼의 포인터
- len : 수신할 데이터의 길이
- flags : 함수의 호출이 어떤 일을 할지 나타내는 플래그
- return value : 전송 성공 시 전송한 바이트 수 반환, 실패 시 –1 반환
→ 서버 측에서는 accept하여 생성된 소켓을, 클라이언트 측에서는 connect에 사용한 소켓을 이용하여 데이터를 송수신
한다.
Bluetooth socket programming
(3) UDP data transm/receiving
: sendto() / recvfrom()함수는 address를 지정할 수 있기 때문에, 연결지향 방식인 UDP 통신에서 사용된다.
○ sendto(int sockfd, const void *buf, size_t len, int flags, const struct sockaddr *dest_addr, socklen_t addrlen)
- sockfd : 소켓의 파일 디스크립터
- buf : 전송될 데이터를 가지고 있는 버퍼
- len : 버퍼의 데이터의 길이
- flags : 함수의 호출이 어떤 일을 할지 나타내는 플래그
- dest_addr : 데이터가 전송될 원격 호스트의 주소
- len : 주소정보 구조체의 길이(구조체의 길이를 입력 받으므로 포인터형)
○ recvfrom(int sockfd, void *buf, size_t len, int flags, struct sockaddr *src_addr, socklen_t addrlen)
- sockfd : 바인드된 소켓의 파일 디스크립터
- buf : 수신되는 데이터를 저장할 버퍼의 포인터
- len : 버퍼의 길이
- flags : 함수의 호출이 어떤 일을 할지 나타내는 플래그
- src_addr : 수신받은 데이터를 송신한 단말의 주소를 리턴받기 위한 버퍼 포인터
- addrlen : 주소정보 구조체의 길이(구조체의 길이를 입력 받으므로 포인터형)
Bluetooth socket programming
(4) System call for data transm./receiver
- write : Data transmission through stream socket
- read : Data receiving through stream socket
- send : data transmission(stream socket) with option
- recv : data receiving(Stream 소켓) with option
- sendto : data transmission through datagram socket
- recvfrom : data receiving through datagram socket
socket : Software medium for connecting remote two hosts
○ Socket generation
int socket(int domain, int type, iont protocol)
○ Address and port assignment
int bind(int sockfd, struct sockaddr *myaddr, int addrlen)
○ Waiting for connection request
int listen(int sockfd, int backlog)
○ Accept for connection
int accept(int sockfd, struct sockaddr *addr, int *addrlen)