Mobile Motion Tracking using Onboard Camera
Supervisor:
Prof. LYU, Rung Tsong Michael
Prepared by:
Lam Man Kit
Wong Yuk Man
Agenda
Motivation & Objective
Previous Work
Improvement & New components
Sample Applications
Conclusion
Q&A
Motivation
Rapid increase in the use of cameraphone
Camera-phone can perform image
processing tasks on the device itself
Enhance human-computer interaction
by mobile phone
Objective
Implement real-time motion tracking on
Symbian phone, without requiring
additional hardware
Translational motion tracking
Rotational motion tracking
To develop a tracking engine for other
mobile devices developers to use
Previous Work
A translational motion tracking engine
Blocking matching
• SEA (Successive Elimination Algorithm)
• PPNM
• PDE (Partial Distortion Elimination)
• Adaptive Window Search
• Spiral Scan
Feature selection
A translational motion tracking engine
Block Matching Algorithm
Block Matching Algorithm
Evaluate the "goodness" of a match by Sum
of Absolute Difference
N
N
SAD( x, y)   | X (i, j )  Y (i, j ) |
i 1 j 1
Select the candidate block with the
lowest error
Feature Selection
A good feature block:
High variance -> complex block
Not in a repeated pattern
If a good feature block is found, the
performance of the motion tracking is
improved
We have made improvements to our existing
algorithm
Old Feature Selection
Divide the current frame into squares
Apply Feature Selection Algorithm to each squares
However, the best feature block may appear between
two squares
squares
New Feature Selection
More blocks are sampled
Sample 26x26 variances of blocks with size
15x15 in a 54x54 window
Search in spiral way and stop searching
when selection criteria are matched
Prefer to find a feature block in the center
Prevent out of bound problem (block
appear out of screen)
New Feature Selection
If a feature block is
found on the edge,
the following will
happen:
The tracking
algorithm will fail to
find the exact match
Solution:
Apply additional
constraint
New Feature selection
In order to find a feature block that is not on the
edge, we apply a checking algorithm
Important difference in all directions => interest point
New Feature Selection
Now our newest feature selection algorithm becomes:
Find a block with a large variance -> which indicate the
complexity of the block
Check if the block is on the edge or not by calculating the
SAD between the candidate block and its 4 neighbors
• If either one of the SAD is small -> the block is on edge ->
reject
• Else the block is not on edge
SSD is used instead of SAD
SSD = Sum of Squared Difference
Last term, SAD is used as matching
criteria
SAD is commonly used because of its
lower complexity (faster)
But we chose to use SSD finally
SSD is used instead of SAD
Experiment shows that SSD has better
performance in accuracy and
Algorithm with SSD run as fast as that
with SAD
Reason (of why as fast as SAD):
Elimination effects from SEA, PPNM & PDE
become large when SSD is used
Compensate the higher complexity of SSD
SSD is used instead of SAD
SEA and PPNM
lower bound is
adjusted using the
inequality (12)
proposed in the
following paper
J.J. Francis and G. de Jager. A Sum
Square Error based Successive
Elimination Algorithm for Block
Motion Estimation (2002)
Experimental Result on Symbian
phone
Frame rate
Maximum frame rate supported by Nokia
6600 is about 14 frames/sec
Running our algorithm (1/0.007 frames/sec)
once only for each frame do not slow down
the displaying frame rate, in other word,
the frame rate can still be 14 frames/sec,
our algorithm doesn’t lag the display
Rotation Tracking Engine
Observation and Motivation
As we rotate the
phone, the object can
still be tracked
correctly (center of
object roughly equals
center of green box)
Rotation Tracking Engine
Our approach
If two blocks are
tracked at the same
time, angle of line
connecting the two
blocks reflect the
tiling angle of the
phone
Two-block approach
Rotation Tracking Engine
Another approach
Track one block by simple
motion tracking engine,
then find which rotation
gives the best match
Fail if tracking object is the
same for different angle
One-block approach
Rotation Tracking Engine
Modification of motion tracking
algorithm to suit rotation tracking
Linear Adaptive method used in translation
motion tracking is not used here because
phone’s motion can be both linear motion
and circular motion
Rotation Tracking Engine
What to predict?
Given the coordinates of the tracking blocks in the last 2
previous frames
Predict the coordinates of the blocks in the next frame
Line L1
Line L2
Line L3
θ
θ
A simplified mathematic problem (the
following assumptions are approximately
correct)
All three lines pass through the circle’s
center
End points of the three lines lie on the
circle
Rotation Tracking Engine
Solution
Angle between L2 and L3 = θ
• θ = Tan-1(slope(L2)) – Tan-1(slope(L3))
Coordinates of the next tracking block (xL1, yL1)
are calculated by multiplying the column matrix
of coordinates of the previous block with a
rotation matrix
 xL1  cos 
y   
 L1   sin 
 sin    xL2 
y 

cos    L2 
Rotation Tracking Engine
Solution
The prediction of the position of the next tracking block
should also take the translational movement into account
Horizontal displacement = Tx
• Tx = ( xL21 + xL22– xL31 - xL32 )/2
Vertical displacement = Ty
• Ty = ( yL21 + yL22– yL31 - yL32 )/2
Coordinates of the next tracking block (xL1, yL1) is
calculated by
 xL1  cos 
 y    sin 
 L1  
 1   1
 sin 
cos 
1
Tx   xL2 
Ty   yL2 
1   1 
Rotation Tracking Engine
A simple drawing to show the detected
rotation angle of the phone
Rotation Tracking Engine
Reducing error by using level
Apply threshold on output, increase/decrease one
level only when change is large
To give less sensitive but more desirable output
for game
e.g. skiing game: skier face only to 7 directions
Reduce difficulty, increase reliability
A small rotation or a small detection error will not
increase/decrease one level
Conditions to be a good
background for both engines
Objective
Max. performance measurement
Increase usability
Condition:
Feature selection can always find a
good feature point
Within certain distance
• No repeat pattern
• Very distinct pattern
Pattern is nearly the same when
rotated. E.g. Circle which is good for
rotation detection
Virtual Mouse
An application that make full use of the
translational motion detection engine
Remote control the mouse of PC by Symbian
phone using motion tracking
Advantages:
It allows input in all directions
It provides high levels of control
Joystick can still be used as rough moving while
camera input can be used as fine translation
Virtual Mouse
Server
In server side (Windows), the Bluetooth is
configured to provide RFComm Service and
server regards the Bluetooth transmission
port as Comm. Port
Server receives message from that Comm.
Port
Call function in MouseAction.h to make the
mouse move and trigger mouse click event
Virtual Mouse
Client
Search for Bluetooth device nearby
Connect to the selected Bluetooth device
Every time motion tracking algorithm
finishes, results are sent to server (12
times/sec)
Joystick and keypad can also be used to
control mouse in PC (Multi-button mouse)
Car Racing Game
A “Car Racing Game” is developed using the
motion tracking engine.
Game lags. It is because:
The engine takes time to find the motion vector
CPU speed of the Symbian phones are low
The game engine thread uses most of the CPU
power while the thread for updating the screen
cannot proceed
Solutions:
Double Buffering
Direct Screen Access
Single Thread
Double Buffered Area
Double-buffering – two complete color buffers
One is displayed while the other is
calculating next scene to be drawn,
When the drawing of the next frame is
completed, the content of the back buffer
is copied to the front screen
Front
Screen
Computing
Frame 2
Frame 1
Computing
……Frame 2
Show Frame 2
Buffer 1
copy
Buffer 2
Frame 2 ready
Direct Screen Access
In Symbian, traditional drawing requires the help of
Window Server
Overhead in context switching
Lower speed
Access the screen directly in 3 ways:
Creating and using CfbsScreenDevice
Accessing screen memory directly.
Using CdirectScreenAccess.
Using DSA can bypass the Window Server
Get rid of context switching
Faster
Traditional Graphic Programming
Device Screen and Keypad
Key Presses
Update Display
Window Server
RWsSession
RWsSession
RWsSession
Application 1
Application 1
Application 1
Direct Screen Access
1. Request to Window
Server to work in Direct
Screen Access mode
Application
Window Server
2. If successful, a region for
drawing to is returned to
the application.
Single Thread
Prevent context switching
No need to do synchronization
Car Racing Game
With the use of efficient graphic programming
algorithms, the tracking engine will not affect the
performance of the game
It shows that we can use the engine to develop other
applications without performance degradation
Skiing Game
Sample program from the book “Developing Series 60
Applications: A Guide for Symbian Os C++
Developers”
Based on this game, we plug in our rotation tracking
engine into it
The game becomes more interactive
Rotate the phone to control the angle of the skier
Skiing Game
Skiing Game
The process of using the engine is very
simple
Create an instance of the tracking engine
Call the function of the engine to find the motion
vector
Use the motion vector for the game logic
Skiing Game
Demo
Experimental Result on Symbian
phone
Testing Environment
Platform
• Symbian Phone Nokia 6600
Algorithm
• Our final (hybrid-type) Algorithm (Block matching algorithm
featured with Adaptive Window, Spiral Scan, SEA, PPNM and
PDE method)
Algorithm parameter
• Block size
• Search window size
= 17 x 17 (pixels)
= 16 x 16 (pixels)
Number of block to track in each run of algorithm = 1 block
ONLY
Experimental Result on Symbian
phone
Testing Result (avg time to run)
Final algorithm
• 7ms
Final algorithm without adaptive window
method and SEA, PPNM method
• 22ms
Full Exhaustive Search algorithm (the most
simplest one)
• 55ms
Q&A