Query
A ti Q
i g ffor M
bil
ti S
h
yS
Sensing
Mobile
Location
Search
Active
bl L
Felix X. Yu Rongrong
Felix
l X Yu
Shih‐Fu
h h Fu Chang
Fu
h g
Feli
Y Rongrong
Rongrong Ji Shih
F Chang
Chan
Digital Video and Multimedia Laboratory, Columbia University, 10027, New York, United States
d and
d Multimedia
l
d Laboratory
b
l b Universityy, 10027, New Yorkk, United
d States
Digital
g l Video
y, Columbia
Abstract
•
•
System Architecture
y
System
Architecture
Problem Justification
Problem
Justification
P bl
J tifi ti
Wh
h first
fi
f il to find
fi d the
h right
igh target
q
g ((up
p to 50%
When
the
queryy fails
likelihood),
likelihood) how should the user form his/her search strategy
in the subsequent interaction?
We propose a novel Active Query Sensing system to suggest
the best way for sensing the surrounding scenes while forming
the
queryy for
th second
dq
f location
l
ti search.
h
search
•
•
•
•
•
Most locations are recognizable only with a subset of the
ie s
views.
views
Each location has a unique subset of preferred views for
g iti
recognition.
recognition
Location Dependence
Dependence:: different locations have different
degrees
off “difficulty”
difficulty
d g
“diffi
lty”.
Locations of the same search difficulty do not significantly
l t together.
t g th
cluster
together
There is no single dominant view that can successfully
ll locations.
l
ti
recognize
g i all
locations
iterations
Failure rates over successive query iterations.
•
•
Application
Application Scenario
A
li ti SScenario
i
User Interface
User
U Interface
I t f
NAVTEQ 0.3M NYC Data Set
Q 0 3M NYC Data
D t Set
S t
NAVTEQ
•
Experiment
E p i
t
Offline analysis: to discover the best query view(s) for each location indexed by the system.
system
user
Online process: to estimate the most likely view of the first query (which has failed); to suggest optimal view change to the user.
•
•
h data
d
i
b
i g off 50,000
0,000 locations
l
i
300,000 images
50
The
set consists
off about
300,000
in Manhattan
collected
Q street view imaging
g g system.
y
h
ll
d by
by the
h NAVTEQ
A
system
Online Analysis
li Analysis
l i
Online
Offline Analysis
Offline
ff
Analysis
•
•
•
First q
queryy
First query view
view (estimated)
(
d))
Conclusions
l i
and
d Future Works
k
Conclusions and Future Works
•
Geographical Distribution of NAVTEQ Data Set
g p
Q Data Set
Geographical
Distribution of NAVTEQ
Problem
Problem Formulation
P
bl
FFormulation
l ti
•
Most
Most salient view
salient view
pti
j k image
A
U
i unlikely
lik ly to
t take
t k “junk
“jjunk
i g ” with
ith no hope
h p for
f
Assumption:
User
is
image”
fi di g the
th true
t
t g t The
Th first
fi t q
f l can be
b used
d
target
query
y even unsuccessful,
unsuccessful
finding
target.
query,
“p
b ” to
t narrow down
d
th solution
l ti space.
space
p
as “probe”
probe
the
•
S
Suggestion:
ti
t
turn
right
i ht 90 degrees
d
Suggestion: turn right 90 degrees
•
Examples of a p es o
ju
junk images
ages
Examples
of “junk”
images
•
Configuration
Cameras
Configuration of Cameras
C
fi
ti off C
•
•
To
the
search
T simulate
i l
h mobile
bil location
l
i
h scenarios,
i , we manually
lly
scenarios
i from
f
G gl Street
S
Vi
i 226 randomly
d ly chosen
h
cropped
pp d q
queries
Google
View
in
l
i
d by
by the
h above
b
i
d routes in
i NYC.
locations
covered
mentioned
NYC
F
h location,
l
location
ti
i query images
i g
d from
f
i i g
For
each
six
are cropped
viewing
angles
orientations
used
database
This
gl similar
i il to
t the
th view
i
i t ti
d in
i the
th database.
d t b
Thi
1 356 images
results
with
tags.
lt in
i 1,356
i
ith angles
l and
d ground
d truth
t th locations
l
ti
t
RESEARCH POSTER PRESENTATION DESIGN © 2011
P t P
t ti
www.PosterPresentations.com
•
•
We
developed
Sensing
that
W have
h
d l
d a novell Active
A ti Query
Q
S i system
t
th t actively
ti l
gg
h best
b q
gy if/when
if/ h the
h first
fi visual
i lq
ffails
il
suggests
the
queryy strategy
queryy fails.
This the first
effort
in activelyy gguidingg users to achieve more
f
ff
satisfactory experience in using mobile visual search.
search
We
measurementt based
to
W develop
d l saliency
li
b d on score distributions
di t ib ti
t
di t the
th robustness
b t
h query view
i and
d the
th search
h difficulty
diffi lt off
predict
off each
each
h location.
llocation
i
h future,
f
h idea to multi‐view
l view object
bj search.
h
, we willll extend the
In the
future
multi
search
Th
t ways
y for
f estimating
ti ti g the
th first
fi t query
q y view
i
There are two ways for estimating the first query view. There
are two
Examples of query images
p of q
g
Examples
queryy images
•
q
q
A subsequent
queryy should maximize the discriminabilityy
((uncertainty
y reduction)) over candidate locations,
locations, narrowed down byy
h q
l dy been
b
taken
k , by
by selecting
l i g the
h best
b
view
i
q
the
queryy already
taken,
queryy view.
Train view classifiers offline
Without
showing
process
details
p
Wi h
h i g mathematical
h
i l details,
d il , we found
f
d the
h above
b
b wellll approximated
pp i t d by
by
can be
View
e alignment
a g e t based on
o the
t e image
age matching
atc g
•
•
W simulate
i l t the
th first
fi t query with
ith a randomly
d l chosen
h
i i angle.
angle
l
We
viewing
Only
query
succeed
O ly 47% off the
h random
d
i g succeed,
d, resulting
l i g in
i a 53%
q y images
failure rate after the first q
queryy
query.
We evaluate the performance of reducing the failure rates in
s bseq ent queries
q eries by
b using
sing different active
acti e query
q er strategies.
strategies
subsequent
Th performance
f
i achieved
hi d by
b the
th saliency
li
b d AQS scheme
h
i
The
gain
based
is
quite
q
query.
i impressive
i p
i – 12% error rate after
f only
ly one additional
ddi i
lq
queryy
ggrade the
The offline measures of saliencyy can also be used to “grade”
searchability of each location,
location indicating the locations where the
system
performs
more robustly.
t
f
b tl
LN is the candidate locations narrowed down by the first query.
query
We use a majority
view
j y votingg mechanism to estimate the optimal
p
angle change and suggest the user turns to the most salient view.
view
•
A ideal
id l score distribution
di ib i is
i the
h one that
h has
h maximal
i l separation
p
i
An
between the scores of the p
positive results and those of the negative
g
ones
ones.
b using
sing the reference image of
Predict online search performance by
h {location,
{l ti
{location
i } to
t retrieve
ti
th same location:
l ti
each
view}
the
Whi h view
i
ill be
b the
th best
b t query?
y?
Which
will
g
Acknowledgement
W thank
th k NAVTEQ for
f providing
idi g th
i g data
d t set
t Dr
D
Xi Chen
Ch
We
the NYC image
set,
Dr. Xin
help
and
Bach
help,
d Dr.
D Jeff
J ff B
h for
f their
th i generous
g
h
l and
d Tongtao
T gt Zhang
Zh g for
f
d ig i g the
th mobile
bil interface.
iinterface
t f
designing