Bio-inspired Vision
Dr.
D.J.
Lee,
Professor
Department
of
Electrical
and
Computer
Engineering
www.ece.byu.edu/faculty/djlee
Robo<c
Vision
Lab
http://www.ece.byu.edu/roboticvision/
Brigham
Young
University
Outline
  Can
we
can
do
anything
we
want
with
exis<ng
compu<ng
technologies?
  Is
robot
vision
possible?
  Bio‐inspired
applica<on
examples
–  Stereo
vision
–  Op<cal
flow
for
obstacle
detec<on
–  feature
and
target
tracking
for
a
micro
quad‐rotor
UAV
2/25/11
ECEn
631
2
Human Vision
2/25/11
ECEn
631
3
Human Vision
2/25/11
ECEn
631
4
Human Vision
2/25/11
ECEn
631
5
Dog Vision
2/25/11
ECEn
631
6
Bird Vision
2/25/11
ECEn
631
7
Insect Vision
2/25/11
ECEn
631
8
Image vs. Data or Signal
2/25/11
ECEn
631
9
2/25/11
ECEn
631
10
2/25/11
ECEn
631
11
2/25/11
ECEn
631
12
Robot Vision?
How could it be possible?
2/25/11
ECEn
631
13
Intelligent Ground Vehicle
Competition
2/25/11
ECEn
631
14
Autonomous Vehicles
2/25/11
ECEn
631
15
Road and Lane Detection
2/25/11
ECEn
631
16
Fixed-wing Micro UAV
2/25/11
ECEn
631
17
UAV Obstacle Avoidance
2/25/11
ECEn
631
18
UAV Ground Target Tracking
2/25/11
ECEn
631
19
Target Takeoff
2/25/11
ECEn
631
20
Midair Plane to Plane
2/25/11
ECEn
631
21
Parachute Deployment
2/25/11
ECEn
631
22
Motion Estimation - Differential
2/25/11
ECEn
631
23
Differential
2/25/11
ECEn
631
24
Motion Estimation – Motion
Zoom out
Zoom in
Pan to left
Focus of contraction
Focus of expansion
Motion vectors are parallel
Motion vectors converge
Motion vectors diverge
Sensor is receding
Sensor is approaching
2/25/11
ECEn
631
25
Time-to-Impact
2/25/11
ECEn
631
26
Grand Challenge
2/25/11
ECEn
631
27
Unmanned Ground Vehicle
2/25/11
ECEn
631
28
Unmanned Air Vehicle
2/25/11
ECEn
631
29
Unmanned Air Vehicle
2/25/11
ECEn
631
30
Ground Vehicle
2/25/11
ECEn
631
31
Motivation
  Algorithms
for
vision
compu<ng
systems
for
real‐world
applica<ons
are
becoming
complex.
  Latest
development
on
vision
compu<ng
algorithms
for
medical,
robo<c,
and
surveillance
applica<ons
requires
advanced
compu<ng
plaOorms
for
prac<cal
uses.
2/25/11
ECEn
631
32
Challenges
  Many
applica<ons
require
different
profiles
and
levels
of
performance.
  Advanced
compu<ng
plaOorms
must
provide
higher
performance,
beSer
flexibility,
lower
power
consump<on,
and/or
compact
size.
2/25/11
ECEn
631
33
Requirements
  Concurrent
design
of
algorithm
and
architecture
is
cri<cal.
  Development
of
a
fully
specified
algorithm
to
a
selected
architecture
is
not
adequate
anymore.
  Vision
compu<ng
algorithmic
development
and
mul<‐core
or
reconfigurable
architecture
development
must
be
done
concurrently.
2/25/11
ECEn
631
34
Options
  Algorithmic
and
architecture
op<miza<ons
on
emerging
compu<ng
plaOorms
having
mul<ple/
many
cores
and
reconfigurable
architecture,
including
FPGA,
mul<‐core
DSP,
mul<‐core
CPU,
graphical
processing
unit
(GPU),
and
grids
of
Cell
Processors.
2/25/11
ECEn
631
35
“A rough, quickly calculated motion estimation
is arguably more useful for navigation than a
more accurate, but slowly calculated estimate”
Vision Algorithm Implementations
System on a Chip using FPGAs
DSPs
GPUs
Cell Processors
General Purpose Processors
Visual C++, OpenCV
2/25/11
ECEn
631
36
Hardware vs. Software
  Processing
speed
is
cri<cal
for
many
vision
compu<ng
tasks.
  Many
algorithms
generate
accurate
results
but
run
too
slowly
to
produce
results
in
real‐<me.
  In
many
cases,
algorithms
process
at
camera
frame
rates
but
with
reduced
accuracy,
a
more
useful
combina<on
for
real‐<me
applica<ons.
2/25/11
ECEn
631
37
Hardware vs. Software
  SoZware‐based
vision
algorithms
–  Sophis<cated
but
accurate
–  Easy
to
implement
but
process
slowly
–  High
power
consump<on
  Restric<ons
on
hardware‐based
designs
–  Simplified
but
coarse
–  Hard
to
implement
but
could
process
at
frame
rate
–  Compact
and
low
power
2/25/11
ECEn
631
38
Hardware Implementation
  Prefer
non‐itera<ve
scheme
  Prefer
simple
arithme<c
opera<ons
  Prefer
pipeline
and
parallel
2/25/11
ECEn
631
39
CMOS Imager
  Solu<on:
Micron
MT9V022
Image
Sensor
–
–
–
–
Global
shuSer
sensor
1/3”
format
60fps
640x480
3.3V,
<320mW
at
maximum
data
rate
  Challenge:
–  Not
a
system‐on‐chip,
required
addi<onal
preprocessing:
demosaicing,
color
balance
2/25/11
ECEn
631
40
Daughterboard
  Autonomous
Vehicle
Toolkit
(AVT)
–  Removable
flash
memory
storage
–  Two
camera
inputs
–  Analog
transmission
system
with
DAC
–  Wireless
headers
for
Zigbee
or
2.4GHz
–  Spartan‐III
FPGA
for
pre‐processing
–  Servo,
Encoder,
5‐24V
Vin
regulator,
3.3V
and
5V
GPIOs
2/25/11
ECEn
631
41
The Platform
  On‐board
Processing
Solu<on:
Helios
FPGA
–  Approx.
size
of
a
deck
of
cards
–  Low
power
consump<on
(1‐3
WaSs
based
on
configura<on)
–  SDRAM
(32M),
SRAM
(4M),
Virtex‐4
FX‐20
and
FX‐60
variants
–  2
400
MHz
PowerPC
processors
–  USB
2.0
Connec<vity
2/25/11
ECEn
631
42
FPGA Architecture
(Xilinx Vertex-4 FX60)
DAC
Cameras
32M
SDRAM
Camera
Interface
2 PowerPC
Processors
ECEn
631
Vision
Processing
Engine
PLB to
OBP
Bridge
On-chip Peripheral Bus
(OPB)
2/25/11
DAC
Generator
Processor Local Bus (PLB)
4M SRAM
Video
Transmitter
Compass
GPS
Modems
Feedback
Signal
43
Embedded Vision Sensor
  3‐D
model
reconstruc<on







2/25/11
Obstacle
Detec<on
Mo<on
es<ma<on
Mo<on
segmenta<on
Structure
from
mo<on
Autonomous
vehicle
naviga<on
Smart
Vehicles
Unmanned
Air
and
Ground
Vehicles
ECEn
631
44
Example - Seeing Eye Glasses
2/25/11
ECEn
631
45
Motivation
  Canes
and/or
Seeing
Eye
dogs
are
tradi<onally
used
to
assist
the
visually
impaired
  They
are
usually
used
for
obstacles
on
the
ground
and
don’t
find
overhangs
  A
vision‐based
system
with
feedback
signal
would
help
overcome
these
and
other
problems
and
be
a
good
supplement
(not
replacement)
to
the
tradi<onal
methods
  Use
technologies
developed
for
military
applica<ons
(micro
Unmanned
Air
Vehicles)
2/25/11
ECEn
631
46
Goals
  Visual‐based
system
that
provides
real‐<me
visual
informa<on
by
aler<ng
the
user
to:
–  The
presence
of
obstacles
–  The
es<mate
distance
to
the
obstacles
–  The
es<mate
loca<on
of
objects
–  The
approaching
of
obstacles
  Two
similar
technologies
–  Tongue
Display:
A
single
camera
(camcorder)
with
voltage
s<mula<on
to
the
tongue
–  vOICe
vision
technology:
a
pair
of
cameras
and
a
PC
with
audio
as
the
feedback
signal
  Our
Embedded
Stereo
Vision
System
–  FPGA‐based
real‐<me
embedded
stereo
vision
sensor
–  Using
vibra<on
as
the
feedback
signal
–  No
awkward
or
embarrassing
features
(cosme<c
appeal)
–  Low
cost,
compact,
low
power
consump<on,
&
easy
to
use
2/25/11
ECEn
631
47
Approach
  An
es<ma<on
of
3‐D
informa<on,
accurate
3‐D
reconstruc<on
is
not
necessary
  Combine
custom
hardware
and
soZware
to
perform
1‐
D
signal
matching
for
stereo
vision
  Use
Spline‐based
representa<on
and
gene<c
algorithm
to
determine
the
disparity
in
hardware
in
parallel
  Does
not
need
edge
detec<on
and
window
size
selec<on
  Obtain
a
dense
disparity
in
or
near
real
<me
2/25/11
ECEn
631
48
Simulation Results
Test
images
were
taken
using
two
digital
cameras
Images
were
without
rec<fica<on
Tolerates
significant
intensity
varia<ons
An
automated
threshold
is
used
to
separate
out
imminent
threats
from
other
objects
  Erosion
and
dila<on
clear
up
much
of
the
noise
in
the
image
  Processing
9~11
640x480
fps




2/25/11
ECEn
631
49
Simulation 1
2/25/11
ECEn
631
50
Simulation 2
2/25/11
ECEn
631
51
Simulation 3
2/25/11
ECEn
631
52
Simulation 4
2/25/11
ECEn
631
53
Optical Flow algorithm for
obstacle Avoidance
  Mo<va<ons
–  Adequate
accuracy
is
needed
for
real
applica<ons.
–  Removal
of
erroneous
mo<on
vectors
due
to
lack
of
smoothing.
–  BeSer
hardware
plaOorm
became
available.
–  Slower
speed
is
acceptable
as
long
as
the
hardware
cores
can
catch
up
with
the
camera
frame
rates.
2/25/11
ECEn
631
54
Example – Optical Flow Computation
2/25/11
ECEn
631
55
Yosemite
2/25/11
ECEn
631
56
Flower Garden
2/25/11
ECEn
631
57
System Diagram
2/25/11
ECEn
631
58
DER Module
  Calculate
deriva<ve
frames
from
raw
images
  Deriva<ve
Mask
[1
‐8
0
8
‐1]
2/25/11
ECEn
631
59
OFC Module
  Calculate
op<cal
flow
from
deriva<ve
frames
2/25/11
ECEn
631
60
Motion Estimation
2/25/11
ECEn
631
61
Obstacle Avoidance
2/25/11
ECEn
631
62
Quad-rotor Hovering Micro UAV
2/25/11
ECEn
631
63
Example – Flight Stabilization
  Solu<on:
–  Carbon‐fiber
based
frame
•  Reduced
weight
without
loss
of
strength
–  Axi
brushless
motors
•  Increased
efficiency
and
cooler
opera<ng
range
–  Es<mated
5lb
liZ
capacity
–  Total
frame
weight:
2.5
lbs
–  Flight
<me:
30~40
minutes
2/25/11
ECEn
631
64
Helio-copter Platform
  Helios
FPGA
board
  Quad‐rotor
frame
  Kestrel
Autopilot
  CMOS
image
sensor
2/25/11
ECEn
631
65
Results
  On‐board
real‐<me
image
processing
  >30
fps
  Stabilize
driZ
  Track
target
2/25/11
ECEn
631
66
Quad-rotor Hovering Micro UAV
2/25/11
ECEn
631
67
Helio-copter System Overview
2/25/11
ECEn
631
68
Harris Feature / PQ Correlation
  Harris
feature
strengths
are
thresholded
and
priori<zed
  RANSAC
finds
two
points
that
best
represent
set
2/25/11
ECEn
631
69
Harris Feature Results
  Harris
–  Robustly
detects
features
  Priority
queue
–  Corresponds
features
well,
20+
per
frame
–  Not
robust
to
similar
feature
strengths
–  Speed
related
to
number
of
features
priori<zed
2/25/11
ECEn
631
70
Harris Feature / Template Match
  Filters
features
too
close
to
image
edges
  Template
match
core
reads
image
data
from
memory
–  Template
–  Search
area
–  At
each
feature
2/25/11
ECEn
631
71
Template Match Results
  Uses
sum
squared
differences
  Tracks
>120
features/frame
  Template
robust
to
similar
feature
strength
values
2/25/11
ECEn
631
72
RANSAC Homography
  Randomly
pick
sets
of
two
correlated
points
  Calculate
similarity‐
constrained
homography
  If
within
threshold,
increment
vote
  Repeat
2/25/11
ECEn
631
73
Least Squares vs RANSAC
  Simula<on
results
show
RANSAC
achieves
comparable
results
without
computa<on
of
itera<ve
least
squares
approach
2/25/11
ECEn
631
74
Color Connected Components
  RGB
to
HSV
  Streaks
are
stored
in
memory
  Streaks
are
connected
  Center
of
mass
used
for
Kalman
filter
2/25/11
ECEn
631
75
Radial Distortion
  Scale
depends
on
distance
between
dots
  Quad‐rotor
drops
when
target
moves
away
from
center
  Quadra<c
approxima<on
of
inverse
mapping
  Successful
correc<on
2/25/11
ECEn
631
76
Kalman Filter
  Update:
dot
loca<on
in
image
  Predict:
if
found
red
2/25/11
ECEn
631
else
77
Kalman Filter
  Kalman
filter
predic<on
of
dot
loca<on
when
it
is
lost
  Keeps
quad‐
rotor
stable
2/25/11
ECEn
631
78
FPGA Hardware Resources
  Current
configura<on
of
vision
algorithms
consume
1/3
of
FPGA
resources
  Capable
of
addi<onal
algorithms
–  Stereo
vision
–  Op<cal
flow
2/25/11
ECEn
631
79
Results
2/25/11
ECEn
631
80
Conclusion
  Using
the
exis<ng
compu<ng
technologies,
there
is
no
way
we
can
mimic
complex
vision
capabili<es
  Let’s
seSle
for
low‐level
bio‐inspired
primi<ve
vision
  Accurate
but
slow
is
not
as
useful
as
coarse
but
fast
  Real‐<me
vision
has
great
poten<al
in
solving
well‐defined
vision
problems
2/25/11
ECEn
631
81