Absolute Localization of Mobile Robots in Forest Environments
By Correlating Ground LiDAR to Overhead Imagery
by
Marwan Hussein
B.A.Sc., York University, Toronto, Canada (2007)
Submitted to the Department of Mechanical Engineering and Department of Engineering
Systems Division in partial fulfillment of the requirements for the degrees of
MAssACHUSEMrS MN
OF TECHNOLOGY
Master of Science in Mechanical Engineering
and
JUN 2 6 2014
Master of Science in Engineering and Management
LIBRARIES
at the
MASSACHUSETTS INSTITUTE OF TECHNOLOGY
May 2014
( Massachusetts Institute of Technology 2014. A rights reserved.
Signature redacted
Author
Department of Mechanical EnginL'ering, System Design and Management
May 2014
Signature redacted
Certified by
Dr. Karl Jagnemma
Department of Mechanical Engineering
Thesi Spervisor
Certified and Accepted by.
-Signature redacted
,Jrctor,_SDM
Accepted by.
_--
_-_-_-_-_
Tatrick Hale
*lvs Program
Signature redacted
Dr. David Hardt
Chairman, Committee on Graduate Studies
E
-Blaink-
Pagc 2
of'140
Absolute Localization of Mobile Robots in Forest Environments By Correlating
Ground LiDAR to Overhead Imagery
by
Marwan Hussein
Submitted to the Department of Mechanical Engineering and the Department of
Engineering Systems Division on May 2014, in partial fulfillment of the requirements for
the degrees of
Master of Science in Mechanical Engineering
and
Master of Science in Engineering and Management
Abstract
A method for the autonomous geolocation of ground vehicles in forest
environments is presented. The method provides an estimate of the global horizontal
position of a vehicle strictly based on finding a geometric match between a map of
observed tree stems, scanned in 3D by Light Detection and Ranging (LiDAR) sensors
onboard the vehicle, to another stem map generated from the structure of tree crowns
analyzed from aerial orthoimagery of the forest canopy. Identification of stems from 3D
data is achieved by using Support Vector Machine (SVM) classifiers and height above
ground filters that separate ground points from vertical stem features. Identifications of
stems from overhead imagery is achieved by calculating the centroids of tree crowns
extracted using a watershed segmentation algorithm. Matching of the two stem maps is
achieved by using a robust Iterative Closest Point (ICP) algorithm that determines the
rotation and translation required to align both datasets. The final alignment is used to
calculate the absolute location of the vehicle. The localization algorithm has been tested
with real-world data and has been able to estimate vehicle geoposition with errors of less
than 2.1 m RMS. It is noted that the algorithm's accuracy is currently limited by the
accuracy and resolution of the aerial orthoimagery used.
This method can be used in real-time as a complement to the Global Positioning
System (GPS) and Simultaneous Localization and Mapping (SLAM) in areas where
signal coverage is inadequate due to attenuation by the forest canopy, or due to
intentional denied access. The method has two key properties that are significant: i) It
does not require a priori knowledge of the area surrounding the robot. ii) Based on
estimated vehicle state, it uses the geometry of detected tree stems as the only input to
determine horizontal geoposition.
Page 3 o0
140
Thesis supervisor: Dr. Karl lagnemma
Title: Principal Investigator
Robotic Mobility Group
Department of Mechanical Engineering
Thesis supervisor: Mr. Patrick Hale
Title: Director
SDM Fellows Program
Engineering Systems Division
Page 4 of 140
-Blank-
Pagc 5 of 140
Acknowledgements
I would like to thank my research advisor Dr. Karl lagnemma for his guidance
and support during the last two years. His pragmatic approach helped me adapt to the
daunting experience of transitioning from industry to academia after an almost 5 year
gap. He has taught and inspired me in many ways and got me excited about the future of
autonomous robotics. He has become a mentor and a good friend. I also would like to
thank Mr. Patrick Hale for his advice throughout my studies at MIT. I have never felt
away from home thanks to his warm and friendly attitude. My mother Lubna (Halawa!),
my sister Leen, my brother Sinan and my aunt Ghada can't be thanked enough for their
encouragement, love and support during the time in which this research was performed.
This research was supported by the U.S. Army Engineer Research and
Development Center (ERDC) under contract number W912HZ-12-C-0027. The contract
was monitored by Mr. Matthew Renner who I also thank for providing feedback and
ideas throughout the project. Other financial support to the author was provided in the
form of scholarships and grants from the National Sciences and Engineering Research
Council of Canada (NSERC) known as the Alexander Graham Bell Canada postgraduate
scholarship and the NSERC postgraduate scholarship.
At the time of writing of this thesis the consumer electronics market is ripe for yet
another wave of disruption. Wearable computers and virtual reality devices are starting to
appear in the market, albeit at ridiculously low technology readiness levels. After a lot of
time spent thinking, interviewing, and choosing, I have decided to join a quirky little
company where we plan to change the face of the computer industry as we now know it. I
realize this is going to be a challenging endeavor given the size of the incumbents and the
complexity of the technology. Nevertheless, I'm willing to take my chances. As Ralph
Emerson once said: "Do not go where the path may lead, go instead where there is no
path and leave a trail". This is my calling!
"Per audacia ad astra"
Through Boldness to the Stars
Marwan Hussein
Cambridge, Massachusetts, May 2014
Page 6 of 140
-B~lank-
Pagc 7
of 140
Contents
CHAPTER 1: INTRODUCTION ...................................................................................................
23
1.1
RESEARCH BACKGROUND AND MOTIVATION ...........................................................
25
1.2
THESIS PROBLEM FORMULATION .................................................................................
26
1 .2 .1
T H E SIS O B JE CT IVES ................................................................................................................................
26
1.2.2
MATHEMATICAL PROBLEM DEFINITION ........................................................................................
28
1.2.3
CONSTRAINT ON T HESIS SCOPE ......................................................................................................
29
1.3
P R EV IO U S W O RK ......................................................................................................................
29
1.4
T H ESIS O V ERV IEW ...................................................................................................................
35
CHAPTER 2:
MISSION
DEFINITION
AND
ARCHITECTURE
OF
LOCALIZATION
A LG O R IT H M ..........................................................................................................................................
39
2.1
M ISSIO N CO N T TEXT ...................................................................................................................
39
2.2
CONCEPT OF OPERATIONS.................................................................................................
45
2.3
MISSION REQUIREMENTS ...................................................................................................
48
2.4
ARCHITECTURE: SUBSYSTEMS AND COMPONENTS...................................................
49
2.5
INPUTS AND OUTPUTS ...........................................................................................................
53
CHAPTER 3: TREE CROWN IDENTIFICATION, DELINEATION
AND
STEM CENTER
ESTIMATION FROM OVERHEAD IMAGERY .............................................................................
56
3.1
TREE MORPHOLOGY AND MODELING ..........................................................................
56
3.2
STEM CENTER ESTIMATION HYPOTHESIS.......................................................................
60
3.3
IMAGE PROCESSING METHODOLOGY .............................................................................
61
3.3.1
GREEN CHANNEL EXTRACTION AND DENOISING .........................................................................
63
3 .3 .2
B IN A RY IM A GE C REAT IO N .....................................................................................................................
67
3 .3 .3
W ATERSH ED SEGM ENTATIO N ..............................................................................................................
68
3 .3 .4
3 .3 .5
C RO W N C LA SSIFIE R ................................................................................................................................
C EN TRO lD E STIM ATION .........................................................................................................................
70
72
3.4
TRADE SPACE AND PERFORMANCE ASSESSMENT ....................................................
Page 8
of 140
73
CHAPTER 4: TREE STEM IDENTIFICATION, EXTRACTION AND CENTER ESTIMATION
FROM ROVER LIDAR DATA ..............................................................................................................
78
4.1
STEM CENTER IDENTIFICATION HYPOTHESIS............................................................
78
4.2
DATA PROCESSING METHODOLOGY.............................................................................
80
4 .2 .1
4.2.2
4 .2 .3
4.2.4
4 .2 .5
4.3
G R O U N D H EIG H T F ILT ER .......................................................................................................................
SUPPORT VECTOR MACHINE CLASSIFIER ......................................................................................
B REA ST H EIGHT STEM SLICER .............................................................................................................
LEAST SQUARES PRIM ITIVE FITTING .............................................................................................
STEM C EN TER C ENTROID IN G ................................................................................................................
ACCURACY OF STEM CENTER LOCATIONS .......................................................................
81
84
84
85
88
89
CHAPTER 5: ROVER LOCATION ESTIMATION BY MATCHING TREE CENTERS MAPS
GENERATED FROM ROVER LIDAR TO OVERHEAD IMAGERY ...............................................
93
5.1
MAP-BASED MATCHING HYPOTHESIS...........................................................................
93
5.2
OVERVIEW OF MAP-BASED MATCHING ALGORITHMS ................................................
95
5.3
DATA MATCHING METHODOLOGY .................................................................................
97
5 .3 .1
5.3.2
5.3.3
5.3.4
S E A R C H S PA C E .........................................................................................................................................
ROBUST ITERATIVE CLOSEST POINT ALGORITHM .........................................................................
CALCULATION OF MOBILE ROBOT GEOPOSITION BASED ON MATCH ........................................
PERFORMANCE ANALYSIS OF ROBUST ICP ALGORITHM ..............................................................
98
100
102
103
CHAPTER 6: EXPERIMENTAL RESULTS...............................................................................107
6.1
IN T R O D U CT IO N ......................................................................................................................
6.1.1
STUDY A REA AND DATA PREPARATION ..........................................................................................
107
107
6.2
SIM U LA TIO N SET U P ..............................................................................................................
110
6 .3
R E SU LT S ....................................................................................................................................
1 12
6.3
ACCURACY ASSESSMENT......................................................................................................123
6.4
SENSITIVITY ANALYSIS ........................................................................................................
125
6.5
KALMAN FILTER APPLICATION .........................................................................................
127
CHAPTER 7: THESIS SUMMARY, CONCLUSIONS AND RECOMMENDATIONS..................130
7.1
T H ESIS SU M M A RY ..................................................................................................................
130
7.2
CO N T R IB U TION S ....................................................................................................................
132
7.3
LIM IT A TIO N S ...........................................................................................................................
13 2
IPac 9 of' 140
7.4
RECO M M EN DA TIO NS FO R FUTU RE W O RK ....................................................................
R E FE R EN C ES ....................................................................................................................................
Page 10 of 140
132
135
List of Figures
Figure 1: Horizontal locations of features (7-point stars) as observed by a LiDAR sensor
onboard a m obile robot on the ground...................................................................
26
Figure 2: Overall thesis development roadmap (Chapters in bold square brackets)........ 36
Figure 3: Envisioned operational scenario of a solo mobile robot ...............................
40
Figure 4: Example of dense coniferous forest during summer season (Harvard
Arboretum, Cambridge, MA, from Google Earth) ..............................................
42
Figure 5: Sensor and mobile robot mission configuration showing the anticipated
m inim um working distances and FOV .................................................................
43
Figure 6: Sequence of operations (flowchart) for localization framework showing the
teleoperation and autonomous control scenarios .................................................
47
Figure 7: Functional breakdown of localization algorithm with three levels of hierarchy:
system , subsystem and com ponents......................................................................
50
Figure 8: Block diagram of localization algorithm showing the three main subsystems
and their com ponents............................................................................................
52
Figure 9: Optical tree model showing various parameters that define a particular tree . . 57
Figure 10: Different morphologies of trees including pyramidal (conifer) trees.......... 58
Figure 11: Different types of horizontal crown profiles as seen by an overhead camera
showing isolated, touching and clusters of crowns...............................................
60
Figure 12: Image processing flowchart given an input orthophoto image of the
exp lo ration site .....................................................................................................
. 62
-
Figure 13: Original orthophoto input image of Lake Mize test site (Source: USGS
G oog le E arth ).......................................................................................................
Figure 14: Resultant image after green object extraction with all non-green objects
re m o v e d .....................................................................................................................
Page I I of 140
. 64
65
Figure 15: Resultant image after application of luminance algorithm that converts RGB to
8 -b it g ray scale ...........................................................................................................
66
Figure 16: Result after application of Gaussian blur function using erosion and dilation
techniques to rem ove spurious pixels ...................................................................
67
Figure 17: Resultant image after creating a binary copy of the grayscale image............. 68
Figure 18: Result of applying watershed segmentation to binary image N, to delineate
individual tree crow ns..........................................................................................
70
Figure 19: Result after applying the two-stage classifier with green objects denoting valid
crowns and red objects denoting invalid crowns ...................................................
71
Figure 20: Final result showing constellation map S with all crowns delineated and valid
centroids denoted by (+) ........................................................................................
72
Figure 21: Nearest neighbor search results from comparing stem locations in overhead
image to those manually estimated from ground LiDAR data ..............................
74
Figure 22: Comparison between stems centers detected in overhead image to handlabeled stem centers observed in ground LiDAR data...........................................
75
Figure 23: 3D slices of different tree stems for a tree stand observed in 3D LiDAR point
c lo u d [5 9 ] ..................................................................................................................
79
Figure 24: Centroid calculation by fitting a cylinder primitive [60] .............................
79
Figure 25: Data processing flowchart for the stem center estimation algorithm given an
input LiDAR dataset of the area surrounding the mobile robot.............................
80
Figure 26: Sample data showing LiDAR scans of forest grounds at the Lake Mize test site
with vertical height color-coding (red: high, blue: low)......................................
82
Figure 27: Example of a LiDAR dataset at a particular pose showing the simulated
m obile robot, tree stem s and underbrush ...............................................................
82
Figure 28: Cartesian space tessellated into 0.5 m x 0.5 m cells [60]............................
83
Figure 29: Consecutive LiDAR scans and corresponding stem slices [60]...................
85
Figure 30: Example showing cylinder primitive fitment to a single slice of LiDAR data of
a tree stem [6 0 ]..........................................................................................................
Page
12
of 140
86
Figure 31: An example of a bad fit showing a large ratio of R to r [60] .......................
87
Figure 32: An example of a bad fit showing a large ratio of D to d [60] .....................
87
Figure 33: Cylinder placement with respect to the LiDAR sensor with 5 correct cylinder
placements and one wrong (top left corner) [60].................................................
88
Figure 34: Manually placed markers by hand to identify true stems in ground LiDAR data
of the L ake M ize site............................................................................................
. 89
Figure 35: Comparison between the locations of stem centers hand labeled (red) to those
identified automatically by the stem center estimation algorithm (blue) given LiDAR
data of Lake Mize site as input. Simulated robot path is shown in pink .............. 90
Figure 36: Matching scheme between local maps M and Mi to global map S by finding
the rotation and translation param eters .................................................................
94
Figure 37: SAD correlation process using an input local map and a constellation map [63]
...................................................................................................................................
95
Figure 38: An example of the performance of GMM ICP using two datasets (red and
blue) that include outliers and jitter [68]...............................................................
96
Figure 39: Data matching flowchart showing the major data processing steps............ 97
Figure 40: ICP involves search space positioning based on last known robot geolocation
and systematic search that is comprised of linear and angular stepping................ 98
Figure 41: Result from running standard ICP showing skewed position estimate due
presence of one outlier injected to m ap M ..............................................................
104
Figure 42: Result from running robust ICP with the same datasets where the outlier was
automatically identified by the algorithm and removed .........................................
104
Figure 43: Geographical area of the test site at Lake Mize, FL, USA (Source: Google
E arth ).......................................................................................................................
10 8
Figure 44: Lake Mize test site dimensions and simulated rover path (Image source:
U S G S ) .....................................................................................................................
10 8
Figure 45: Simulated mobile robot path in red along with all stem centers identified by
the stem identification and center estimation algorithm.........................................
Page 13 of 140
111
Figure 46: Geoposition results for path 1 comparing true mobile robot position to
estim ates by the localization algorithm ...............................................................
113
Figure 47: Histogram of position error results as provided by the localization algorithm
for p ath I .................................................................................................................
1 13
Figure 48: Geoposition results for path 2 comparing true mobile robot position to
estim ates by the localization algorithm ...................................................................
1 14
Figure 49: Histogram of position error results as provided by the localization algorithm
fo r p a th 2 .................................................................................................................
1 14
Figure 50: Geoposition results for path 3 comparing true mobile robot position to
estimates by the localization algorithm...............................................................1
15
Figure 51: Histogram of position error results as provided by the localization algorithm
fo r p ath 3 .................................................................................................................
Figure 52: Geoposition results for path 4 comparing true mobile robot position to
estim ates by the localization algorithm ...................................................................
11 5
116
Figure 53: Histogram of position error results as provided by the localization algorithm
fo r p ath 4 .................................................................................................................
Figure 54: Geoposition results for path 5 comparing true mobile robot position to
estimates by the localization algorithm...............................................................
1 16
117
Figure 55: Histogram of position error results as provided by the localization algorithm
fo r p ath 5 .................................................................................................................
1 17
Figure 56: Geoposition results for path 6 comparing true mobile robot position to
estimates by the localization algorithm...............................................................1
18
Figure 57: Histogram of position error results as provided by the localization algorithm
fo r p ath 6 .................................................................................................................
1 18
Figure 58: Geoposition results for path 7 comparing true mobile robot position to
estim ates by the localization algorithm ...................................................................
119
Figure 59: Histogram of position error results as provided by the localization algorithm
fo r p a th 7 .................................................................................................................
Figure 60: Geoposition results for path 8 comparing true mobile robot position to
estim ates by the localization algorithm ...................................................................
Page 14 ol' 140
1 19
120
Figure 61: Histogram of position error results as provided by the localization algorithm
fo r p ath 8 .................................................................................................................
12 0
Figure 62: Geoposition results for path 9 comparing true mobile robot position to
estim ates by the localization algorithm ...................................................................
121
Figure 63: Histogram of position error results as provided by the localization algorithm
fo r p ath 9 .................................................................................................................
12 1
Figure 64: Geoposition results for path 10 comparing true mobile robot position to
estim ates by the localization algorithm ...................................................................
122
Figure 65: Histogram of position error results as provided by the localization algorithm
fo r p ath 10 ...............................................................................................................
12 2
Figure 66: Position error results including mean error, standard deviation and RMSF
produced by localization algorithm for all paths 1-10 ............................................
124
Figure 67: 2 parameter DOE results versus estimated position error (RMSE) for path 1
d ata (can o n ical case)...............................................................................................
12 5
Figure 68: Impact of number of detected stems per pose on RMSE for path I (canonical
c a se ) ........................................................................................................................
12 6
Figure 69: Impact of minimum stem radius on RMSE for path I (canonical case)....... 126
Figure 70: Unfiltered geoposition estimates by the localization algorithm for path 1 data
(ICP in blue) as compared to the filtered version using a DKF (KF in green) and the
true position data (G PS in red)................................................................................
127
Page 15 of 140
List of Tables
Table 1: Description of functional components of localization algorithm................... 50
Table 2: Descriptions of system level inputs and outputs of the localization algorithm.. 53
Table 3: C row n C lassifier Features...............................................................................
71
Table 4: DOE parameter space with 6 variables and ranges for each ..........................
73
Table 5: Eight neighborhood classification features for tessellated data...................... 83
Table 6: A robust algorithm for point-to-point matching of tree center maps in presence
o f o u tlie rs ................................................................................................................
10 1
Table 7: Properties of input data (3 aerial images and one LiDAR point cloud)........... 109
Table 8: Input parameters to localization algorithm and simulation program................ I11
Table 9: Accuracy results of localization algorithm for all test runs 1-10......................
Page 16 of 140
123
-Blank-
Pagc 17
o
140
Nomenclature
Abbreviations
ConOps
DEM
DKF
DOE
EEPROM
ERDC
FOV
GMM
GN&C
GPS
GPU
GUI
ICP
IMU
INS
LiDAR
MAV
MER
NIR
NSERC
RANSAC
RGB
RMS
RMSE
RTK
SAD
SDP
SIFT
SLAM
SNR
SURF
SVM
UAV
Concept of Operations
Digital Elevation Model
Discrete Kalman Filter
Design Of Experiments
Electrically Erasable Programmable Read Only Memory
Engineering Research Development Center
Field Of View
Gaussian Mixture Model
Guidance Navigation and Control
Global Positioning System
Graphics Processing Unit
Graphical User Interface
Iterative Closest Point
Inertial Measurement Unit
Inertial Navigation System
Light Detection And Ranging
Micro Air Vehicle
Mars Exploration Rover
Near Infra Red
National Science and Engineering Research Council
RANdom SAmple Consensus
Red Green Blue
Root Mean Squared
Root Mean Squared Error
Real Time Kinematic
Sum of Absolute Differences
Swiftest Descending Path
Scale Invariant Feature Translorm
Simultaneous Localization And Mapping
Signal to Noise Ratio
Speeded Up Robust Features
Support Vector Machine
Unmanned Aerial Vehicle
Page 18
of 140
UGV
USGS
UTM
WGS84
Unmanned Ground Vehicle
US Geological Survey
Universal Transverse Mercator
World Geodetic System
Symbols
Listed in order of appearance
M
XM
yM
Rv
Xr
yr
S
xs
Es
Rr
x
y
z
ch
cc
Xt
Yt
bh
brh
cr
gin
xin
Yin
g
b
t
N
G
Local map generated from ground LiDAR data
x-coordinate of local map M
y-coordinate of local map M
Mobile robot absolute horizontal location vector
Absolute location of rover along Easting
Absolute location of rover along Northing
Global map generated from overhead imagery
Coordinate of a point in global map S along Easting
Coordinate of a point in global map S along Northing
Distance error
Rotation matrix
Translation vector
x-coordinate of 3D profile of tree crown relative to st em base
y-coordinate of 3D profile of tree crown relative to st -m base
z-coordinate of 3D profile of tree crown relative to st(em base
Height of base of crown
Normalized radius of curvature
x-coordinate of apex of tree crown relative to stem ba se
y-coordinate of apex of tree crown relative to stem ba se
z-coordinate of apex of tree crown relative to stem ba se
Base height
Breast height
Radius of crown at base
Filtered green image
Column index of image
Row index of image
Green channel
Red channel
Blue channel
Green contrast threshold
Image following green channel extraction
Gaussian Blur Function
Page 19 of 140
ai
yi
Ne
d
Gaussian disk radius
Column index of Gaussian blur disk
Row index of Gaussian blur disk
Image following application of Gaussian blur function
Binary image
Input image
Euclidean distance transform
Input point in 3D space
Input point in 3D space
Watershed segmentation transform
Column address of connected pixel
Row address of connected pixel
Immediate neighbor
Diameter of crown object
e
Eccentricity
X,
R
Set of points comprising a stem
Diameter of cylinder primitive base
r
Radius of curvature of stem point cluster
P
Lowest point in stem
F
Feature
C
Centroid
C',
Classification cost
OM
Os
Frame origin of local map M
Frame origin of global map S
XSS
Easting coordinate of center of search space
YSS
Northing coordinate of center of search space
xOD
Distance along Easting measured by odometry
yOD
Distance along Northing measured by odometry
Rss
Rotation matrix of search space
tSS
Translation vector of search space
nySs
Number of rotation steps in angular search space
nss
Number of linear steps in linear search space
o
Angular step
dX
Linear step along Easting
dy
Linear step along Northing
MGEO
Local map in global frame Os
XGEO
Absolute Easting coordinates of local map M
yGEO
Absolute Northing coordinates of local map M
Xe.o/
Linear correction along Easting
j
N,
Nb
I
d
p
q
WS
xi
Page 20 of 140
Ps
Linear correction along Northing
State vector of simulated mobile robot
Pose
011(11
Maximum angular search space
iMOXs
Maximum number of iterations
Minimum crown diameter
Yu11
Xstate
diaminin
diammxax
Maximum crown diameter
Sm~in
Linear dimension of search space
Minimum number of stems per pose
enl,
Distance error threshold
x0
True Easting position of mobile robot
y"
True Northing position of mobile robot
Estimated Easting position of mobile robot
Estimated Northing position of mobile robot
1,,
Subscripts and Superscripts
(-
(ij) entry of a matrix
()
ith entry of a vector
Units
Units in this thesis are typically given in the SI-system (systeme international) and are
enclosed in square brackets, e.g. [m/s].
Page 21 of 140
-Blank-
Page 22
of 140
Chapter 1
Introduction
The Global Positioning System (GPS) has been extensively utilized for decades to
localize ground and airborne vehicles [1]. Vehicles with GPS receivers integrated into
their navigation systems and running standalone can localize instantly with sub-meter
accuracies [1]. Centimeter accuracy has been achieved using augmentation systems and
other online correction methods such as Real-Time Kinematic (RTK) [2]. Mobile robots
have utilized GPS extensively for outdoor path planning. Sequential traverse waypoints
with respect to a global reference system are programmed into the robot and then used
during motion for course comparison and correction using GPS.
Despite advances in GPS measurement methods that have resulted in drastic
improvements to localization accuracies, GPS can be rendered ineffective and hence
unusable under different scenarios, mainly:
* Attenuation of signals in forest environments by dense tree canopies for rovers
operating on the ground to the point where signals can be hardly received [3].
*
Multi-path of GPS signals in cluttered environments such as urban city centers
that result in significant positioning errors [3]. Known as the "urban canyon"
effect, facades of tall and large buildings reflect GPS signals multiple times
before reaching the robot's receiver, thus increasing pseudoranges and skewing
final position estimates.
*
Intentional GPS signal spoofing, dithering or denied access could render the
service inaccessible and potentially unreliable for continuous and real-time
localization purposes [3].
Due to the limitations noted above in the performance and reliability of GPS
based navigation, non-GPS based techniques have been utilized for robot geopositioning.
Dead reckoning frameworks that do not rely on GPS and use Inertial Measurement Units
(IMU) have been used extensively [4]. Based on a known reference position, attitudes
and accelerations of the robot in all three axes are integrated over time to produce
Page 23 of 140
positional estimates in real-time. However, due to the continuous drift in attitude
estimation by the IMU, this technique is prone to ever increasing errors in measured
position [4]. Such drifts, which theoretically could be unbounded, cause considerable
challenges for rovers that need to perform precision operations within close proximity to
waypoints of interest.
Non-GPS and non dead-reckoning techniques have been developed over the past
few years to address the limitations discussed above. Of note are Simultaneous
Localization and Mapping (SLAM) algorithms, which have been successfully utilized to
localize rovers in a variety of settings and scenarios [5]. SLAM focuses on building a
local map of landmarks as observed by a rover and using it to estimate the rover's local
position in an iterative way [5,6]. Landmarks are observed using external sensors that
rely on optical (i.e. visible camera or Light Detection and Ranging - LiDAR), radar or
sonar based means. Positions of landmarks and that of the rover are estimated a priori
and post priori lbr each pose and then iteratively refined [6]. Position errors could be
initially high but tend to converge as more landmarks are observed and as errors are
filtered. Of note are schemes that utilize visual odometry and scan matching whereby
successive frames at different poses are matched using least squares methods to
determine changes in motion [7]. Although these methods could be computationally
expensive, they have been successfully integrated into SLAM to provide real-time
position estimation. It is noted that the absolute global location of the starting position of
the rover needs to be known in order for subsequent poses to be georeferenced [6].
This thesis builds upon the concept of visual based scan matching by developing a
novel vision based algorithm that provides a non-iterative estimate of the global position
of a rover. The algorithm estimates robot position by comparing landmarks, in this case
tree stems observed on the ground, to tree stems detected in overhead georeferenced
imagery of the forest canopy. The method is envisaged to be used in two possible
scenarios: i) cooperative robotic exploration scheme whereby airborne images of tree
canopy captured by an Unmanned Aerial Vehicle (UAV) are matched to 3D maps of the
ground beneath acquired by a LiDAR sensor onboard a rover. ii) Solo robotic exploration
scheme whereby 3D maps of the forest are captured by a rover mounted LiDAR sensor
and compared to an overhead image library preloaded into the rover a priori. In either
scenario, only vision based data products are used to estimate the global position of a
rover without reliance on GPS. The proposed scheme is similar in concept to the
"kidnaped robot" problem where a robot is required to ascertain its position after being
placed in an unknown previously unexplored environment [8]. The presented method is
envisaged to be complementary in nature to traditional vision and map based navigation
frameworks where it could be invoked on-demand by the rover as the need arises. In
essence, we anticipate the algorithm to run in the background at all times as a backup that
kicks in whenever the primary GPS localization service is down or becomes unreliable.
Page 24 of 140
Alternatively, and to enable real-time redundancy in geoposition estimation, the method
could be used in tandem with the primary localization service.
1.1
Research Background and Motivation
The work described in this thesis can be categorized under vision based
localization and mapping, a field currently garnering lots of attention by roboticists
worldwide. In particular, absolute localization without the use of GPS is a new field of
study that is now being investigated. The topic is becoming more important considering
the different physical environments (e.g. forests and urban areas), and adversarial
conditions such as intentional denied access that make it challenging and sometimes
impossible to operate using GPS.
An interesting problem this thesis seeks out to solve is to investigate what
localization methods and/or ways that can be used to circumvent limitations of GPS
navigation. With limited access to radio-wave means of localization, inertial techniques
are usually considered. However, as mentioned earlier and as will be discussed later in
the thesis, inertial navigation is prone to drifts that are unbounded if corrections are not
continuously applied. Vision based localization becomes a feasible choice by offering
"spoof-proof' localization that only requires exteroceptive sensor inputs from visible
light camera and/or LiDAR sensors.
Localization and mapping are two intertwined concepts in the field of vision
based navigation. To build a map of its surroundings, a rover uses exteroceptive sensory
inputs to construct feature maps composed of the relative locations of objects. To put
objects in their respective coordinates accurately, the rover needs to know its location
first. To localize, the rover can determine its absolute or relative location based on
matching what it observes to what it expects to observe. This presents a "chicken and
egg" problem where in order to construct accurate maps of its surroundings; the rover
needs to know its location a priori. However, the rover needs to first construct maps of
objects around it in order to estimate its location, and vice versa. This particular problem
has been the focus of extensive research by the SLAM community and is considered to
be outside of the scope of this thesis. The work presented in this thesis is focused on the
localization aspect by decoupling it from SLAM and applying novel techniques to
determine geoposition through the use of exteroceptive data sources.
Figure 1 illustrates the top-level research problem of the thesis. Any static object
detected by a mobile robot on the ground can be used to construct a spatial map
composed of the horizontal locations of these objects relative to the robot. Theoretically
speaking, at least two features need to be observed in order for the mobile robot to
localize itself horizontally. Vertical localization (height) is not considered in this thesis.
Relative horizontal locations of objects are typically measured at their centroid for
Page 25 of 140
*
symmetric or pseudo symmetric objects such as tree trunks and crowns, or at discrete
edges at particular heights for corners of buildings in urban environments.
I-
Direction of Travel
4
Figure 1: Horizontal locations of features (7-point stars) as observed by a LiDAR sensor onboard a mobile robot
on the ground
Using exteroceptive sensor inputs from camera, sonar and LiDAR of the
immediate environment, a rover can determine its position by constructing spatial feature
maps that can be compared to maps previously collected of the same area. Highresolution orthoimagery of the exploration site provide an excellent ground truth upon
which local maps generated by the rover on the ground can be compared against. As
illustrated in Figure 1, features detected relative to the rover on the ground will have the
same geometric relationship between them regardless of the origin or type of sensor used,
given reasonable accuracies. The spatial invariance property of these features is exploited
to construct maps that can be compared to preloaded ground truth data of the exploration
site. This application of multi-sensor fusion is novel within the context of absolute rover
localization.
1.2
Thesis Problem Formulation
1.2.1 Thesis Objectives
The first objective of this thesis is to advance the field of autonomous rover
navigation by developing a vision based localization algorithm that relies on
Page 26 of 140
exteroceptive sensor data to match feature maps collected on the ground to georeferenced
ground truth of the exploration site. In other words, given a local feature map constructed
from on board vehicle sensors, M= [(x"AI,yx"f), (x-,V 2),..., (x'',y',)],
where
xly
and y
are horizontal coordinates of features relative to the rover, the algorithm attempts to
estimate the rover's absolute horizontal location, R, = [x',y'], by matching M to another
feature map S = [(xs s I)
2J)2)
s
-, (s ss7)], where xsi and y S are the absolute
horizontal coordinates of ground truth features observed by other means (e.g. aerial,
satellite). Through a least squares approach, the algorithm can generate estimates of the
absolute location of the rover depending on the matching result of both datasets. The
theoretical questions to be answered by this thesis are:
. .
1. What kind of features and feature properties can be used to invariantly specify
feature locations, within the context of an outdoor environment, as they are
detected by the rover on the ground (local map M)?
2. What kind of data, features and feature properties can be used to invariantly
specify absolute positions (world coordinates) of features for use as ground
truths (global map S)?
3. How to register, in 2D, local maps M to constellation map S while achieving
absolute accuracies on the order of the individual accuracies of the data inputs
used E, = [El, E2,..., Ek]? Where E; are the individual metric accuracies of the
inputs.
A second objective of the thesis is to demonstrate the efficacy and properties of
the localization algorithm in the context of finding the absolute position of a single rover
operating in a dense forest environment. In particular, the thesis attempts to answer the
following applied questions:
1. What are the specific parameters that mostly affect the localization accuracy,
its convergence and repeatability?
2. What is the maximum achievable accuracy performance of the localization
algorithm?
3. What are the functional and performance constraints of the localization
algorithm and what are some future steps that can be taken to improve
performance, if any?
A third objective of the thesis is to implement the localization algorithm and
experimentally demonstrate its ability to provide correct and accurate absolute position
estimates using real-world data:
1. Implement the vision based localization algorithm in a common and userfriendly technical computing environment, and thoroughly test it with various
real-world sensory datasets of different properties.
Page 27 of 140
2. Validate the resulting performance data with theoretical estimates given
properties of the exteroceptive sensor inputs.
3. Obtain insights on the maximum level of accuracy attainable and expected
real-world performance given experimental field data.
Finally, the fourth objective of the thesis is to study and specify typical Concepts
of Operations (ConOps) that describe how Unmanned Ground Vehicles (UGV), i.e.
rovers, are envisioned to operate using the vision based localization algorithm. In
particular, the following are systems questions that will be addressed:
1. What are the important system-level requirements describing the
functionality, operability and performance of rover localization in outdoor
environments?
2. What is the system architecture and use context envisioned for the localization
algorithm? Who are the primary stakeholders, beneficiaries and agents of such
a system?
3. What is the optimal envisioned operations concept and how can the algorithm
be integrated into a rover system? What inputs/outputs are expected and what
internal processes are employed to provide the anticipated results?
1.2.2 Mathematical Problem Definition
The horizontal map that comprises the global horizontal coordinates of tree stems
in aerial imagery is described by:
S
=
[(x, yr), (x,y), ... , (xi, ys)],
xs, ys E R
i = 1, 2 . . , m
Where xs; and Vsi are the absolute horizontal coordinates of the centroids (stems)
of tree crowns detected in overhead imagery with respect to a worldwide reference frame,
e.g. WGS84. mi represents the number of tree stems in the set. The set S can be
considered as a constellation of tree crowns, in analogy to star constellations, which can
be used for navigation. In this case, tree crowns are used instead of stars for matching
purposes.
Local feature maps comprising the relative horizontal coordinates of tree stems
detected from onboard LiDAR data can be described by:
M=[(xm, yM), (xM, ym), . . , (xM,, ym")],
xm~ym
E R
i = 1, 2 . . , n
Where x 1 ; and v";' are horizontal coordinates of tree stems relative to the rover at
each pose. n represents the number of tree stems observed at each pose.
Page 28 ol 140
The localization problem can be described as follows: Given a constellation set S
and a relative set M both comprising horizontal locations of stems, how can set M be
accurately matched with set S in order to determine the absolute horizontal location of the
rover?
Mathematically, the above can be solved using a constrained non-linear
optimization scheme such that:
2
arg min Es,GEj2 =
IIRXM
+ t
-
SI
Such that R > 0, t > 0
Where R and i are the horizontal rotation matrix and translation vector needed to
transform M to match S. E is the square of the Euclidean distance error between the
transformed M and constellation set S, i.e. the square of the norm of the error vector E. In
theory, E should be proportional to ES:
E oc Es, Es = [El, E2 ..., E]
i = 1,2, . . , k
Where k is the number of local sets M, or number of poses, assuming one set M
per pose. The transformed set M is then used to retrace the origin of observations for each
stem to estimate the absolute location of the rover.
1.2.3 Constraint on Thesis Scope
The scope of this thesis is limited to finding the absolute location of a rover
within a constrained search space centered on the best-known position of the rover. This
scheme is in contrast to performing a brute force global search that is not limited to a
particular geographic area. Considering real-time applications of this algorithm, a global
search scheme would not be time efficient, or necessarily accurate enough. The
implication of such a constraint would require the algorithm to expect a constant stream
of estimated rover poses as determined by its onboard Guidance Navigation and Control
(GN&C) system. The algorithm would therefore take horizontal position estimates from
the GN&C system, determine the absolute position of the rover using its frameworks and
then feed those location estimates post priori to the GN&C system, which in turn
employs its own Kalman Filter.
The scenario described above is considered more realistic and is in line with
current state-of-the-art in rover navigation and localization.
1.3
Previous Work
This section gives a short overview of the scientific literature relevant to the
development and validation of the localization algorithm. The literature search begins
Page 29 of 140
with reviewing papers on the fundamentals of rover localization using different
technologies and techniques. The current state-of-the-art in localization and mapping is
discussed along with initial work by other researchers in the field of vision based
localization. A specific part of this thesis is based on work performed by past members of
the Robotic Mobility Group at MIT specifically in the area of tree stem center estimation
from LiDAR data. Their work is also introduced in this section and described in more
detail in the main body of the thesis.
Early terrestrial mobile robot localization techniques employed wheel odometry,
inclinometers and compass sensors to determine the robot's position relative to a known
starting point [9]. Wheel odometry sensors measure the number of wheel rotations to
determine distance traveled while inclinometers measure the direction of the gravity
vector relative to the body frame of the vehicle, thus providing a direct measurement of
the pitch and roll of the vehicle. A compass is used to measure magnetic heading of the
mobile robot with respect to the body frame. Common localization techniques
incorporate all of the above measurements as part of a vehicle model and integrate
measurements to determine the robot's displacement as a function of time [9,10]. This
technique is known as dead reckoning [10]. However, due to wheel slip and azimuthal
angle drift in wheel odometers, as well as temperature drifts in inclinometer and magnetic
compass readings, errors from dead reckoning increase with time and are theoretically
unbounded [10]. For example, during the Mars Exploration Rover (MER) mission, the
Opportunity rover wheel odometry sensor once underestimated a 19 m traverse by about
8% [11].
More sophisticated technologies such as Inertial Navigation Systems (INS) have
been successfully used to localize rovers with moderate accuracies [12]. Although
considered to belong to the class of dead reckoning techniques, INS technology has
garnered immense interest during the second half of the last century with many sensors
deployed onboard vehicles of all kinds and applications. INS technology typically
employs 3-axis gyroscopes integrated with 3-axis accelerometers to form an Inertial
Measurement Unit (IMU) [13]. The gyroscope assembly measures vehicle attitude along
roll, pitch and yaw with respect to a known body frame, while the accelerometer
assembly measures the accelerations along all three axes with respect to a known body
frame [13]. To determine location with respect to a starting position, the vehicle's state
model comprised of the attitude and acceleration data is integrated over time. This
technique still suffers from error drifts that are theoretically unbounded. However, the
technique provides much higher accuracy than inclinometers and compasses because of
the low drift rate and low noise of the 3-axis gyroscope and the 3-axis accelerometers
[13]. Regardless of what sensor technology is used, the starting position of the vehicle
needs to be known in order for subsequent data to be referenced.
Page 30 of
140
The satellite based GPS system has been extensively used for terrestrial
navigation and localization applications. GPS technology relies on a satellite
constellation that provides global coverage of the earth (US Navstar system) [14]. To
localize, a vehicle carrying a GPS receiver requires at least 4 satellites overhead with
direct line-of-sight [14]. Each satellite sends a carrier signal with timing information that
is decoded by the receiver. By comparing the time when the signal was sent to the time it
was received, a receiver can determine the distance traveled through the speed of light
relationship. With satellite-to-receiver distance information known, and by using the
ephemeris of satellites overhead, a receiver can triangulate its 3D position with accuracy
limited to the timing error between the clock on the satellite and that used by the receiver
[15].
Real-time GPS accuracies, with no corrections applied, have been reported on the
order of 5 m (spherical) [16]. For higher accuracy applications, techniques such as
Differential GPS (D-GPS) are used to attain accuracies at the sub-meter and decimeter
level [16]. Since the GPS signal is dithered intentionally for commercial use (C/A mode),
non-corrected GPS position readings arc inaccurate. D-GPS relies on using real-time
corrections calculated by static stations on the ground, thus the term "differential". To
attain cm-level accuracy, raw GPS position measurements are post-processed using the
logged corrections [17]. Other techniques such as Real-Time Kinematic (RTK) provide
cm-level accuracy by correlating the incoming carrier signal over time. This technique
has the advantage of lower ambiguity interval, which is on the order of 19 cm for the L 1
carrier [18].
In general, GPS is considered a universal localization method that is reliable
under normal conditions. However, GPS signals can be easily attenuated in forest and
urban environments. In particular, dense forest canopies have the potential of completely
blocking GPS signals from receivers operating on the ground [19]. In similar fashion,
GPS signals in dense urban centers (city centers with dense vertical developments) tend
to be either attenuated or bounced around and delayed in a phenomenon known as multipath [20]. This "urban canyon" effect has been investigated and several solutions were
implemented. Of note are solutions that integrate INS and GPS to form full positionorientation sensor systems. For example, systems offered by companies such as
Trimble/Applanix and Novatel rely on GPS as long as signal coverage is reliable and
automatically switch to INS based positioning whenever GPS becomes unreliable [21].
Although effective, these systems are sophisticated, bulky and expensive. It is noted that
the localization accuracy of this method will degrade in an unbounded fashion in
scenarios where there are long GPS outages. This is due to the method's reliance on INS
dead reckoning.
Techniques that are different from INS and GPS rely on visual-based sensors and
algorithms e.g. visual odometry and scan matching. Visual odometry relies primarily on
Page 31 of 140
passive camera sensors that are integrated into a mobile robot [22]. Cameras acquire
successive images of the terrain and other features surrounding the mobile robot. Visual
odometry algorithms track features across successive pairs of images to measure
variations in the pose of the robot [22]. Features are typically extracted using various
image processing algorithms such as Speeded Up Robust Features (SURF) or Scale
Invariant Feature Transforms (SIFT) [23,24]. Visual odometry has been extensively used
in terrestrial and space based applications e.g. MER and Curiosity rovers on Mars [25].
The algorithm is fully automated and can be operated in real-time at the expense of
moderate processing speeds, which may require a dedicated Graphics Processing Unit
(GPU) [25]. Although effective lor relative navigation, visual odometry still requires
continuous tie-in to known global benchmarks in order to provide global localization
measurements. In addition, visual odometry is prone to unbounded error growth due to
the dependence of future poses on previous position estimates [26]. Due to these
characteristics, visual odometry is also classified as a dead reckoning technique.
Scan matching is another vision based navigation method that relies on matching
3D point cloud data captured by laser scanning sensors onboard a mobile robot [27].
Unlike visual odometry, scan matching typically relies on point-to-point matching
algorithms such as Iterative Closest Point (ICP) [27]. Pairs of scans taken at successive
poses are provided to ICP to determine the translation and rotation between the frames.
After an alignment is found, a change in the location of the robot can be determined.
Similar to visual odometry, scan matching requires moderate processing power in realtime and therefore could require a dedicated GPU. By matching successive frames, scan
matching is also prone to unbounded errors that grow from frame to frame. The algorithm
still requires the absolute starting locations of the mobile robot in order to georeference
successive pose estimates. An example is that of Song et al (2012) who employs scan
matching to the problem of determining the relative pose of a mobile robot in dense
forest environments [28]. Tree trunks scanned by an onboard 3D laser scanner are
processed and abstracted to form an occupancy map of trunks observed at a particular
pose. Successive pairs of maps are compared and correspondences are determined based
on the observed 3D point patterns. Levinson et al (2011) applies scan matching to a
driverless car application by utilizing a particle filter that correlates successive pairs of
360' LiDAR scans to estimate the vehicle pose [29]. The algorithm reduces the
complexity of LiDAR data by filtering out non-ground points and retain road data. Road
lane stripes and other white markers on the road are extracted and used for matching
between successive scans at different poses.
A different class of localization methods does not rely on distance traversed (dead
reckoning) or timing signals (GPS), but instead matches local maps of the immediate
environment around the robot to global maps of the area the robot is exploring. Maps
here imply 2D imagery, 3D point clouds or occupancy grids. The techniques most
relevant to this description follow the procedure of feature detection, extraction and
Page 32 of 140
matching. In this process, features are first detected, delineated and extracted from the
global and local maps using image-processing techniques. Local feature maps are then
matched to global feature maps to find correspondences. Once the correspondences are
found, the features are aligned in order to produce an estimate of the location of the
mobile robot with respect to a global rieference system.
Various researchers have tackled the problem of global map matching and have
developed algorithms specifically for this purpose. Planitz et al (2005) and Mian et al
(2005) evaluated several algorithms that automatically determine correspondences
between 3D point clouds [30,31]. However, researchers were not restricted to 3D
applications. Mikolajczyk and Schmid (2005) focused on finding feature descriptors in
2D images [32]. Se et al (2005) utilizes a Hough transform method to extract visual
landmarks in 2D frames and a RANdom SAmple Consensus (RANSAC) scheme to
compare and match landmarks in local frames to a database map. The method is effective
but expects the global map to be free of outliers or "false positives" in order for pair-wise
alignment and matching to perform reliably [33]. Zhang (1994) examined use of ICP for
surface alignments of 3D maps in several formats, e.g. Digital Elevation Model (DEM)
and point clouds [34]. Early work by Hayashi and Dean (1989) aimed at extracting,
comparing and matching data from robot based laser scanners to a high resolution global
DEM. Their pioneering work was interesting but their simulations yielded poor accuracy
because of the large difference in resolutions between the local and global maps used
[35]. Stein and Medioni (1995) worked on comparing global aerial maps to local rover
maps by comparing skylines captured by a camera onboard a mobile robot to a satellite
based topographic map. This method required an area with dense topographic features
such as mountainous landscapes and has resulted in accuracies on the order of a hundred
meters [36]. Behar et al (2005) explored using high-resolution aerial images for use as
global maps instead of coarse orbital imagery. However, his method still relied on dense
topographic features for feature detection, extraction and matching [37].
Ramalingam et al (2010) utilized a similar approach for use in urban canyons
where skyline imagery of buildings are processed to extract features such as edges of
buildings, which are compared to aerial or satellite maps of the same area [38].
Accuracies of 2 m were achieved but the algorithm has several limitations mainly that
skylines of buildings are expected to be close to the camera. Another limitation is the fact
that nighttime operation is not possible due to use of visible light cameras. Majdik et al
(2013) utilize a similar approach for localizing a Micro Aerial Vehicle (MAV) looking
down onto buildings [39]. Features detected in the downward looking image, such as
building edges, are compared using a SIFT algorithm to a global feature database
extracted from Google Street View. By matching features in the horizontal plane, the
vehicle is capable of determining its global horizontal location. Kretzschmar et al (2009)
utilize a similar approach that is augmented with data from open-source photo websites
such as Flickr [40]. Carle et al (2010) build upon prior work by utilizing long-range 3D
Page 33 of 140
laser scanner data from a mobile robot that is compared to preloaded DEMs. The position
estimates are enhanced by incorporating visual odometry and sun sensor measurements.
Accuracies on the order of 50 m were reported [41]. Although accurate when compared
to results from earlier work, the algorithm relies on external sensor data to drastically
improve accuracy, which is not an optimal or reliable scheme in uncooperative
environments.
A different method is Simultaneous Localization and Mapping (SLAM), which
has been successfully utilized to localize rovers in a variety of settings and scenarios
[42,43]. SLAM focuses on building a local map of landmarks as observed by a rover to
estimate its local position in an iterative way [42]. Landmarks are observed using external
sensors that rely on optical (i.e. visible camera or LiDAR), radar or sonar based means.
Positions of landmarks and that of the rover are estimated a priori and post priori for
each pose and then iteratively refined. Position errors could be initially high but tend to
converge as more landmarks are observed and as errors are filtered. SLAM can be
integrated into schemes that utilize visual odometry and scan matching to provide realtime position estimation. Although such techniques do not require a priori knowledge of
the locations of landmarks or that of the rover, the global location of the starting position
of the rover needs to be known in order for subsequent poses to be georeferenced. Iser et
al (2010) implements a particular instantiation of a SLAM algorithm to globally localize
a rover [441. At each pose, a local map of the surroundings is produced using onboard
sensors. Iser's work is sensor agnostic and only requires at a minimum a 2D occupancy
map of features. Global localization is achieved by invoking RANSAC to perform coarse
matching between local maps and a global map of the area. The difference in this
implementation is that Iser implemented RANSAC as part of a SLAM framework where
matches between local and global maps are refined iteratively as maps get built.
In summary, previous work has focused on globally localizing mobile robots
using a variety of sensors, algorithms and methods. Considering map-based localization
methods, the majority of researchers focused on either matching successive data frames
to determine changes in pose, or by matching unobscured data observed locally by the
mobile robot to global maps of the exploration area. Localizing robots in obscured
environments such as dense forest grounds is a problem that has just recently been
garnering interest by researchers. Considering a GPS denied environment, map-based
localization becomes challenging in forest environments simply because features
observed on the ground by the mobile robot may not necessarily be observable in aerial
or orbital based imagery. Therefore, standard map-based localization algorithms will be
challenged in providing reliable data in uncooperative environments.
The research undertaken in this thesis tackles the specific scenario of localizing a
mobile robot in dense forests using vision based sensors and matching methods as
discussed in the following sections.
Page 34
of'140
1.4
Thesis Overview
A thesis roadmap is shown in Figure 2. The purpose of this roadmap is to
logically organize the thesis research, which solves the mathematical problem definition
given in Section 1.2. The flow diagram in Figure 2 comprises the development of the
vision based localization framework. Specific chapters of the thesis addressing particular
details of the modules shown in Figure 2 are specified in bold and in square brackets.
The methodology adopted to undertake the research and develop this thesis has
followed the flow shown in Figure 2. Initial work comprised defining the operations
concept of a mobile robot in a dense forest. In particular, the use context was investigated
in order to determine likely scenarios and applications that would utilize vision based
localization. Chapter 2 takes a detailed look at particular mission scenarios that would
make use of vision based localization and how, in particular, a kidnapped robot scenario
can be addressed. The work concludes by specifying a set of operations and functional
requirements that detail scenarios, operations aspects and performance parameters for
mobile robot localization in dense forest environments.
Based on the operations concepts and requirements established, a top-level
architecture of the vision based localization framework was defined. The architecture
seeks to capture the important functional building blocks required to perform localization
of a mobile robot. This includes defining the subsystems (modules) that are to be
integrated as part of the overall framework. In addition, inputs and outputs at the toplevel of the framework are defined in order to clarify how the framework can be
integrated to mobile robots and their Guidance, Navigation and Control (GN&C)
systems. Chapter 2 captures the architectural details of the algorithm and provides
specific information regarding compatibility of this algorithm to current navigation
workflows.
Following the definition of the mission concepts and requirements, detailed
design and implementation of the various components and subsystems of the localization
framework were considered. Starting with Chapter 3, detailed designs of each of the
three important sub-systems of the vision based localization algorithm are given. The
chapter tackles the automatic tree crown identification, delineation and extraction task.
This subsystem is responsible for estimating tree crown centers, and hence tree stems,
from aerial or orbital imagery. Various image processing algorithms are implemented and
explained, most notably, the watershed segmentation algorithm that enables unambiguous
delineation of tree crowns. The output from this subsystem is a global map (constellation)
comprised of the detected tree stems that are used by the matching algorithm.
Page 35 of 140
Vision-Based Localization Algorithm Operations Concept and Requirements [CH 2]
Vision-Based Localization Algorithm Architecture [CH 2]
Tree Crown Identification, Delineation and Center Estimation (Overhead Image) [CH 4]
Arthihot
Aerial
Orthophoto
Orthophoto
Projection
Green Extraction
& Denoising
Watershed
Segmentation
Binary Image
Creation
8-connected
OjctnEctd
& entrin
Tree Stem Identification, Extraction and Center Estimation (Rover LiDAR) [CH 5)
LiDAR
Data Per
Pose
Ground
Height Filter
SVM Classifier
Breast Height
Stem Points
Least Squares
Primitive Fitting
Matching of Tree Centers from Overhead Imagery and Rover LiDAR Data [CH 6)
Stem Center
Centroiding
Ground Stems
Stem
Set M
Constellation Set
Establish Search Space Around Best Known Position
Invoke ICP Variant
Rover
EPose
Estimate
Calculate Rover Position Based on Match Results
Rover Absolute Location
Estimate [CH 7]
Figure 2: Overall thesis development roadmap (Chapters in bold square brackets)
Generation of the local map for matching purposes at each pose is the focus of
Chapter 4. The development of this subsystem borrowed significantly from prior work
by Matt McDaniel of the Robotic Mobility Group at MIT. The algorithm takes as input
3D point clouds of the forest grounds around the mobile robot and processes data to
extract individual tree stems and their relative horizontal locations. The local map
generated in this step is used for subsequent matching with the global map for
geopositioning purposes. Although discussed as if this component was developed after
the tree-crown delineation algorithm, it was actually modified from its original source
(Matt McDaniel) to be integrated into the framework discussed in this thesis.
The last component of the vision based algorithm is the matching of locally
generated maps to global maps of the area explored by the rover. A variant of the ICP
algorithm performs matching and alignment of locally generated maps of tree stems
around the rover to global stem maps estimated from overhead imagery. The main
difference in the ICP implementation in this thesis is its improved robustness in handling
outliers (false positive stems identified in ground LiDAR data). The method employs an
exhaustive search approach that systematically eliminates outliers by minimizing
resultant position error between the best known rover position and that estimated by the
algorithm. Chapter 5 details the design, implementation and inner workings of the
matching algorithm as well as its constraints and limitations.
Page 36 of 140
After implementing the localization framework, several tests were undertaken
using real-world data to validate and verify the performance and functionality of the
localization framework. Real-world data of a dense forest near Lake Mize in Florida,
USA, was used in different test regimens to assess performance. Data from the Lake
Mize site consisted of high-resolution 3D LiDAR scans of the forest grounds coupled
with open-source high-resolution aerial imagery of the same area. Chapter 6 details the
data products used, tests regime, processing workflow and localization results. To
characterize accuracy, position error results from the vision based localization algorithm
were compared to estimated true measurements (ground truth).
Chapter 7 provides concluding remarks on the overall thesis project and the
achieved results. Conclusions on the applicability and usefulness of the algorithm are
discussed. Potential future work is discussed and mentions emerging uses for autonomous
ground vehicle localization in dense urban environments.
Pagc 37 of 140
-Blank-
Pagc 38 of' 140
Chapter 2
Mission Definition and Architecture of
Localization Algorithm
This chapter defines the mission and the top-level architecture of the localization
algorithm. Candidate operational scenarios are also discussed within the context of
navigating a mobile robot in a forest environment without access to GPS.
2.1
Mission Context
The main goal of the localization framework is to provide reliable and accurate
horizontal geoposition estimates of a mobile robot while traversing an unknown area in a
forest environment. The localization framework is envisaged as part of two possible
scenarios:
1. Cooperative _robotic exploration scheme: Airborne or orbital images of the tree
canopy are matched to 3D maps of the ground acquired by LiDAR sensors
onboard a mobile robot. Overhead imagery acquired in real-time is transmitted
from the overhead platform to the mobile robot as required. Overhead data and
ground LiDAR data are processed online onboard the mobile robot.
2. Solo robotic exploration scheme: 3D maps of the forest grounds are captured by a
LiDAR sensor onboard a mobile robot and compared to an overhead aerial or
orbital map loaded into the rover a priori. This scheme is similar in concept to the
"kidnaped robot" problem where a robot is required to ascertain its position after
being placed in a foreign unexplored environment.
The localization framework is envisaged to be complementary to traditional GPS
and map based navigation frameworks where it could be invoked on-demand by the
mobile robot as the need arises. In essence, we anticipate that the algorithm is run in the
background at all times as a backup whenever the primary GPS localization service is
down or becomes unreliable. Alternatively, and to enable real-time redundancy in
Page 39 of 140
geoposition estimation, the method could be used in tandem with the primary localization
service. To simplify and focus the research effort, the solo robotic exploration scheme
(scenario 2) was considered as the primary scenario upon which the simulations, data
acquisition and algorithm development were all based on.
To provide a detailed definition of typical missions and scenarios considered for
use with the localization algorithm, a rundown of a hypothetical operational scenario is
given. Figure 3 illustrates a typical operational scenario with examples of typical sensors
used. A mobile robot is assumed to be operating solo and is tasked to explore a particular
area of interest. The mobile robot is initially assumed to be driving autonomously or via
teleoperation (human in-the-loop) in a forest area that includes heavy tree canopy and
underbrush of various densities and proportions. The mobile robot is anticipated to
initially use GPS for localization prior to entering the dense forest area. Once the rover
enters the heavy forest environment, GPS signals become inaccessible or unreliable due
to attenuation by the forest canopy, at which point the localization method is used for the
duration of the GPS blackout.
Mapping
GPS Satellite
Orthophoto (Aerial or Satelte)
o
00
OQ
Mobile Robot
Figure 3: Envisioned
operational
scenario
Page 40 of 140
of a
solo
mobile
robot
Satellite
The vision based localization method requires two types of input data. The first
input is an overhead image of the exploration site that is used to generate the ground truth
map S. The overhead image can be acquired either by satellite or aerial means and needs
to be orthorectified and georeferenced. The mission scenario assumes that single or
multiple high-resolution orthoimages can be preloaded into the mobile robot prior to
mission execution. The imagery can be processed a priorior in real-time at every pose by
the onboard navigation computer. Processing of aerial imagery involves delineation of
individual tree crowns followed by tree stem center location estimation as will be
discussed in subsequent sections. The second input are 3D scans of the environment
surrounding the mobile robot by onboard Light Detection and Ranging (LiDAR) sensors.
The mobile robot is expected to utilize a LiDAR sensor with 40' x 3600 horizontal FieldOf-View (FOV) to sufficiently image the geometry of tree stems surrounding the rover.
The reason why this particular FOV was selected is explained below. Configuration of
the sensors and required characteristics are discussed in Section 2.2.
In the case when GPS is unreliable, the mobile robot's GN&C subsystem is
expected to automatically invoke the vision based localization algorithm and incorporate
its location estimations. The following discuss aspects related to the overall mission and
operation of the localization method.
Environment
The localization algorithm developed in this thesis is targeted at addressing the
limitations of the state-of-the-art in providing reliable and accurate horizontal geoposition
estimates of mobile robots in forest environments. Attenuation of the GPS L-band carrier,
which carries time code data for pseudorange determination, is significant in forests [45].
Wright et al (2008) measured up to 50 dB attenuation in the Signal to Noise Ratio (SNR)
of received GPS signals under a dense forest canopy compared to a clear line of sight
receiver [45]. This level of attenuation renders GPS unusable in forests. The localization
algorithm developed in this thesis is focused on addressing such limitations of GPS.
The algorithm is not constrained to a particular geographic zone. The complete
Universal Traverse Mercator (UTM) system with its 60 zones bounds the operational
envelope of the localization algorithm. Features and types of trees that can be processed
by the localization algorithm are strictly the following:
1. Coniferous trees in all seasons.
2.
Deciduous trees from spring to beginning of autumn.
It is important to note that non-green objects cannot be processed by the algorithm
because it relies on changes in intensity gradients between the edges of crowns and the
terrain beneath. An example of an overhead image of a dense coniferous forest is shown
in Figure 4.
Page 41 of 140
)
Figure 4: Example of dense coniferous forest during summer season (Harvard Arboretum, Cambridge, MA,
from Google Earth
Characteristics pertaining to the configuration and properties of expected forests
are listed below:
1. Shades of green: To differentiate between canopy and other objects such as
nearby bushes and dirt, different levels (shades) of green are detected and
encoded prior to crown delineation. The forest area will need to be comprised of
diverse shades of green to facilitate crown differentiation and delineation. As
detailed in Chapter 3, the tree crown delineation and center estimation algorithm
relies on the luminance relationship to extract the green channel from an overhead
image.
2. Tree placement: In order to ensure successful geopositioning of the mobile robot
using the localization framework, trees need to be located in a random fashion,
i.e. naturally grown such that they don't form a regular grid. This is important
when considering that matching between overhead and local maps is based on
geometry. A regular grid of trees would lead to many solutions to the matching
problem, dramatically increasing the likelihood of a false estimate. A formal
treatment of this aspect is given in Chapter 3.
*
3. Tree height and stem diameter: The minimum stem height would need to be
approximately 2 m from ground in order to enable clear and accurate
identification of the stem diameter. Mature coniferous trees can reach heights
upwards of 20 m [46], therefore, there is no foreseen impact of the 2 m limit on
the applicability of the algorithm. As detailed in Chapter 4, manual examination
has empirically shown that a threshold of 2 m is adequate for our particular
http://www.google.com/earth/
Page 42 of 140
research site. This parameter and others discussed in subsequent chapters are
treated as part of a heuristic parameter set customized according to their particular
datasets. In similar fashion, there is no minimum or maximum limit by design on
acceptable diameters of tree stems. The algorithm can work with all sizes of stem
diameters. Nevertheless, there is a range of tree stem diameters that needs to be
set in order to discard abnormally large diameter estimates coming from spurious
false positives. This is discussed in more detail in Chapter 4.
Mobile Robot
The mission concept does not limit the use of any type of mobile robot. Tracked
and wheeled robots can be utilized because the concept of operations is agnostic to the
locomotion system used. The following describes aspects related to the mobile robot
within the context of the solo robotic mission scenario.
1. Locomotion: The mobile robot will employ its own means for locomotion.
2. Sensors: The mobile robot will at least employ a single LiDAR sensor to image its
surroundings in 3D. The LiDAR is mounted on a mast on top of the robot. The
LiDAR is expected to have an intrinsic wide FOV or integrated with a pan and tilt
mechanism to augment a smaller FOV. Figure 5 illustrates the configuration of
the mobile robot and its onboard sensor.
-
-
-------
40-......
11M
Mobile Robot
3m
-..............
M..
3m
Figure 5: Sensor and mobile robot mission configuration showing the anticipated minimum working distances
and FOV
Page 43 of 140
The typical configuration is comprised of a LiDAR sensor with a panoramic FOV
of at least 400 x 360' (horizontal x vertical) placed on a mast with a height of I
m. With an assumed minimum distance of 3 m for the range capability of the
LiDAR, the FOV and mast height combination enable the LiDAR to image a 3 m
tall tree stem at a distance of 3 m. It is noted that mature coniferous trees can
typically reach heights of 20 m. Therefore, acquiring 3D images that contain at
least 3 m of a stem would give 50% margin compared to the stem height threshold
(2 m) [46]. It is noted that more than 3 m of stem height can be achieved by a
larger FOV or by augmenting a LiDAR sensor with a pan/tilt mechanism. In our
case for example, the LiDAR sensor is a Leica Scan Station with a FOV of 270' x
3600 [47]. This large FOV allowed us to capture the whole tree from base to
crown. The mission configuration analyzed in this thesis assumes a baseline set of
minimum attainable specifications of off-the-shelf sensors.
3. GN&C: The mission scenario assumes two types of control schemes for the
mobile robot:
a.
Teleoperation: A human operator is responsible for controlling the mobile
robot and monitoring its behavior. The operator is assumed to be viewing
a real-time feed from sensors onboard the mobile robot such as the LiDAR
sensor, cameras and health monitoring devices. Attitude control of the
mobile robot is achieved through a joystick device. The operator manually
invokes the localization framework and interprets results before acting on
the data.
b. Autonomous: The mobile robot automatically navigates itself using the
localization framework. The mobile robot is expected to invoke, process
and act on the geoposition estimates without any intervention from a
human operator. The localization framework is therefore tightly integrated
to the GN&C system (closed-loop).
The localization algorithm's executable program is assumed to reside in a nonvolatile memory of the GN&C computer such as Electrically Erasable
Programmable Read-Only Memory (EEPROM). The executable is loaded and
invoked online whenever the GN&C computer is booted. The GN&C system is
expected to have the logic to switch from GPS positioning to the localization
framework whenever GPS coverage is lost or readings become unreliable.
4. Power: The mobile robot is expected to incorporate and use its own power source
that would most likely be a battery.
Section 2.5 provides details on the architecture of the localization framework and
how it interfaces with the various subsystems.
Page 44 of 140
2.2
Concept of Operations
The concept of operations discussed in this section considers the different control
scenarios along with the interactions and interfaces required to achieve localization using
the developed algorithm. The following sequence of operations summarizes the steps
required to obtain a geoposition estimate of the mobile robot using the localization
framework. An illustration of the flow of steps is given in Figure 6.
1. Whenever the GPS service is down and the mobile robot is required to obtain a
geoposition estimate, the rover executes a LiDAR scan of the surrounding area at
a particular pose.
2. LiDAR data is processed whereby cylinder primitives are fit in a least squares
sense to point clusters pertaining to tree stems. Fitted data clusters are labeled as
tree sterns and their geometric centroid is calculated. Tree stem centroid locations
relative to the mobile robot are subsequently used to create a map of the relative
horizontal locations of tree stem centers.
3. The overhead georeferenced image of the forest canopy above the mobile robot is
processed whereby crowns are delineated and their geometric centroids are
calculated. An absolute constellation map is subsequently generated that contains
the absolute horizontal locations of tree stem centers. This step can be executed
once for a large overhead image before the mobile robot commences its mission,
or can be executed at each pose. The former scenario is the one assumed in this
thesis.
4. Maps of tree stem centers estimated from LiDAR (map M) are then compared to
the absolute constellation map generated from overhead imagery (map S). A
robust ICP algorithm is invoked to determine the horizontal translation and
rotation required to align map M to map S. The closest match is selected to
calculate the algorithm's best estimate of the rover's horizontal geoposition.
The above process is repeated at each pose of the mobile robot for the duration of
the GPS blackout, or whenever redundancy in position estimates is desired. Redundancy
in geoposition estimates might be required in cases when GPS signals are under
intentional or abnormal spoofing.
As shown in Figure 6, the operation of the localization framework is very similar
for the teleoperation and autonomous scenarios. In the teleoperation scenario, the
operator is responsible for checking the reliability of GPS based localization data. In the
case when GPS data become unavailable due to attenuation, or become unreliable, the
operator manually invokes the localization algorithm through a command on the control
panel (Graphical User Interface - GUI). Following receipt of the command, the
localization algorithm acquires data and performs matching to produce an estimate of the
horizontal geoposition of the mobile robot. The estimated geoposition data is then
Page 45 of 140
transmitted back to the operator. The operator can then interpret the data and issue
another localization command if needed. This scenario involves manual operator
interaction with the localization algorithm without the use of a GN&C subsystem.
The autonomous scenario replaces the operator with a GN&C subsystem, which
assumes responsibility for navigating the mobile robot based on waypoints set by the
operator. In this scenario, the GN&C subsystem is responsible for assessing the reliability
and availability of GPS data. In the case when GPS data become unavailable, the GN&C
subsystem invokes the localization algorithm and interprets its results. The GN&C
subsystem can elect to continue to use the localization algorithm as long as GPS data is
unavailable or unreliable.
It is important to note that the architecture of the localization algorithm should be
invariant to the operational scenario. This is in fact the case as discussed in the following
sections. The localization algorithm should also utilize the same inputs and outputs for
both scenarios in order to facilitate integration and interoperability. Section 2.5 discusses
the architecture of the localization algorithm in detail.
Pagc 46 of 140
.
-
-1 \/
/
1
OE
0 0
- - - - - - - - - --- - - - --
-----------
CLC
4E'I
.E
L"
0
E
0
E
-o4j
E
ci
I
0
0
EM
-
Zn E
Io3.v'
-
BPI
a
cE
E
Ii
S
E
cC.u
_
3- C
CE
I
-'a.<
0)
.2
0o<2
41
0
0
___
z
z
0\
/
Figure 6: Sequence of operations (flowchart) for localization framework showing the teleoperation and
autonomous control scenarios
Page 47 of 140
2.3
Mission Requirements
To constrain the research and focus the development of the localization
framework, mission requirements were developed that describe the functional,
performance, operations and communication aspects of the localization framework within
the context of navigating a mobile robot in a forest environment. The following lists the
major requirements that define the localization framework.
Functional requirements:
" MR-1: Localization of the mobile robot shall be achieved by invoking a
localization algorithm that resides on the mobile robot computer system.
e
MR-2: The localization framework shall accept manual command by an operator
in teleoperation scenario as well as through automatic command by the GN&C
system as part of an autonomous scenario.
e
MR-3: The localization framework shall accept data inputs from an onboard
LiDAR sensor as well as overhead imagery from a preloaded library.
e
MR-4: The localization framework shall perform its own computations to
produce an estimate of the geoposition of the mobile robot.
e
MR-5: The localization framework shall be implemented such that it can be
integrated into a Discrete Kalman Filter (DKF) for SLAM purposes.
" MR-6: The localization framework shall produce a geoposition estimate
whenever it is invoked. The framework is still expected to produce an estimate
even if the result is inaccurate.
Performance requirements:
-
MR-7: The localization framework shall produce geoposition estimates with
accuracies proportional to the accuracies of the input data used. In other words,
the error function of the output should be O(E) where E is the set of errors of
input data.
e
MR-8: Geoposition estimates by the localization framework shall be produced
free of systematic errors (bias) and should only include random errors that are on
the order of the errors of input data (E).
*
MR-9: The localization framework shall produce geoposition estimates in a time
frame that is 1:1 compared to acquisition time. In other words, processing of data
to produce an estimate shall take the same time duration as data capture. This is
important when considering real-time operations.
Page 48 of 140
2.4
Architecture: Subsystems and Components
The architecture and interfaces of the localization algorithm are discussed in
detail in this section. Systems engineering methodology embodied in the process of
deFining the system's components and their inherent interfaces is adopted here. The
localization algorithm is treated as a system with several layers of components that
interact together to embody the functionality of the algorithm as a whole. This section
discusses the functional breakdown of the algorithm and the internal and external
interfaces. Aspects related to the integration of the algorithm to GN&C subsystems are
also discussed.
The localization algorithm can be treated as a standalone system that is composed
of three main components (subsystems):
*
Tree Crown Identification,
Overhead Imagery:
Delineation
and Center Estimation from
Delineates tree crowns, extracts their shapes and estimates locations of tree
centers by calculating the geometric centroid.
" Tree Stem Identification, Extraction and Center Estimation from Rover
LiDAR Data:
Identifies tree stems, extracts their 3D profile and estimates stem center horizontal
locations at a particular height above ground.
e
Matching of Tree Centers from Overhead Imagery and LiDAR Data:
Estimates the rover's horizontal geoposition by matching tree center maps
generated from LiDAR to a map obtained from overhead imagery.
The breakdown of the subsystems identified above is purely based on functional
aspects. Each subsystem is comprised of several functional modules that interact
internally to produce an emergent function at the subsystem level. In similar fashion, the
three subsystems interact with each other to produce an emergent function at the system
level. Figure 7 illustrates a functional (hierarchal) breakdown showing 3 functional
levels: System, subsystem and component. Table I describes the functionality of each of
the components.
The block diagram in Figure 8 illustrates the functional interactions between the
different modules and delineates the subsystem boundaries. Inputs and outputs of the
algorithm at the system level are also illustrated along with the flow of data from the
various components. Section 2.5 discusses the inputs and outputs of the algorithm in
more detail.
Page 49 of 140
Localization
Algorithm
Tree tem
ree tem
ide ntification from
Overhead Imagery
Identification from
Ground LiDAR
Green Channel
Extractor
Ground Height
Filter
Binary Image
Creator
SVM Classfier
Watershed
Segme nter
--
Matc ing of
Overhead and LiDAR
based Stems
Robust ICP
Matcher
Rover Position
Calculator
Brast Height
Stem Points
Slicer
Least Squares
Primitive Fitter
-+ Crown Classifier
Stem Center
Centroider
Centroid
Estimator
Figure 7: Functional breakdown of localization algorithm with three levels of hierarchy: system, subsystem and
components
Table 1: Description of functional components of localization algorithm
Tree Crown
Identification,
Delineation and Center
Estimation from
Overhead Imagery
__
__ _ __ _ __ _I_ _
I __ _ _ __ _ _
Green Channel
Extractor
Extracts green objects from image and
discards non-green objects.
Binary Image
Creator
Reduces noise produced from the
green channel extractor and creates a
binary representation of the image.
Watershed
Segmenter
Delineates boundaries of detected tree
crown objects for standalone trees as
well as tree clusters based on a
distance transform method.
Crown Classifier
Identifies valid crowns and extracts
them.
Centroid
Estimator
Calculates geometric centroid of
extracted crowns in the horizontal
__ _ _ __ _ _ __ _ _ __ _ _ _
Page 50 of 140
plane. Produces constellation set S.
Tree Stem Identification,
Extraction and Center
Estimation from Rover
LiDAR Data
Matching of Tree
Centers from Overhead
Imagery and LiDAR
Data
Ground Height
Filter
Identifies tree stems and selects lowest
point for ground plane estimation.
SVM classifier
Validates ground plane estimates from
the ground height filter by comparing
against training templates.
Breast Height
Stem Points Slicer
Performs a horizontal slice of
identified stems at a predetennined
height threshold.
Least Squares
Primitive Fitter
Performs a least squares fit of a
cylinder primitive onto the extracted
horizontal slice to validate candidate
stem objects.
Stem Center
Centroiding
Calculates the geometric centroid of
sliced data to determine horizontal
location of center of stem. Produces
local set M at each pose.
Robust ICP
Matcher
Invokes ICP to find a geometric match
between the local dataset M and
constellation S. Produces an estimate
of the translation and rotation needed
to align M to S.
Rover Position
Calculator
Based on the translation and rotation
estimates, the geoposition of the
mobile robot is calculated.
As illustrated in Figure 8, the tree crown identification and delineation algorithm
sequentially invokes several components based on an input orthophoto image. The
overhead georeferenced image of the forest canopy is first processed by extracting
vegetation blobs using the green channel extractor. Next, the resultant image is converted
to its binary representation by the binary image creator. Crowns are then delineated by
the watershed segmenter before calculating their geometric centroids. Centroid data is
then used to form the constellation map S. This whole process can be executed once for a
large orthoimage before the rover commences its mission, or can be executed at each
pose. The former scenario is the one assumed for our research purposes.
Page 51 of 140
LiDAR Data Per
Pose
Orthophoto
C
0
Green Extraction
& Denoising
E
Binary Image
Creation
C
M
C
0
C
I
Ground Height
Filter
SVM Classifier
C
C
0
Watershed
Segmentation
C
0
Breast Height
Slicer
0
M0
0
Crown Classifier
Least Squares
Primitive Fitting
C
C
3:
E
Centroid
Estimation
0
4.0
LIn
Stem Center
Centroiding
Ground Stems M
Stem Constellation S
I
C
0
E
Establish Search Space
C
0
0
Invoke Robust ICP
Calculate Location Based on ICP Match
Absolute Horizontal
Localization Estimate
Figure 8: Block diagram of localization algorithm showing the three main subsystems and their components
Page 52 ol 140
The LiDAR based tree stem identification algorithm invokes several components
after a LiDAR scan is initiated. The ground height filter and the SVM classifier are
utilized to identify and differentiate between stem features and the ground plane. After
extraction of valid stem 3D data, the breast height slicer extracts a horizontal profile of
identified stems at a particular height threshold. 3D data pertaining to stems are processed
by the primitive fitter whereby cylinder primitives are fit in a least squares sense to
clusters pertaining to tree stems. Fitted data clusters are labeled as tree stems and their
geometric centroid is calculated. Centroid locations relative to the rover are used to create
a map of the relative horizontal locations of tree stem centers (map M).
The data matching and location estimation algorithm takes a map of tree stem
centers estimated from LiDAR (map A) and compares it to the absolute constellation
map generated from overhead imagery (map S). At each pose, the search space size and
location are defined. The robust ICP algorithm is then invoked within the search space
provided to determine the best alignment between map M and map S. The best alignment
(match) is selected and the best estimate of the robot's horizontal geoposition is
calculated.
2.5
Inputs and Outputs
The inputs and outputs of the localization framework at the system level are
described in this section. Table 2 describes the inputs and outputs to enable clear
integration into GN&C subsystems.
Table 2: Descriptions of system level inputs and outputs of the localization algorithm
Input
Overhead
-
Image-
LiDAR Scan
Color: RGB
Correction: Ortho-rectification (Constant image scale)
-
Format: GEOTIFF
*
Reference System: UTM [Easting, Northing]
e
Resolution: No restriction
-
Coverage Area: No restriction
*
Range: No restriction
*
FOV: No restriction
-
Resolution: No restriction
*
Format: Metric triplets (x, v, z) with units [m, m, m],
Page 53 of 140
floating or double precision
* Reference System: Local, relative to mobile robot
reference frame
Output
Geoposition
-
Format: Metric doublet (x, y) with units [in, m], floating
or double precision
e
Reference System: UTM [Easting, Northing]
Page 54 of 140
-Blank-
Page 55 of 140
Chapter 3
Tree Crown Identification, Delineation and Stem
Center Estimation from Overhead Imagery
This subsystem creates a map of the absolute horizontal gcopositions of tree stem
centers extracted from a high-resolution orthophoto of the forest canopy. The purpose of
the map is to serve as a reference for comparison to smaller local maps of tree stems
observed by a LiDAR sensor onboard the robot. Our chosen scenario assumes that the
robot is equipped with a large orthophoto library of the exploration area and that this
subsystem is invoked once before mission execution. The absolute constellation map is
stored onboard the mobile robot for use whenever GPS is down.
3.1
Tree Morphology and Modeling
In order to develop a robust tree crown extraction and centroiding algorithm, the
growth profile of trees, or morphology, is first defined. In addition, the interaction of light
with the 3D profile of crowns needs to be understood. This section provides a focused
treatment on the development of a tree growth model suitable for this thesis. The growth
profile of a tree can be described by an optical model that is based on the work by Larsen
and Rudemo [48], which in turn extends the original model proposed by Pollock [49].
The model is comprised of the following basic elements: tree crown, tree stem, the
ground plane and diffuse light captured by a camera sensor.
Light originating from the sun is modeled as a collimated beam of light that
illuminates the tree crown and its surrounding terrain. More specifically, clear sky is
modeled as a collection of discrete collimated light rays emanating from evenly
distributed directions across the hemisphere (jr steradian) as described by Woodham and
Gray [50]. Standard Lambertian diffuse reflection is assumed for light rays reflecting off
the tree canopy and ground. It is noted that shadowing from neighboring trees is not
considered in this model for simplicity. The overhead imagery was selected with the sun
near or at zenith to eliminate any effects of shadowing. The optical axis of the camera
Page 56 of 140
sensor, whether airborne or spaceborne, is assumed to be co-aligned to the zenith of the
sky looking down onto the tree canopy. Figure 9 shows an illustration of the optical tree
model. The tree crown shape can be modeled as a generalized ellipsoid in 3D coordinates
(x, y, z) that define the external profile of the crown:
(z
((x - xt) 2 + (y
+ ch - zt)cc
cc
-
yt 23-
Crcc
chcc
zt - ch
z
zt
Where ch and cr are the vertical and horizontal extensions of the crown and cc is
a coefficient of crown curvature. cc = 1 represents a vertical crown structure (straight
line), while cc < 1 represents a concave crown structure. The tree crown model can be
located in 3D by the triplet (x,, yl, z,) that represents the coordinates of the apex of the tree
crown. bh and brh represent the base height of the stem and the breast height
respectively.
Sun
Y
Camera
Crown
Curvature (cc)
Crown
Height (ch)
Tree Crown
Crown
- -
-- Rad iu s(cr)
Base
)
Tree Stem
Breast Height
(brh)
Ground
Figure 9: Optical tree model showing various parameters that define a particular tree
Page 57 of 140
Crown height ch, crown radius cr and base height bh are the three fundamental
parameters needed to define the morphology of a tree. Although actual trees in a natural
environment have various morphology parameters and shapes to be defined by a simple
model, field data collected for the purposes of this thesis at Lake Mize in Florida contain
trees that can be approximately described by this model. In particular, the assumptions of
the model hold well for most conifer trees in natural stands due to their nearly vertical
pyramidal structure and crown symmetry. The data at Lake Mize contain Pine trees that
are known for their symmetrical and nearly vertical crown structure. Figure 10 shows a
comparison of the morphology of several species of trees including conifer trees (Pine).
Columnar
Upright
Pyramidal
Oval
Spreading
Rounded
Vase Shaped
Weeping
Figure 10: Different morphologies of trees including pyramidal (conifer) treest
The tree model also assumes that the interior of the tree crown contains randomly
oriented scattering objects such as leaves, needles and twigs that make up the majority of
the volume of the crown aside from the branches. For simplicity, light is assumed to be
scattered evenly by the crown such that the full horizontal extents of the crown are
clearly observed. In the case of clusters of crowns that touch each other, it is assumed that
such crown blobs involve few trees that are surrounded by terrain as shown in Figure 11.
t http://3.bp.blogspot.com/-mnMAPG s8-bl/UaOdDcalIFI/AAAAAAAADD8/Dfd241bJUiI-g/sl600/Forn-'I'rces.JPG
Page 58 of 140
The ground plane is included in the model to provide color contrast for tree canopy
visible in the foreground. In other words, the presence of ground (terrain) is important,
but not critical, to the identification and classification of individual tree crowns. This is
because the changes in color gradients between a candidate crown object and ground
facilitate the delineation of the horizontal extents of a crown. However, changes in the
color gradient across a cluster of crowns can in principle help in delineating individual
tree crowns without the need for terrain in the background. These aspects are captured in
the design of the delineation algorithm.
The 2D mathematical representation of the tree morphological profile is given
below. The relationship denotes the elliptical nature of the cross section of the crown
when viewed in 2D from above. Figure 11 shows an illustration of the 2D projection of
tree crowns as viewed from an overhead camera sensor.
crCCC
An edge in an image can be detected due to the intensity discontinuity of the
underlying scene. The discontinuity can be due to a jump in 3D depth of an object, a
surface normal discontinuity, a diffuse reflectance discontinuity or an illumination
discontinuity (Marr and Hildreth) [5 1]. Discontinuities in dense forests are caused by the
change of contrast between tree crowns and background (ground) or by the discrete
changes in color gradients from crown to crown. As shown in Figure 11, three main types
of tree crown objects have been classified:
1. Isolated Trees: Single crown objects that are separated completely from each
other by a distance greater than or equal to the pixel resolution of the image.
2. Touching Trees: Tree crowns are clustered in one direction or the other with
their crowns touching.
3. Tree Clusters: Several tree crowns are touching in all directions forming an
extended object of an oblong or irregular shape.
The dotted lines in Figure 11 denote the expected extents of the crowns of the
individual trees. All types listed above and illustrated in Figure I I will need to be
handled and analyzed by the delineation algorithm. It is important to note that the tree
model assumes that tree clusters within a frame are located within close proximity to the
principal axis of the camera. This is an assumption that only holds for large format
images covering wide swaths with the area of interest located at the center of image and
making up a small fraction of the total area. This assumption simplifies the tree optical
model by eliminating secondary positional errors due to parallax.
Page 59 of 140
Touching Trees
Camera Principal Axis
Tree Clusters
isolated
T rees
Figure 11: Different types of horizontal crown profiles as seen by an overhead camera showing isolated,
touching and clusters of crowns
Stem Center Estimation Hypothesis
3.2
Based on the tree morphological model described in Section 3.1, a hypothesis for
the extraction and estimation of stem centers can be stated. The hypothesis relies on the
central idea that for an ideal pyramidal (i.e. conifer) tree, the horizontal centroid of the
base of the crown should coincide with the horizontal centroid of the tree stem at breast
height brh. The following statements expand upon the central idea just discussed:
-
e
An ideal standalone conifer tree will have identical horizontal centroid locations
for the crown and stem at breast height. This configuration will result in the
highest classification accuracy, whereby classification means that the correct stem
center has been estimated for a particular tree crown.
A configuration of crowns that are slightly touching or densely clustered can be
decomposed into their constituent crowns. It is noted however that the
classification accuracy of these crown clusters may not reach the classification
accuracy of the standalone tree configuration. This is mainly due to the
uncertainties associated with the delineation of the merged crown objects in the
image.
Page 60 of 140
e
If a tree crown is detected and delineated, the location of a tree center can in
principle be calculated with a horizontal accuracy limited by the pixel resolution
of the image.
The central hypothesis above and the aspects related to delineating tree crowns in
different types of crown formations are taken into consideration as part of the
development of the crown delineation and center estimation algorithm discussed in
Section 3.3.
3.3
Image Processing Methodology
This section focuses on discussing the details of the crown identification and
delineation algorithm. Given an overhead image, the algorithm will automatically
identify isolated and clustered trees, segment the clusters and provide an estimate of the
horizontal centroid location of each identified crown.
The current state of the art in crown detection can be classified according to the
type of input data: visible image based [52], infrared based [53], or LiDAR based [54].
Data from infrared and LiDAR sensors provide intensity and metric measurements of tree
canopies, respectively. However, due to their high acquisition costs, the availability of
such data is very limited compared to visible imagery (aerial or satellite). Significant
research by the robotics community on segmentation algorithms, such as the watershed
and Hough transform, influenced the development of the segmentation algorithm for this
thesis.
Several automatic tree crown delineation algorithms have been previously
developed. Wang et al (2004) utilize a multi-step approach where Near Infra-Red (NIR)
images of tree canopies are processed to pinpoint crown centers by finding pixels with
the maximum grayscale intensity. They hypothesize that due to the arrangement of tree
leaves and branches at the top of the crown, its apex will have the largest surface area,
and thus will reflect radiation the most. The pixel with the highest intensity is used as a
seed by a marker-controlled watershed segmentation algorithm [55].
As discussed in more detail below, the watershed segmentation algorithm allows
the delineation of complex crown clusters to extract individual tree crowns [55].
Although effective, Wang's approach was not entirely followed because it relies on NIR
imagery. For the purposes of this thesis, a variant was developed that uses visible
imagery readily available from open source map libraries such as Google Earth.
The following summarizes the major steps undertaken by the tree crown
identification and delineation algorithm:
1. The input orthoimage is processed to delineate and extract individual tree crowns.
Page 61 of
140
2. Stem center pixel locations in the image are determined by calculating the
locations of the geometric centroids of each detected crown.
3. Absolute horizontal geoposition of each stem center is determined by indexing
pixel locations given the absolute geoposition of the four corners of the image.
Figure 12 illustrates the algorithm's methodology given an overhead image as
input. The following subsections detail each of the steps shown below in further detail.
Orthophoto
----------------Green Extraction
& Denoising
Binary Image
Creation
Watershed
Segmentation
I
Crown Classifier
Centroid
Estimation
Stem Constellation S
Figure 12: Image processing flowchart given an input orthophoto image of the exploration site
The algorithm uses the 2-dimensional Cartesian Universal Transverse Mercator
(UTM) reference system. Therefore, prior to any image processing step, all orthoimages
are reprojected onto UTM if needed. The open source QGIS software was used for this
purpose [56]. We have found that not all of the orthoimages supplied by USGS through
Google Earth are UTM referenced. Orthorectified images are necessary for accurate
geopositioning of observed tree stems. An orthoimage is typically corrected for terrain
irregularities and lens distortion such that the scaling of the image is maintained
Page 62 of 140
throughout the whole frame to enable accurate distance estimations. Chapter 6 discusses
the image rectification process as part of the data setup for the various tests.
To demonstrate the various steps of the algorithm as described in the following
subsections, an example orthophoto of the test site is used. The overhead image is of a
pine forest located northeast of Lake Mize in Florida, USA. The image is an orthophoto
that was acquired from USGS through their Google Earth portal. The image was provided
in GEOTIFF format and was reprojected to UTM prior to processing. The specifics of the
image and its orthophoto projection are discussed in detail in Chapter 6.
Specifications related to area size and image resolution are included below for
ease of reference:
" Image size (pixels): 380
x
400 (Horizontal x Vertical).
-
Pixel resolution: 0.33 m (1 Foot).
*
Image Scale: 3.28 pixels/in.
-
Image Accuracy: 2. 1 RMS.
e
Number of valid tree crowns: 200. This was determined by manually inspecting
the point cloud generated by the ground LiDAR and visually correlating trees
observed in the overhead image to those clearly visible in the point cloud.
In each of the steps discussed below, the input image is processed in successive
fashion to demonstrate the result. The intent of this approach is to facilitate the
presentation of the algorithm and its performance.
3.3.1 Green Channel Extraction and Denoising
Image processing starts by discarding non-green pixels (i.e. non-vegetation) using
a simple color contrast threshold approach. This step removes terrain, roads and other
non-green objects leaving behind vegetation. It is noted that this approach is not effective
for deciduous trees that lose leaves in the fall. The green channel filter can be expressed
as follows:
gin(Xin,Yin) = G(xinyin)
r (xin, Yin)
b (xin, Yin)
2
2
-
Such that gin(Xin, yin) > t
Where gia~xin,y'i) is the green color index for a particular pixel at column vi and
row xin of the image. g(vin,yin), r(xin,yvi) and b(xiv,y~,,) are the intensity values (0-255) of a
pixel at column yi, and row xi, for green, red and blue respectively. t is the threshold upon
which gi, is compared against. The higher the index, the higher the green intensity for a
particular pixel.
Page 63 of 140
Pixels with gin that are greater than t are therefore selected. Based on empirical
tests performed on the data of the site near Lake Mize, a useful threshold was found to be
in the range between 13-15. Figure 13 shows the input image used of the Lake Mize site
while Figure 14 shows the resultant image after extraction of all green objects.
Figure 13: Original orthophoto input image of Lake Mize test site (Source: USGS - Google Earth)
The resultant color image is then converted to a grayscale image with 8-bit
resolution (i.e. standard 256 color levels) using the luminance algorithm [57]:
Ng(xin,Yin) = 0.299 x r(xin,yi,) + 0.587 x g(xin,yin) + 0.114 x b(xin,yin)
Where Ng is the resultant grey scale image. Figure 15 shows the resultant image
after application of the luminance algorithm. A Gaussian disk filter is then used for
denoising with a specified diameter equal to the smallest expected crown diameter (in
pixels). The Gaussian blur function G(xin, yin) is defined as follows:
Page 64 of 140
1
G(xi.,yin) = 2'ro
_i2+2
2
e
2a2
Where a is the radius of the smallest expected crown diameter (in pixels) and i
andj represent indices of the blur kernel (matrix).
Figure 14: Resultant image after green object extraction with all non-green objects removed
The image is then convolved with the Gaussian blur function:
Nr(xin, Yin) =
Y.
X= _a
a
I
G(i, j) I(xin - i, Yin -
)
a
Y=-CT
Where N,(Xin, yi) is the resultant image and I(xin, Yin) is the input image.
Figure 16 shows the resultant image after application of the Gaussian blur
function through an erosion and dilation technique.
Page 65 of 140
Figure 15: Resultant image after application of luminance algorithm that converts RGB to 8-bit grayscale
Page 66 of 140
Figure 16: Result after application of Gaussian blur function using erosion and dilation techniques to remove
spurious pixels
3.3.2 Binary Image Creation
The next step involves creating a binary image that is used to calculate the
Euclidean distance transform for all crown objects in the image. The conversion
relationship is given below:
Nb(XiY)
1
0
Nr (xin, Yin) > 1
Nr (xin, Yin) < 0
Where Nb (xin, in) is the binary image representation of N, (xin,yin). In effect,
N, (xin, yin) is a 2D map of logical 1 or 0 where 1 represents the presence of a pixel of
intensity greater than 1 (out of 255) and 0 represents the presence of a pixel of intensity
less than 1. Figure 17 shows the resultant image after conversion to a binary image.
Page 67 of 140
V4
Figure 17: Resultant image after creating a binary copy of the grayscale image
3.3.3 Watershed Segmentation
The next step involves applying watershed segmentation to the binary image with
the distance transform as input. The Euclidean distance transform d between two points
(pixels), p and q, is defined as follows (in 2D):
2
d(p,q)=
(q - p,)2
The purpose of the distance transform and watershed segmentation is to delineate
individual tree crowns or segment any cluster of treetops into their constituent trees. This
step is necessary in the case of dcnsc forests that have touching or clustered tree crowns.
Page 68 of 140
Watershed segmentation relies on detecting "catchment basins" between objects.
In other words, watershed segmentation interprets a grayscale image as a 3D map
composed of 2D pixel coordinates with intensity as the third dimension. Changes in
intensity between clusters of adjacent pixels along catchment basins (watershed lines) are
interpreted as distinct objects.
For completeness, a description of the watershed algorithm is given here. The
watersheds of an object in an image can be mathematically represented as the set of all
points that are considered local minima in the intensity map. By local we mean proximity
within an area enclosed by the typical crown diameter of a tree. The watershed of an
image is the set of watersheds of its local minima [58]:
WS((Xinyin),N) - {(xi,yi) I (Xin, yin) E Downstream[N](xi,yi)}
WS( A, N)
U
WS((xi, yi), N)
(x7,yi)EA
Where WS(xin, yin) is the set of all points of a particular object in image N with a
downstream in which (xin, yin) is a member and WS(A,N) is the union of the watersheds
of its members. The downstream function is defined as the union set of all possible
Swiftest Descending Paths (SDP) for pixel (Xin, yin).
SDP is defined as the finite succession of connected pixels (x,yi) such that each
pixel is strictly lower than its predecessor and is one of its lowest neighbors. SDP is
described below and downstream is defined as the union of all possible SDPs for that
pixel.
SDP[N](xin, yin) = {(xi, yi)1((xi, yi) E Ne(xi
1 , yi1),
I(xi, yi) < I(x-
I(xi, yi) = min tI(xj, yj)I(xj, yj) E Ne (xi,
1 , yi_),
yi_1),
(xO,yO) = (x,y)}
Downstream[N](x, y) -
SDP[N] (x,y)j
I
Where Ne(xin, yin) is defined as the immediate neighbor to pixel (xin, yin) for 8connectivity on square raster:
Ne (xin,yin) = {(xn, yn) E NI(x, y) = (Xin + i, Yin + i),
(i, j) E f{(- 1,0), (0, - 1), (0, 0), (0, 1), (1, 0)}}1
Figure 18 shows the result of applying the watershed algorithm to the binary
image N1 (xin, Yin).
Page 69 of 140
Figure 18: Result of applying watershed segmentation to binary image
Nb
to delineate individual tree crowns
3.3.4 Crown Classifier
Following the segmentation step, an image is produced that contains delineated
objects representing potential tree crowns of various sizes. To reduce the likelihood of
identifying false positives (underbrush), a two-feature classifier is used. The first feature
utilizes the estimated diameter of the crown. Crowns with equivalent diameters of less
than a predetermined threshold are discarded. For the purposes of the thesis and
considering the sizes of tree crowns in our test data, a threshold of 1.5 m was used.
The second feature utilizes the estimated eccentricity of the crown. Crowns with
eccentricities larger than a predetermined threshold are discarded. Eccentricities below
0.9 were used to allow for highly segmented crown structures. The classifier features are
described in Table 3. Figure 19 shows the result after applying the classifier. Areas
shown in red failed to pass the classifier thresholds and were therefore discarded. Areas
in green are considered valid crowns. 200 valid crowns were identified (green) and 108
Page 70 of 140
objects were deemed invalid (red). To ascertain classification accuracy, the algorithm is
tested using ground truth data as discussed in Section 3.4.
Table 3: Crown Classifier Features
F1
Estimated diameter of object
max diamij =
F2
1.5m < diamij < 12m
(xi - xX + (yi - y2
Estimated eccentricity of object
e < 0.9
2
e=
1-diammmn
d iaminax
Figure 19: Result after applying the two-stage classifier with green objects denoting valid crowns and red
objects denoting invalid crowns
Page 71 of 140
.
..........................
...
3.3.5 Centroid Estimation
The remaining valid crown objects are subsequently analyzed to estimate their
centers by calculating the Euclidean centroid of each delineated object. The centroid is
calculated by:
ci Ci=X1
=
+ X21 + -- + Xn'
n
CL, X E R2
Where Ci is the centroid vector of the crown object i and X, are discrete points
that comprise crown C,. With center pixel coordinates C(xin, yi) of each valid crown
obtained, relative pixel coordinates are indexed and mapped to the absolute global
positions of any of the four georeferenced corners of the original orthophoto.
Figure 20: Final result showing constellation map S with all crowns delineated and valid centroids denoted by
(+)
Page 72 of 140
The final product is a constellation map composed of the absolute horizontal
coordinates (Easting and Northing) of the centroid of each valid crown. Figure 20 shows
the result of applying the last step of centroid estimation.
3.4
Trade Space and Performance Assessment
A sensitivity study was performed on the tree crown delineation and center
estimation algorithm to determine the most suitable parameter settings. The goal was to
achieve the least positioning error by iterating over a wide parameter set. A Design Of
Experiments (DOE) was therefore conducted. For every simulation run of the DOE, a
constellation map S is produced based on a particular parameter set. The map is then
compared to ground truth data by using a nearest neighbor search to quantify the distance
error. The parameter space range and increments are summarized in Table 4. The ground
truth dataset was obtained from a ground LiDAR survey of the Lake Mize site. The same
LiDAR dataset served in our assessment of the localization accuracy of the algorithm as
well.
Table 4: DOE parameter space with 6 variables and ranges for each
t
Ureen channel intensity threshold [integer]
(Y
Gaussian disk diameter [pixels]
Ne
Erosion and construction noise threshold
[integer]
20 - 100, increments of
20
dmin
Minimum crown diameter [m]
1.5 - 5, increments of
0.5.
dma
Maximum crown diameter [m]
6 - 12, increments of 0.5
e
Eccentricity of detected object
0.7 - 1, increments of
0.05
8 - 20, increments of 1
- 5, increments of I
Results from the DOE simulation span a 6 dimensional trade space and therefore
it is not possible to visualize completely on a 2D plot. The optimal parameter space that
achieved the lowest Euclidean distance error between map S and the LiDAR ground truth
is listed below:
*
Green color threshold: 13
*
Denoising disk diameter: 1
e
Image erode/construction noise threshold: 20
Page 73 of 140
e
Min crown diam: 1.5
e
Max crown diam: 12
e
Eccentricity threshold: 0.9
To determine the horizontal accuracy of the output, the resulting maps from the
algorithm were compared to the ground truth dataset of the forest grounds. As will be
explained in Chapter 6, ground LiDAR data of the test site was acquired using a surveygrade LiDAR instrument. The 3D data was examined manually whereby tree stems were
picked and labeled by hand. The handpicked horizontal geolocations of stems were then
compared to the geolocations reported by the algorithm of the same stems. To estimate
the error between both maps, a nearest neighbor search algorithm was used to find
corresponding point pairs and determine the Euclidean distance error between them.
Figure 21 shows a plot of stem center maps that were automatically estimated from the
image and those hand-labeled from ground LiDAR data.
o
o
100
+
0
0
40
0
F
0
o
0
0
0
0
I
0
0
0
0
0
00
0
00
0
0
0
0 0
0
00
G+
0
e
0
0
0
0
0
0
80
0
0
0 0
01
E
GroundTruth(LDAR)
NWarest Neighbour
0
00
80
_-
+
0
0
0
GF0
o ConeltionsetS
0
0
G-
0
0
0
0
C
0
0
G+
0
0
0
0
9D
0
a
Q+0
f
0
0
0
0
0
0
I
Z1
40 1;
0
A
04
0
0
G44
0
0-4-
0
0
0
r
-
201
0
0
00
00
0 0
0 -0
0
0
-1r
0
0
0
1 0
+4
0
0
0
0
0
0
0
0 -
t
10
10
0
0G_
-
0
20
0
30
40
Eating(m)+ 382360m
~
0
-OD
50
60
0- *1
70
-P0+O
100
80
Figure 21: Nearest neighbor search results from comparing stem locations in overhead image to those manually
estimated from ground LiDAR data
Results from the nearest neighbor search indicate 1.35 m RMSE (Root Mean
Squared Error) between corresponding point pairs in the two data sets. This error is less
than the estimated 2.1 m RMSE of the orthophoto used. The indicated accuracy of 1.35
Page 74 of 140
RMSE is treated as the classification accuracy of the delineation algorithm. It is noted
that the total error reported for an image includes the contributions of the following error
sources:
*
Orthorectification error of image supplied.
*
Absolute geoposition error of image supplied.
*
Centroid estimation error due to suboptimal green gradient identification.
e
Incorrect labeling of underbrush as valid tree crown (false positive).
For final comparison purposes, Figure 22 shows stem constellation map S
superimposed onto the input orthophoto image along with stems manually picked from
ground truth LiDAR data. Valid crowns are delineated in white.
OConstellation set S
Ground Truth WUAR)
R111+
Figure 22: Comparison between stems centers detected in overhead image to hand-labeled stem centers
observed in ground LiDAR data
Page 75 of 140
There are some general considerations as well as constraints that need to be noted
about the performance of the tree crown delineation and center estimation algorithm:
1. Accuracy of estimated horizontal locations of tree centers depends on the view
angle (perspective) of the image. Therefore, for best results and to reduce the
effect of parallax, the area of the forest of interest needs to ideally be at the nadir
of the image (optical axis). Inaccuracies should be expected for areas that are not
located at the nadir of the image.
2. Not all stems visible in the LiDAR point cloud are visible from the overhead
image and vice versa. This is quite noticeable in Figure 22 where red circles have
no nearby green crosses and visa versa. This limitation is inevitable in dense
forests due to the nature of growth of trees as well as the difference in FOV and
perspectives of the various sensors involved. In Figure 22, hand-picked LiDAR
based stems do not span the whole overhead image, which explains the gaps seen
in stems reported by the LiDAR as compared to sterns estimated from the
overhead image. In addition, dense underbrush obscures stems that could
otherwise be seen on the ground by the LiDAR. All these factors lead to
asymmetrical coverage and different results from sensors on the ground and those
overhead.
-
The average diameter of a tree crown is recommended to be at least an order of
magnitude larger than the pixel resolution of the input image [55]. For example,
the pixel resolution of an input image needs to be around 0.3 m in order to
adequately detect 3 m crowns.
Page 76 of 140
-Blank-
Page 77 of 140
Chapter 4
Tree Stem Identification, Extraction and Center
Estimation from Rover LiDAR Data
This subsystem creates a local map M of the horizontal coordinates of the
centroids of tree stems observed in ground LiDAR data. The purpose of the map is to
enable comparison to the constellation stem map S, which was derived from the overhead
image of the area being explored. The chosen operational scenario assumes that the rover
is equipped with a LiDAR sensor that scans the area surrounding the mobile robot at
every pose. At each pose, the localization algorithm is invoked where it correlates the
local stem map M and constellation map S to estimate the absolute horizontal location of
the mobile robot.
4.1
Stem Center Identification Hypothesis
Stem center identification from ground LiDAR relies on the central concept of
identifying candidate tree stems and calculating their centroids. As explained in Section
3.1, the centroids of conifer tree crowns should, in principle, align horizontally with the
centroids of stems determined from ground LiDAR. The following statements expand
upon the central idea just discussed:
*
In contrast to inter-crown growth morphology, each tree is expected to have its
own standalone stem that can be directly extracted. Figure 23 shows processed 3D
slices of neighboring tree stems [59]. For centroid calculation, a single slice is
extracted at breast height (brh) for each tree and included into local map M for
comparison to the constellation map S.
* The centroid accuracies are limited by the accuracy and precision of the LiDAR
sensor used. The accuracy of LiDAR data should be of the same order as the
Page 78 of 140
. .
. .....................
..
absolute accuracy (RMSE) of the overhead imagery. Figure 24 shows a single
slice of a tree stem at breast height along with the calculated centroid.
-
If a tree stem is detected and extracted, the location of a stem center can in
principle be calculated with a horizontal accuracy limited by the accuracy and
resolution of the data used.
Figure 23: 3D slices of different tree stems for a tree stand observed in 3D LiDAR point cloud [591
As will be explained in Section 4.2, a cylinder primitive with radius R is fitted in a
least squares sense to a 3D slice in order to determine the correct radius of curvature r.
The estimated radius of curvature is then used to calculate the geometric centroid of the
slice, and hence the tree stem it belongs to.
.,
r
R
Figure 24: Centroid calculation by fitting a cylinder primitive [601
The central hypothesis above is taken into consideration in Section 4.2 as part of
the development of the tree stem identification, extraction and center estimation
algorithm.
Page 79 of 140
4.2
Data Processing Methodology
This section focuses on the different components of the tree center estimation
algorithm. As discussed in Chapter 2, the mobile robot acquires LiDAR data of the forest
grounds at each pose and passes it to the tree stern center estimation algorithm. The
algorithm is designed to separate the ground plane from points comprising tree stems and
estimate horizontal locations of the centers of stems. This is accomplished by fitting
cylinder primitives to extracted 3D slices of candidate stems. The algorithm discussed in
this section draws heavily upon prior work by McDaniel et al [60]. Therefore, this section
represents a summary of McDaniel's work with few minor changes to its parameters.
Certain changes were introduced in order to address the main operational scenario
considered in this thesis.
Considering the baseline mission scenario, a single panoramic LiDAR scan is
taken at each pose for analysis and subsequent comparison to the stem constellation map.
Figure 25 shows the different steps of the tree stem identification and center estimation
algorithm.
LiDAR Data Per
Pose
Ground Height
Filter
SVM Classifier
Breast Height
Slicer
Least Squares
Primitive Fitting
Stem Center
Centroiding
Ground Stems M
Figure 25: Data processing flowchart for the stein center estimation algorithm given an input LiDAR dataset of
the area surrounding the mobile robot
Page 80 of 140
The following summarizes the major steps undertaken by the tree stem
identification and center estimation algorithm:
e
Estimate the ground plane based on local height filters.
e
Apply SVM based classifier to discard points comprising the ground plane
and leave behind candidate tree stems.
*
Slice candidate tree stems at breast height and extract 3D points that comprise
tree stems.
*
Perform fitting of cylinder primitives to candidate slices to determine valid
stems.
*
Calculate the relative location of the geometric centroid of each extracted
stem slice to build local map M.
The following subsections detail each of the steps above and presents
intermediate results to demonstrate the operation of the algorithm. Real-world data
acquired from the Lake Mize site were used to produce some of the results discussed in
the following subsections.
4.2.1 Ground Height Filter
In the first step, the ground plane is estimated in order to constrain the search
space and facilitate the identification of candidate tree stems. This step is divided into
two main stages. The first stage is a local height-based filter that labels the lowest point
in any vertical column of data. In practice, this eliminates a large percentage of nonground points from further consideration and facilitates the identification of candidate
stems in subsequent steps. The second stage is a feature classifier that combines eight
geometric measures, the classification features, to determine which of the remaining
points belong to the ground plane.
Figure 26 shows a trimetric snapshot of ground LiDAR data of the Lake Mize site
with absolute height color encoding. As will be explained in Chapter 6, the ground data
was acquired using a commercial survey grade LiDAR sensor with multiple scans
combined to form the large scanned area shown in Figure 26.
Figure 27 shows an example of LiDAR data acquired at a single pose. The ground
plane is clearly visible along with several nearby tree stems and underbrush. It is also
noted that the mobile robot and pose data were simulated using survey grade LiDAR data
of the Lake Mize site as explained in Chapter 6.
Page 81 of 140
Figure 26: Sample data showing LiDAR scans of forest grounds at the Lake Mize test site with vertical height
color-coding (red: high, blue: low)
Figure 27: Example of a LiDAR dataset at a particular pose showing the simulated mobile robot, tree stems and
underbrush
Stage 1: Local Height Filter
In the first stage, the input LiDAR point cloud is tessellated into 0.5 m x 0.5 m
cells across the horizontal plane. Each cell is identified by a unique row and column
index. Based on the cluster of points within each cell, the point with the lowest height is
Page 82 of 140
initially labeled as ground and denoted by P(ij). Figure 28 shows the different voxels
tessellated into 0.5 m x 0.5 m cells. Each of the columns is denoted by its own P(ij)
index.
1.2
1
0.8N
0.6N
E
0.4
0.2
0
-0.2
3
2.52
3
Y (m)
1.5
3.5
2.5
2
X
(m)
Figure 28: Cartesian space tessellated into 0.5 m x 0.5
m cells [601
Stage 2: Feature Extraction
Due to the presence of underbrush, not every point selected in the previous step
represents true ground. Therefore, the second stage involves coding a variety of features
to represent attributes of each lowest point P(ij) and the lowest points in each of the
neighboring columns, which are defined using eight 8-connected neighborhood columns.
Table 5 lists the 8 features used to discriminate between ground and non-ground points.
These features are combined into a feature vector F(ij) for each point, which are then
used by the SVM classifier to identify whether that point belongs to ground or not.
Table 5: Eight neighborhood classification features for tessellated data
Fi
Number of occupied columns in the neighborhood of column (ij)
F2
Minimum height of all neighbors minus height of column at (ij)
F3
Value of height at column (ij)
F4
Average of all heights in neighborhood of column (ij)
F5
Normal to best fit plane of points in neighborhood to column (ij). Defined
Page 83 of 140
as:
N
F5
=
argminn C3
n-
1
((P -
P -n)
--
2
k=1
F6
Residual sum of squares of best fit plane of points in neighborhood of
column (ij):
N
(P-P)n
F6
k=1
F7
Pyramid filter is computed as the number of minimum heights which lie
within a pyramid of 0.5 n cubic voxels with apex at P(i,j)
F8
Ray tracing score where points lying on flat ground are given a zero score
because they cannot lie between the LiDAR and any point it observes. This
feature is defined as the sum of the number of line segments that pass
through a voxel in column (ij) below P(ij).
4.2.2 Support Vector Machine Classifier
In this step, the Support Vector Machine (SVM) classifier is trained given the
feature vectors comprised of the eight features detailed in Table 5 and the hand-attributed
labels of whether each point P(ij) lies on the ground surface. If a cell contains a point
labeled by the classifier, it is treated as true ground.
Using a Delaunay triangulation scheme, interpolation is performed for cells that
do not have labeled points. The tunable kernel parameters and the misclassification cost
C, were found by minimizing the cross-validation error rate within the training data. The
SVM classifier implemented a linear kernel with classification cost C, = 100 [60]. It is
noted that these parameters were based on prior work by McDaniel et al and were reused
in this thesis [60].
4.2.3 Breast Height Stem Slicer
The next step involves identifying the vertical extents of stems using thresholding
and clustering methods. To identify which LiDAR point belongs to the main stem, a
simple height-above-ground filter is used. Any point below the selected height threshold
is discarded and classified as underbrush. A very common way of measuring the heights
of trees is to determine the stem diameter at breast height. The most commonly used
measurement standard for breast height is D1 30, which denotes the diameter of a tree at a
height of 1.3 m above ground. This particular height corresponds to the average height of
a person's chest (breast). Although a common standard, the height of 1.3 m above ground
Page 84 of 140
is not adequate for the data at hand. This is mainly due to the presence of heavy
underbrush in areas within close proximity to the majority of stems. Manual inspection of
the Lake Mize data showed that a breast height of 2 m is needed in order to clear tall
underbrush.
Figure 29 shows examples of LiDAR scan slices intersecting two stems and the
height difference observed from scan line to scan line.
Height
Difference
Figure 29: Consecutive LiDAR scans and corresponding stem slices [601
To extract candidate stem slices and obtain a single representative slice per tree, a
two-step process is followed. In the first step, stem locations in the horizontal plane are
identified. For the second step a slice of data is selected at each of the horizontal
locations identified earlier. The discretization of stem locations is performed at the voxel
resolution of 0.5 m x 0.5 m. To obtain candidate stem data, a vertical search along each
candidate main stem is performed. Considering a breast height (brh) of 2 m, the search is
performed within a window of 2 m - 0.2 m. If a data column contains any LiDAR point
within that range, that column is selected. Another search is invoked whereby for each of
the columns previously selected, those having a vertical 60 cm slice of data (defined as 2
m 0.3 m) are selected as a candidate stem. The 40 cm and 60 cm windows were defined
based on empirical observations of the Lake Mize data.
The two-step process described above improves the performance of the algorithm
as it guards against scenarios where dense underbrush could be present within a 60 cm
window but not in the 40 cm range. It is noted that the 60 cm window is small enough to
exclude non-stem data such as underbrush or crowns.
4.2.4 Least Squares Primitive Fitting
Based on stem slice data obtained from the previous step, cylinder primitives are
fitted to sliced data using a least squares scheme. Figure 30 shows how a cylinder
Page 85 of 140
primitive is fit to a horizontal slice of LiDAR data that pertains to a tree stem. R denotes
the radius of the base of the cylinder primitive, while r denotes the estimated radius of
curvature of the stem slice. The fit between the cylinder primitive and data is found by
solving the minimization problem below where n is the total number of points in a
cluster, X, is the transect containing LiDAR points pertaining to the identified stem and di
is the Euclidean distance from the ith point to the surface of the cylinder primitive.
n
arg min,, 9 j3
d? (Xe)
=
2=1
R
r'..
R
Figure 30: Example showing cylinder primitive fitment to a single slice of LiDAR data of a tree stem [601
To determine the quality of the least squares fit, four heuristic constraints that
characterize bad fits were defined. The quality of a fit, i.e. either good or bad, is
determined in order to discard bad fits. The following lists the four criteria used for bad
fit rejection:
* Distance from center of fit to centroid of data: This criterion rejects a certain
fit based on the ratio of the cylinder radius R to the distance r. The threshold
determined for rejection is 2. Any ratio greater than 2 results in rejection of
the fit. An example of a bad fit with a radius ratio of 22.5 is shown in Figure
31.
e
Maximum data spacing: This criterion calculates the ratio of the cylinder
primitive diameter (2R) to the maximum spacing d of main stem data. The
maximum spacing is found by invoking nearest neighbor search for all stem
slice data. This criterion helps in rejecting candidate cylinder fits that are too
large. In similar fashion to the first criterion, the threshold for the ratio of 2R
to d is 2. Any fitment with a ratio greater than 2 results in rejection of the fit.
Figure 32 shows an example of a bad fit with a ratio of 2.95.
Page 86 of 140
Figure 31: An example of a bad fit showing a large ratio of R to r [601
d
Figure 32: An example of a bad fit showing a large ratio of D to d [601
Intersecting tree stems: This criterion focuses on rejecting fits that have
overlapping cylinders. This problem can be caused by a least squares fit that
produces an excessively large radius for the cylinder primitive. This issue can be
detected by calculating the pairwise distances between the centers of each fitted
cylinder and subtracting the radii of both cylinders from the distance. If the
difference is less than zero, the two cylinders must have intersecting boundaries
and the cylinder with the larger radius is discarded.
Page 87 of 140
...
. . ............
*
Data on far side of tree: This criterion rejects bad fits where the cylinder fit is
located at the wrong side of the stem. This problem can be detected by comparing
the distance from the LiDAR sensor to the center of the cylinder, and comparing
the distance from the LiDAR sensor to the centroid of the stem data. If the
cylinder is in the wrong location, the distance to the centroid will be greater than
the distance to the center of the cylinder. The threshold used for distance is 0.25,
which is based on empirical testing and observation of the data used. An example
of this condition is shown in Figure 33 where the black dot shows the location of
the LiDAR sensor. 5 cylinders are placed at the correct side of the stem and one
cylinder (top left) is placed at the wrong side of the stem.
-
8000
-
7000
-
6000
5000
-
V
-
4000
-
3000
-
2000
1000
0
-6000
-4000
-2000
X (mm)
0
2000
Figure 33: Cylinder placement with respect to the LiDAR sensor with 5 correct cylinder placements and one
wrong (top left corner) [601
4.2.5 Stem Center Centroiding
Following the minimization step, horizontal locations of stem centers relative to
the mobile robot are determined by using the curvature of each fitted cylinder. The
horizontal locations of stem centers are then traced with respect to the mobile robot. The
Page 88 of 140
.......................
location of each estimated tree center in the horizontal plane is then incorporated for each
pose into a stem center map (map M).
The resultant local map M is then used by the matching component of the
algorithm as discussed in Chapter 5.
4.3
Accuracy of Stem Center Locations
Survey grade LiDAR data was acquired of heavy forest grounds near Lake Mize
in Florida and was georeferenced using survey grade GPS/Inertial Reference System in
differential GPS mode. As detailed in Chapter 6, LiDAR data was acquired by a surveygrade laser scanner with 1 cm accuracy (RMS). In order to quantify the classification
accuracy of the stem center estimation algorithm, the LiDAR point cloud was manually
examined and all visible tree stems were hand labeled. Figure 34 shows a snapshot of an
area inside the point cloud with colored markers denoting hand labeled stems (real
stems).
Figure 34: Manually placed markers by hand to identify true stems in ground LiDAR data of the Lake Mize site
The manually labeled data was then compared to the output of the stem center
estimation algorithm. This was preceded by performing the necessary georeferencing of
all local stem maps of the same area covered by the LiDAR point cloud. The manual and
automatic stem center location maps were then compared to determine the classification
accuracy. Figure 35 shows both maps plotted using stem center locations with respect to
the UTM frame.
Page 89 of 140
+
0
3.2905
+
+
+
+
3.2905 - +
+
+
True Stems (LiDAR)
AJI Stems (LiDAR)
Rover Path
+
0,
+
32904 -
32904
S32904
-
(
3.2903
-
0i 00)
3..8423.
3.823
3.824
Easting (in)
3.8242
+
3.8244
3.85248
5
i
Figure 35: Comparison between the locations of stem centers hand labeled (red) to those identified automatically
by the stem center estimation algorithm (blue) given LiDAR data of Lake Mize site as input. Simulated robot
path is shown in pink
Figure 35 shows that there is a strong agreement between the manually
handpicked dataset and the one automatically produced by the algorithm. However, there
were stems that were manually labeled but were not identified by the algorithm as seen in
the lack of blue circles in areas that contain red crosses. In total there were 200 valid
stems that were hand labeled. The stem center estimation algorithm did not detect around
70 of those. A reason for this mismatch lies in the presence of dense underbrush that
could reach heights above 2 m in some areas. This was indeed the case in several of the
areas manually examined.
In addition, the algorithm also produced false positive stems, i.e. stems that were
incorrectly labeled by the algorithm due to confusion by dense underbrush. Those are
shown by the blue circles that do not have corresponding red crosses. There were around
100 non-duplicated false positives produced by the algorithm. This prompted the use of
several filters by ICP as described in Chapter 5 in order to guard against data with
moderate false positives.
Although the algorithm produced false positives, stems that were valid matched
well with those manually labeled with errors on the order of the accuracy of the LiDAR
point cloud (1 cm RMS). It was determined that even in the presence of false positives,
Page 90 of 140
the algorithm was able to produce stem center data that can be used to good effect by
ICP.
It is noted that in order to characterize the total accuracy of the localization
algorithm, LiDAR data was treated as the ground truth upon which results from the
localization method is compared against.
Page 91 of 140
-Blank-
Page 92 of 140
Chapter 5
Rover Location Estimation by Matching Tree
Centers Maps Generated from Rover LiDAR to
Overhead Imagery
The third subsystem has the primary function of matching local tree center maps
obtained from ground LiDAR to a constellation map of stems derived from overhead
imagery. Least soares methods are employed by the algorilhm to find the best match
between map M and constellation map S. The best match is utilized to calculate the
absolute geoposition of the mobile robot. This algorithm is the last component of the data
processing chain, which starts with processing input imagery and LiDAR data to produce
an estimate of the horizontal geoposition of the mobile robot.
5.1
Map-Based Matching Hypothesis
As discussed in Section 1.2, the main goal of the matching subsystem is to align
local stem maps M observed by the mobile robot to the global stem map S of the
exploration area. The hypothesis of the matching subsystem relies on the central idea that
the horizontal geoposition of the mobile robot can be determined by calculating the
optimal rotation matrix and translation vector, R and F respectively, to transform map M
to S. The transformation is subsequently used to calculate an estimate for the horizontal
geoposition of the mobile robot. The following statements expand upon the central idea
just discussed:
*
The local map M at every pose is referenced with respect to the mobile robot's
local frame OQ. If relative stem locations observed by the mobile robot can be
located with respect to this frame, local maps M can be geometrically matched to
constellation map S.
Page 93 of 140
e
The constellation map S is referenced with respect to a global frame Os such as
UTM. Every element of the map S is therefore geopositioned with known
horizontal Easting and Northing coordinates.
e
To determine geoposition of the mobile robot, given a local map M and global
map S, a matching algorithm can determine the optimal rotation and translation
parameters that minimize the squared distance error between them.
Figure 36 illustrates the matching problem between local map M generated at
every pose and the global constellation map S. Two poses with their own local maps M
and Mi,+ are shown.
Global Constellation Map
S
*
(Xsy'
=
0
0
00
0
~0
Transformation Estimate
(RM,TM,)
)
RJ ,1
*
Transformation Estimate
{R
-
0
0
0
OU,
Pose: I -
Pose: 1+1
*
Local Map MM
Local Map M,
*
M, =(
y)
Ou
a
}*g
=
{xf,yI })
Figure 36: Matching scheme between local maps M; and M;+1 to global map S by finding the rotation and
translation parameters
The central hypothesis above and the aspects related to correlating input maps are
taken into consideration as part of the development of the matching algorithm as
discussed in Section 5.3.
Page 94 of 140
5.2
Overview of Map-Based Matching Algorithms
There has been extensive research into matching and registration of 2D and 3D
data for purposes of scene matching, object identification and vehicle localization. Most
of registration problems employ some variant of the Iterative Closest Point (ICP)
algorithm [61], and rely on Speeded Up Robust Features (SURF) [62], or correlation
methods such as Sum of Absolute Differences (SAD) [63]. Robust registration is a
specialized field that deals with handling outliers and noise in matching input data. A
robust registration method would be capable of discarding outliers and producing a best
fit based on statistical approaches. As discussed in Chapter 3 and Chapter 4, tree center
maps generated from overhead and ground LiDAR data could include outliers due to the
presence of complex crown profiles. Irregular blobs of crowns could lead to over
segmentation of overhead imagery. In similar fashion, dense underbrush could lead to
false labeling of tree stems observed in LiDAR data. Therefore, a robust registration
algorithm becomes important to avoid erroneous shifts in geoposition estimates.
SURF and SAD rely on correlating two input datasets by finding common
features in the data and maximizing the correlation coefficient. A candidate match is
found by determining the maximum correlation between the two input maps. However,
input datasets need to be of identical resolution for the correlation relationship to be
applied [63]. Figure 37 shows an illustration of the SAD matching process within the
context of mobile robot localization on the moon [63].
Correlation Peak
Figure 37: SAD correlation process using an input local map and a constellation map 1631
Although effective in a general sense, SAD is not useable for the purposes of this
thesis because the stem constellation map is typically of a different horizontal resolution
than the local maps produced from LiDAR data. Since the correlation function relies on a
one-to-one correspondence between data points in each map, different resolutions of
Page 95 of 140
input map cause the point-to-point correspondences to breakdown rendering the
algorithm ineffective.
ICP is a more flexible algorithm that allows the use of datasets of different
resolutions. Many variants of ICP have been developed that are geared towards robust
registration. Trimmed, Picky, or Gaussian Mixture Model (GMM) ICP algorithms
employ statistical methods, such as Gaussian probability distributions, to help identify
and discard outliers [64,65,66,67,68]. An example of the performance of a GMM ICP
algorithm is shown in Figure 38. The plot on the left shows two input maps before
alignment, while the plot on the right shows the same data after alignment. Units of the x
and y axes are in pixels.
250
0
250
+0
0
+ 0t
200-o
200-o
150
+0
8
++0
+
+
+
100
8
"
50 + 0
100
(
100-
0-150
-
15 - 6 +0
50
200
-P
50 100 150 200
Figure 38: An example of the performance of GMM ICP using two datasets (red and blue) that include outliers
and jitter 1681
Even in the presence of outliers, Trimmed, Picky and GMM ICP algorithms still
expect a large percentage of valid data to be of high quality and of low noise. In addition,
the correlation ratio between the horizontal locations of real stems between the two input
maps needs to be in the range of 0.8-1.0 in order for the algorithms to converge
[66,67,68]. Such constraints on data quality are impractical considering that stem maps
are generated from two different types of sensors; one from overhead satellite or aerial
imagery, while the other is from a rover mounted LiDAR sensor. In our case, the one-toone correspondence between stem points, even for real stems, can in theory be lower than
0.8.
A more flexible matching algorithm inspired by probabilistic ICP algorithms was
developed as part of this thesis. The algorithm accepts input data with no constraint on
resolution, size or percentage of outliers. The methodology of the robust matching
algorithm is described in Section 5.3.
Page 96 of 140
5.3
Data Matching Methodology
The main function of the matching algorithm is to estimate the translation and
rotation required to align the input maps. More precisely, the tree center constellation
map derived from the overhead image is treated as the baseline against which the LiDAR
based tree center map is matched to. This configuration is chosen because the map
obtained from the overhead image is usually larger than the local map generated from the
LiDAR dataset. Figure 39 illustrates the data matching methodology.
I
Stem Constellation S
I
Ground Stems M
Establish Search Space
Invoke Robust ICP
Calculate Location Based
on ICP Match
Absolute Horizontal
Localization Estimate
Figure 39: Data matching flowchart showing the major data processing steps
The following summarizes the major steps undertaken by the matching algorithm:
*
The algorithm first establishes the search space based on the input constellation
map S, local ground stem map M and the latest known geoposition of the mobile
robot.
*
Robust ICP is subsequently invoked, where it performs a systematic search to find
a geometric match between local map M to global map S. The match that
minimizes their Euclidean distance error is selected.
" The optimal match is then used to calculate the geoposition of the mobile robot
with respect to the global frame Os.
The following subsections detail each of the steps identified in Figure 39 starting
from providing maps S and M as input and ending with an estimate of the global position
of the mobile robot.
Page 97 of 140
5.3.1 Search Space
Before ICP is invoked, the search space is first defined and located with respect to
the global frame Os. This is necessary in order to reduce the likelihood of ICP converging
to a local minimum. Convergence properties of ICP are discussed in further detail in
Section 5.3.2. Figure 40 shows how the search space is defined with respect to the global
frame. Search space definition and processing involves two steps:
*
*
Search space positioning.
Step through search space.
Global Constellation Map S
.
0
0
1*
ESd=
r --
(xy
*Estimated
I
Search Space
0
tO
I
P
Rover Position
00
0
0
0
0
0
Search Space: Zoom in
*
a
---------
M,=
xf ,y
Linear Step
Angular Step
E
-~~
~ -I~
I............
35 m
Figure 40: ICP involves search space positioning based on last known robot geolocation and systematic search
that is comprised of linear and angular stepping
Page 98 of 140
Search Space Positioning
This step involves defining the location of the center of the search space with
respect to Os as well as the linear dimension of its sides. The locus of the center of the
search space is defined based on the latest robot geoposition estimate as given by the
algorithm, or from GPS if invoked for the first time. Depending on the distance traveled
from pose to pose, the search space center position needs to be adjusted for the difference
in distance traveled by the mobile robot. This distance is typically measured by an
onboard odometer. Practically speaking, if the distance traveled from pose to pose is
between I to 2 meters, there will be no need to adjust the center position of the search
space. In cases where the distance in any axis is larger, e.g. > 5 meters, then an
adjustment is necessary. The threshold of 5 meters was heuristically determined from
manual examination of the data. Lower thresholds can be used with no impact on
performance. In the context of the mission scenario and operations concept, the distance
traveled between each pose by the simulated mobile robot is 0.5 meters. Therefore, an
adjustment in the position of the search space is not required.
The search space position can be mathematically described as follows:
X
+xPD
1 +
'iyi
li-1 I
=
(YSS
<
x
YSSy
y.SS
>5
,OD
t i 1
1Yj
OD <
OD
-
<)
)
XXs
J
r
Where Xfs and Yissare the Easting and Northing locations of the center of the
search space box respectively at pose i. xPD and yPDare the distances measured by the
odometer along easting and northing respectively at pose i.
The metric dimensions of the search space are fixed at 35 m x 35 m for the
purposes of this thesis. This specific dimension was selected based on the mission
requirements as baselined for this research project.
Step Through Search Space
To reduce the likelihood of convergence by ICP to a local minimum, the input
local map Mat each pose is sequentially translated and rotated in discrete steps across the
search box. This process is shown in Figure 40. The translations and rotations are
performed at coarse discrete steps with ICP invoked at each step to find a match. In
essence, the rotations and translations are performed to coarsely align stem map M along
a candidate orientation and position, with ICP invoked to fine tune the match, if possible.
This approach significantly increases the likelihood of ICP finding the right match
especially when considering the presence of outliers. The specific performance of ICP
and its convergence properties are discussed in further detail in Section 5.3.2.
Page 99 of 140
The linear and angular steps through the search space are mathematically
described as follows:
Mfs = RsxM 5 s + t
k = tl,2,3,
... ns
Where Risand t55 are the coarse rotation matrix and translation vector at steps k
and j applied to input stem map Mfs at pose i, which is referenced to local frame OM.
nr is the maximum number of angular steps given the angular step size, while njs is the
maximum of number of linear steps given the search space size and linear step resolution.
nys
s, Rfsand ts are defined as follows:
-27T
0
40
-
- sinj0]
Rs=cosO
1sinjO
-l
=
(Ida
dy
cosjI
,f or translationalong Easting
, for
translaionalong Northing
Where 0 is the angular step size and dx and dy are the linear step sizes. The
angular step size chosen for this thesis is 10', which was determined heuristically based
on the data. In similar fashion, d. and dy were chosen to be 3 meters, which is lOx the
horizontal resolution of the image (0.3 m).
Computationally speaking, the search space stage involves two loops. The outer
loop is responsible for stepping linearly across the search space, while the inner loop
involves stepping angularly at each iteration of the outer loop. At each iteration of the
inner loop, ICP is invoked to provide an estimate of the geoposition of the mobile robot.
5.3.2 Robust Iterative Closest Point Algorithm
The ICP implementation follows the 2D rigid Besl-McKay (point-to-point)
framework given as follows [69]:
N
arg min Epq E
||IRxi
N2
+
- qi|| 2
i=1
Where E is the mean squared distance error between both input maps. R and F are
the rotation matrix and translation vector, respectively. p and q are points in the tree
Page 100 ol 140
center maps from the LiDAR and overhead image datasets, respectively. N is the total
number of points in the LiDAR tree center map M. A match is found by determining the
minimum error E.
The basic expression given above uses least squares to minimize the squared
distance error between both maps. This formulation is adequate for noiseless maps that
do not contain outliers. However, basic ICP will provide skewed results if noisy data is
used. A variant of ICP was developed for the purposes of this thesis that is robust against
moderate presence of outliers. The ICP variant uses the Euclidean distance error and
incorporates it into an iterative probabilistic best match search scheme. This scheme is
inspired by the RANdom SAmple Consensus algorithm (RANSAC) [70].
The matching framework uses a constant input stream of the mobile robot's
estimated pose as generated by the onboard GN&C subsystem. The ICP algorithm uses
the mobile robot's pose estimates as input for comparison to its matching results and tries
to minimize the error between them. To minimize the error in the presence of outliers, the
algorithm systematically discards suspicious outliers by iterating over the matching
process using randomly selected points from the LiDAR generated map M. The major
steps undertaken by the robust ICP algorithm are:
e
Step 1: Attempt a match using all points from map M.
*
Step 2: Compare location estimate from step I t- lAst known best estimate of th
location of the mobile robot.
" Step 3: Perform another matching attempt by systematically discarding one
candidate outlier point from map Mto minimize error from the previous step.
*
Step 4: Repeat step 3 until convergence or maximum iteration limit is reached.
Maximum number of iterations allowed is 100.
A detailed outline of the logic followed by the algorithm is given in Table 6.
Table 6: A robust algorithm for point-to-point matching of tree center maps in presence of outliers
Input: The stem constellation set S, the LiDAR stem set M and the rover pose x.
Output: The optimal transformation parameters R and t that best aligns M to S.
Begin:
" Place the origin of the search space at the current best estimate of the location of the
mobile robot.
Repeat:
* Place the set at the top left corner of the search space (origin).
- Invoke Besl-McKay framework to optimize the objective function E.
Page 101 of 140
- Compute the new estimated rover position based on the match between M and S.
" If Euclidean distance error E between new and previous vehicle positions is not
below threshold:
o
Randomly select one stern point from Mand temporarily set aside.
o
Update M to include one less stem point that was just selected and prepare to
discard if next iteration results in less error E, or keep if error increases.
Repeat Until:
Distance error E is below predefined threshold.
End
The distance threshold selected for the purposes of this thesis is equivalent to the
horizontal accuracy (RMSE) of the overhead imagery, which is 2.1 m. This is a fair
assumption given that the accuracy expected of the localization estimates should not
exceed the accuracies of the input data used. The maximum limit for the number of
iterations performed by ICP was heuristically set at 100.
It is noted that due to the local minima convergence property of ICP, the latest
known geolocation of the mobile robot is used as input in order to avoid local
convergence. This means that the best estimate of the location of the mobile robot is
given to ICP as a seed upon which matching is initiated. Subsequent poses are updated by
the mobile robot's GN&C subsystem based on the latest position estimates by ICP. In
cases where large jumps are observed in position estimates, the latest estimate with the
least deviation from the average is selected. It is noted that the design of the GN&C
subsystem is outside the scope of this thesis.
It is noted that Chapter 6 discusses the simulation of a mobile robot and its
onboard GN&C subsystem using a simple state-space model that incorporates
geoposition estimates from ICP.
5.3.3 Calculation of Mobile Robot Geoposition Based on Match
At the end of each matching step, ICP produces an estimate for R and i that
transform local map Mto global map S. The global position of the mobile robot is found
by applying the rotation and translation to M and then subtracting from that the relative
distances of each stem center to the mobile robot. This is mathematically described as
follows:
MfEO = RxMi + t
Page 102 of
140
X fEO =Z(M
EO,xMv)
j=1
YGEO
E(MGEO,yM)
j= 1
Where MfEOis the georeferenced ground stem center map after applying the
transformation to the relative map M at pose i. M GEOx is the n x 1 vector of the Easting
positions of each stem for pose i, while MGEO,yis the n x 1 vector of the Northing
positions of each stem for pose i.
n is the total number of stem center points. XEO and YGEO are the mean
geopositions of the mobile robot with respect to the global frame OS
5.3.4 Performance Analysis of Robust ICP Algorithm
To analyze the performance improvements of the robust ICP algorithm compared
to standard approaches such as those by Besl-McKay, several tests were perfbrmed using
sample aerial and LiDAR data from the Lake Mize site. Figure 41 and Figure 42 show a
sample run for a single rover pose. A sparse section of the constellation map S and
LiDAR data M was used to facilitate visual demonstration of the effectiveness of the
robust ICP algorithm.
The LiDAR data set M had a single outlier as shown in Figure 41, which also
shows the result from running the standard ICP algorithm. As shown, the estimated rover
position is different from the actual GPS (true) position. This is mainly due to the
presence of the single outlier that skews the least squares approach. The resultant distance
error for this test was 2.71 m compared to the actual GPS position.
Figure 42 shows the result after running robust ICP using the same dataset. The
closeness of both datasets shows that a good match was found. The outlier was discarded
following the logic discussed in Table 6 to produce a good match with a reported average
point-pair distance error of 0.37 m (~l pixel). It is noted that since maps S and M are
generated from two different sensors, it is anticipated that their constituent points may not
perfectly align with each other.
Page
103 of
140
0
LDAR-221
+
70
* *
0
0
75-
0
-
StemConstelMIonM(Overhead kage)
Stem Matching ResUis from StandardICP
Estimated Rover Position
Actual Rover Position
Stem Set S(Rover LDAR)
0
8+
80
0
85
0
0
0
0
(n
CM,
CM,
90
z
95
0
0
Outlier
-
100
0
0
105
0
0
00
+0
+
0
a
15I
-15
-0
-2 -20
-10I
I
-5
a
0
-
1 1 0n-25I
-10
5
10
Easting (m) + 382360 mn
Figure 41: Result from running standard ICP showing skewed position estimate due presence of one outlier
injected to map M
o Stem Consteiton M Ovwrhead mnge)
+ Stem Matching Resuhs fromn CP ftdant
LD)AR-221
,
70
*
*
0o
0
75
-
OP
Eeimated Rover Posion
Actual RovmrPoson
Stem Set S(RomrLIDAR)
0
9-
0
80
0
0
85
90-
+
0
0
0
950
0
-
100
0
0
0
-
105
0
oc
-5
-20
-15
0
-10
-5
Easting(m)+ 382360m
0
5
10
Figure 42: Result from running robust ICP with the same datasets where the outlier was automatically
identified by the algorithm and removed
Page 104 of 140
The robust ICP algorithm has the following properties:
1. The search space is constrained to a 35 m x 35 m box centered on the last known
position of the mobile robot and projected onto the stem constellation dataset S.
The optimum search space size was determined following an empirical
investigation of the accuracy and processing-time performance of ICP using
different search space sizes and shapes. In particular, a 35 m x 35 m search space
enabled us to maximize matching accuracy while minimizing processing time.
2. The ICP algorithm always provides a result and produces the best-found match
along with a mean error metric (parameter E). Good candidate matches produce
low E while inadequate matches result in a large E. The error metric E depends on
the final geometric configuration of both maps and the number of points in the
LiDAR stem center map. More stern points in a map translate to higher consensus
when calculating the mean position of the mobile robot.
3. The algorithm does not always guarantee better results than the standard ICIP
method. In particular, the algorithm may not perform better in situations where
there are moderate outliers present in the data. However, the algorithm will not
provide worse results than standard ICP, as it will default to the standard approach
if distance error E cannot be improved further.
Chnter 6 provides a more detailed assessment of the robust ICP algorithm using
experimental data acquired from a real-world environment. In addition, a DOE was
conducted to understand the sensitivities of key parameters of the robust ICP method
such as stern radius and number of stems observed per pose. These details arc discussed
in Chapter 6.
Page 105 of 140
-Blank-
Page 106 of 140
Chapter 6
Experimental Results
This chapter focuses on analyzing the performance of the localization algorithm
using real-world data. The different types of imagery and LiDAR data are discussed
along with the processing methodology adopted to prepare the different datasets for use
by the localization algorithm. To validate repeatability of the results by the localization
algorithm,
multiple runs were performed using the same data but with different
configurations. The performance of the algorithm was also analyzed using a sensitivity
analysis that examined the impact of variations of some parameters on the overall
accuracy of the algorithm.
An example of how the algorithm can be implemented as part of a Kalman Filter
state-space estimator is also discussed with the intent of showing the flexibility of the
algorithm in various use cases.
The localization algorithm was developed in Matlab and built using the standard
image processing toolbox and with an open source ICP implementation.
6.1
Introduction
6.1.1 Study Area and Data Preparation
Input Data
Survey-grade LiDAR data was collected at a test site northeast of Lake Mize in
Florida, USA (Coordinates: N29.738', W82.216'). Figure 43 shows the geographical
area of the test site. The data was acquired by a Leica ScanStation-2 LiDAR system that
was provided by the University of Florida. The LiDAR was placed at multiple survey
stations within the area bounded by the rectangular box in Figure 44. The area is
approximately 110 m x 110 m and the LiDAR dataset was collected with an average
spatial resolution of approximately 5 cm (on ground). The area has flat terrain and is
Page 107 of 140
4 high-resolution orthorectified images of the test site were acquired from USGS.
The images were provided in GEOTIFF format with three images captured in the visible
spectrum and one in NIR. The NIR image was acquired for backup but was never used in
testing. All images were subsequently projected on the UTM coordinate system. USGS
provides all images in Quarter Quadrangles format that defines the angular extents of
each image to be within 3.75 arc minutes (square). This is the equivalent of 0.0625' x
0.0625g.
Table 7 summarizes the key properties of the acquired data such as size,
resolution and accuracy. It is noted that Aerial Image 1 and Aerial Image 2 were acquired
of the same area but with different sensors and at different dates. Aerial Image 2 was
selected for our tests due to its higher resolution and the closer proximity of the test site
to the nadir (optical axis) of the image (< %/of Quart-Quad angular extent), thus reducing
parallax.
Table 7: Properties of input data (3 aerial images and one LiDAR point cloud)
Aerial
Image 1
Visible, 5000x5000
pixels,
2.1 m
Off-nadir Lower right corner.
Angular position > 0.031250
2.1 m
Close proximity to nadir. Angular
position < 0.03125'
2m
Off-nadir, lower right corner.
Angular position ~ 0.0625'
0.01 m
N/A
0.3 m resolution
Aerial
Image 2
Visible, 5000x5000
pixels,
0.3 m resolution
Aerial
Image 3
Visible, 8000x8000
pixels,
0.5 m resolution
LiDAR
5 cm resolution
(Average onground)
N/A: Not Applicable
Data Alignment
The same stems in the LiDAR dataset and Aerial Image 2 were observed to have
a systematic offset in their horizontal positions (Easting, Northing). Although the LiDAR
dataset was georeferenced using post-processed GPS techniques that guarantee
approximately 5 cm accuracy or better, the overhead imagery was orthorectified by
Page 109 of 140
4 high-resolution orthorectified images of the test site were acquired from USGS.
The images were provided in GEOTIFF format with three images captured in the visible
spectrum and one in NIR. The NIR image was acquired for backup but was never used in
testing. All images were subsequently projected on the UTM coordinate system. USGS
provides all images in Quarter Quadrangles format that defines the angular extents of
each image to be within 3.75 arc minutes (square). This is the equivalent of 0.06250 x
0.06250.
Table 7 summarizes the key properties of the acquired data such as size,
resolution and accuracy. It is noted that Aerial Image 1 and Aerial Image 2 were acquired
of the same area but with different sensors and at different dates. Aerial Image 2 was
selected for our tests due to its higher resolution and the closer proximity of the test site
to the nadir (optical axis) of the image (< 2of Quart-Quad angular extent), thus reducing
parallax.
Table 7: Properties of input data (3 aerial inages and one LiDAR point cloud)
Aerial
Image 1
Visible, 5000x5000
pixels,
2.1 m
Ott-nadir Lower right corner.
Angular position > 0.031250
0.3 m resolution
Aerial
Image 2
Visible, 5000x5OOO
pixels,
2.1 m
Close proximity to nadir. Angular
position < 0.03 125'
2m
Off-nadir, lower right corner.
Angular position ~ 0.0625'
0.01 m
N/A
0.3 m resolution
Aerial
Image 3
Visible, 8000x8000
pixels,
0.5 m resolution
LiDAR
5 cm resolution
(Average onground)
N/A: Not Applicable
Data Alignment
The same stems in the LiDAR dataset and Aerial Image 2 were observed to have
a systematic offset in their horizontal positions (Easting, Northing). Although the LiDAR
dataset was georeferenced using post-processed GPS techniques that guarantee
approximately 5 cm accuracy or better, the overhead imagery was orthorectified by
Page 109 of 140
USGS with a stated accuracy of ~2.1 m RMS. According to USGS, this accuracy metric
was calculated based on 4 independent measurements, each at one of the corners of the
image. It is theoretically possible that the actual accuracy of the image could be worse in
some areas and better in other areas. A linear horizontal alignment was therefore
necessary to facilitate accurate matching of stem objects in both maps.
XWor and Yo, denote the easting and northing corrections needed to align both
datasets. Those correction parameters were calculated manually by determining the pixelequivalent horizontal offsets between several crown and stem objects observed in both
datasets. 3 crown/stem objects were selected in both datasets and used to manually
calculate the average offset in pixels in the horizontal plane. X.o, and Y.o, were then
calculated by converting the pixel-equivalent distance to meters. X.o, was found to be
+1.3 m, while Y.o, was calculated to be -0.4 m. Even after the linear alignment, the stated
accuracy of the image of 2.1 m RMS was assumed as the baseline.
6.2
Simulation Setup
Without access to a real rover, and to simulate the envisaged operational scenario,
a ground vehicle was simulated in Matlab traversing different polygonal paths and using
the acquired LiDAR data as input. The simulated rover path is illustrated in red as shown
in Figure 45. More specifically, the acquired LiDAR data from the Lake Mize site was
gridded and incrementally fed at each pose to the localization algorithm. Figure 45 shows
the simulated rover path and all identified stem centers from the ground based LiDAR
dataset. 10 different paths were simulated with different stems observed by the mobile
robot from path to path. Section 6.3 shows the results from using all of the 10 different
paths.
At each pose, the LiDAR point cloud is sliced into 35 m x 35 m boxes centered on
each simulated pose of the rover. These slices are fed to the stem identification and center
estimation algorithm, which detects and labels stem centers for use by the matching
algorithm. The state-space vector of the simulated mobile robot is described as follows:
True Easting Coordinate
Xstate (Psi) =True
Northing Coordinate
sRelaive-No. of
Stems Observed
.Relative coordinatesof stems observed
Where Ps, is pose at i. It is noted that slicing of 3D data is performed prior to
invoking the matching algorithm.
Since the LiDAR dataset is of high accuracy, it was treated as the ground truth to
which geoposition estimates from the localization algorithm are compared against. More
specifically, the algorithm's estimated mobile robot position is plotted with the
corresponding GPS location derived from the overlay of the simulated robot path onto the
Page 110 of 140
LiDAR data. To assess the accuracy of position estimates, the horizontal position error
between the result provided by the localization algorithm and the GPS estimate is
calculated.
- -Simuided Rover Pakh
SLDAS Besed Stem Ceniters
100
80
90
0
0
"*
E
-0
~ 0 09
00
cc
000
0
-
40
zn
@D
oo
0O00
00 00
C
go
0
0
09
0
20
0
0
00
-2 :0L
0
10
20
30
40
50
Eastng (m)+382360 m
0008~
60
70
80
0~0
90
Figure 45: Simulated mobile robot path in red along with all stem centers identified by the stem identification
and center estimation algorithm
Prior to invoking the simulation, input parameters were set in the simulation
program as listed in Table 8.
Table 8: Input parameters to localization algorithm and simulation program
is
3.28
Overhead image rectilinear scale
pixels/m
t
13
Green contrast integer threshold
a-
1 pixel
Gaussian blur disk radius
diamnn
1.5 m
Minimum detectable crown diameter
diam~nca
12 m
Maximum allowable crown diameter
e
0.9
Maximum allowable eccentricity of crown object
Ne
20
Erosion and construction noise threshold
brh
2m
Breast height of tree stem
lb
35 m
Search space linear dimension
Page III of 140
Sm~in
4
Minimum number ot stem objects in single pose
et,
2m
Minimum distance error threshold between true mobile robot
position and estimate
Xcor
1.3 m
Systematic correction along Easting between LiDAR dataset
and input overhead orthophoto
Ye-01.
-0.4
Systematic correction along Northing between
dataset and input overhead orthophoto
iII('
100
Maximum number of iterations per ICP run
d4
3m
Linear step size along Easting inside search space by ICP
d,
3m
Linear step size along Northing inside search space by ICP
0
100
Angular step size inside search space by ICP
8,naX
400
Maximum angular search space window size by ICP
6.3
LiDAR
Results
10 full tests were performed where processing and labeling of input data were all
conducted by the localization algorithm. The tests were differentiated by the geometry of
the path taken by the mobile robot as part of the simulation. Due to the different
geometry of each path, slicing of LiDAR data was different. This meant that paths would
differ in the total number and geometry of stems observed from pose to pose. Figure 46
to Figure 65 show the localization results for all 10 of the different paths as reported by
the vision based localization algorithm. To characterize accuracy, the position error for
each pose was calculated based on the geoposition estimate from the localization
algorithm and the true position of the mobile robot as simulated in Matlab. The Euclidean
position error is defined by:
Errori =
(x
- x ) 2 + (y
- y )2
Where x"i and y"i are the absolute (true) Easting and Northing coordinates of the
mobile robot at pose i, while x'i and YWi are the algorithm's estimated Easting and
Northing coordinates of the robot at pose i.
For each simulation, the simulated mobile robot's true path is plotted along with
the estimated position as provided by the localization algorithm. A histogram showing
the distribution of the position results is also given for each path.
Page 112 of 140
.................
. .......
Path 1
MU
-
A ctual position of simulated rover
Estimated position by
localization
algorithm
70
F
60
50
-
E
0
C")
-
40
4-.
30
-
0
Z
20
-
10
0'0
I
I
10
20
I
I
30
40
50
Easting (m) + 382360 m
I
I
60
70
80
Figure 46: Geoposition results for path I comparing true mobile robot position to estimates by the localization
algorithm
60
50-
40C
0
Z 30E
z
-
20
10_-
0L
0
1
2
3
4
5
6
Position Error (m)
Figure 47: Histogram of position error results as provided by the localization algorithm for path 1
Page 113 of 140
Path 2
80
A ctual position of simulated rover
E)
Estimated position by
localization algorithm
70-
60
-
E
8
500
40
30
20
20
K
-
Z
0
10
20
30
40
50
60
70
80
Easting (m) + 382360 m
Figure 48: Geoposition results for path 2 comparing true mobile robot position to estimates by the localization
algorithm
80-
I
I
I
I
I
I
4
5
b
I
I
70-
60-
'
50
0 40
S30
20
10
0
0
3a
8
9
Position Error (m)
Figure 49: Histogram of position error results as provided by the localization algorithm for path 2
Page 114 of 140
Path 3
75
t
-e
70
Actual position of simulated rover
Estimated position by localization algorithm
65
E
60
55
zO0 50
45
40
35
30'
15
I
I
I
20
25
30
I
i
35
40
Easting (m) + 382360 m
I
I
I
45
50
55
60
Figure 50: Geoposition results for path 3 comparing true mobile robot position to estimates by the localization
algorithm
35-
30-
254"
200
15
lOFo
5
0
0
1
2
3
4
Position Error (m)
5
6
7
1
8
Figure 51: Histogram of position error results as provided by the localization algorithm for path 3
Page 115 of 140
Path 4
-- ActuaI position of simulated rover
E)Estimated
position by localization
algorithm
60-
50E
o 40-
0
2
0
10k
0'-10
20
30
40
50
Easting (m) + 382360 m
60
70
80
Figure 52: Geoposition results for path 4 comparing true mobile robot position to estimates by the localization
algorithm
90
80
70
60
50
0
40
z
30
20
10
0
0
2
3
4
5
6
7
8
9
10
Position Error (m)
Figure 53: Histogram of position error results as provided by the localization algorithm for path 4
Page 116 of 140
Path 5
75
()A ctual position of simulated rover
E)
Estimated position by localization algorithm
70
65
E
60
55
50
02
0
45
40
35
30 L
10
20
40
Easting (m) + 382360 m
30
50
60
70
Figure 54: Geoposition results for path 5 comparing true mobile robot position to estimates by the localization
algorithm
50
I
I
I
I
I
3
4
5
6
I
I
7
8
45403530-
0
2520-
Z
1510-
5
0L
0
-1
2
9
Position Error (m)
Figure 55: Histogram of position error results as provided by the localization algorithm for path 5
Page 117 of 140
Path 6
60
---e-
Actual position of simulated rover
Estimated position by localization algorithm
50-
+
E
40-
3 0-
CVco
0
2 0-
I0
--
10
20
30
40
50
60
70
80
Easting (m) + 382360 m
Figure 56: Geoposition results for path 6 comparing true mobile robot position to estimates by the localization
algorithm
I
60
Z
40
20
0
0
2
4
6
8
10
12
14
Position Error (m)
Figure 57: Histogram of position error results as provided by the localization algorithm for path 6
Page 118 of 140
Path 7
80
-
-
A ctual position of simulated rover
Estimated position by localization algorithm
70
60
E
o 50 Pc'l
E
S40
I-
I
-
0
Z
30-
20-
10
2(
25
30
35
45
40
Easting (m) + 382360 m
50
55
60
65
Figure 58: Geoposition results for path 7 comparing true mobile robot position to estimates by the localization
algorithm
40,
I
I
2
3
I
I
I
4
0
6
35-
30-
25C
0
20-
Z 1510-
-
5
-
0
0
7
8
9
Position Error (m)
Figure 59: Histogram of position error results as provided by the localization algorithm for path 7
Page 119 of 140
Path 8
80
A ctual position of simulated rover
Estimated position by localization
algorithm
70-
60E
8
50CO
-E 400
Z
30-
20-
100
20
10
30
40
Easting (m) + 382360 m
50
60
70
Figure 60: Geoposition results for path 8 comparing true mobile robot position to estimates by the localization
algorithm
70
60-
-
50
C
1
40
0
1 30
z
20
10
01
0
1
2
3
4
Position Error (m)
5
6
7
8
Figure 61: Histogram of position error results as provided by the localization algorithm for path 8
Page 120 of 140
Path 9
80
---E
Actual position of simulated rover
Estimated position by localization algorithm
70
60
E
o
50
040
-
I
0
0
20
in
30
35
40
45
65
50
55
60
Easting (m) + 382360 m
70
75
80
Figure 62: Geoposition results for path 9 comparing true mobile robot position to estimates by the localization
algorithm
I
I
I
I
I
I
I
4
5
b
7
50 S
40-
o
z
30-
20-
10-
00
a3
8
Position Error (m)
Figure 63: Histogram of position error results as provided by the localization algorithm for path 9
Page 121 of 140
. ........
.............
Path 10
Ou.
--- Actual position of simulated rover
- -Estimated position by
localization algorithm
70-
60E
50-
N-.-
--
C
-
40
0
Z
-
30
-
20
-
10
10
20
30
40
Easting (m) + 382360 m
50
60
70
Figure 64: Geoposition results for path 10 comparing true mobile robot position to estimates by the localization
algorithm
70
F
60
50
40
0
30
z
20-
-
10
0L
0
2
3
4
5
6
Position Error (m)
7
8
9
10
Figure 65: Histogram of position error results as provided by the localization algorithm for path 10
Page 122 of 140
6.3
Accuracy Assessment
Results in Section 6.2 show that, overall, there is a good agreement between true
and estimated position data. Estimates from the localization algorithm generally follow
the true measurements with some minor deviations. In particular, there is a problematic
area lying between the 30 and 60 Easting lines towards the top of the polygon traverse
paths. Estimates from the localization algorithm for those areas do not agree well with
true positions (3-4 m mean error). Manual inspection of LiDAR data revealed that the
problematic areas contain moderate stands of thick and tall (>2 m height) underbrush that
obscure tree stems and lead to false labeling. The moderate presence of false positive
stems limits the success of the ICP algorithm regardless of how robust it is. If there are
many more false positives than true stems in the data, it becomes challenging to
aggregate enough consensus in order to obtain a robust position estimate.
The mean position error, standard deviation and Root Mean Squared (RMS) error
for all test runs are summarized in Table 9. The mean robot position error for all test runs
agrees to within +/- 1 pixel (~ 0.33 in), which is theoretically acceptable given the 1-foot
resolution of the aerial image used. In particular, considering the horizontal accuracy of
the aerial image of 2.1 m RMS, the position accuracy results show very similar
performance. It is noted that due to occlusion by underbrush, some tree stems were not
detected in several poses. These poses were therefore discarded and shown as gaps in the
mobile robot path data as seen in Section 6.2.
Table 9: Accuracy results of localization algorithm for all test runs 1-10
1
234
1.511
0.893
1.754
2
202
1.569
1.354
2.071
3
132
1.553
1.107
1.904
4
211
1.597
1.376
2.106
5
145
1.653
1.311
2.106
6
193
1.490
1.554
2.149
7
104
1.725
1.460
2.255
8
196
1.520
1.259
1.970
9
181
1.669
1.100
1.987
10
191
1.722
1.596
2.344
Page 123 of 140
.
.......
.............
..........
Figure 66 shows the profile of position errors, including mean error, standard
deviation and RMS for all paths 1-10.
1.5
"'"Mean Position Error (m)
0.5
1
"Standard Deviafion of Position Error
Measurements
-
0
2
4
Path 6
8
RMS of Position Error
10
12
Figure 66: Position error results including mean error, standard deviation and RMSE produced by localization
algorithm for all paths 1-10
Table 9 and Figure 66 show that the algorithm produces consistent performance
using different configurations (paths) of LiDAR data as input. The variations in the error
reported from path to path are attributed to the different number of stems reported at each
pose as well as the percentage of false positives observed.
Three factors play a major role in determining the positional accuracy of the
localization algorithm:
* Accuracy of the input datasets (aerial and LiDAR). Some trees that are visible in
the LiDAR dataset may not have visible crowns in aerial imagery due to
occlusions by taller trees.
e
Validity of tree stem centers estimated from aerial imagery. Tree crowns tend to
merge in dense forests rendering the task of delineation somewhat tricky and not
without uncertainty.
e
Validity of tree stem centers estimated from LiDAR. Some trees that are visible in
the aerial image may not be visible in the LiDAR dataset due to occlusions by
other trees, underbrush or other obstructions.
The second and third factors are very important because they can easily affect the
positional accuracy reported by the algorithm. If several tree centers were mislabeled, the
Page 124 of 140
.....
........
matching algorithm would have difficulties finding the true matches. In cases where a
moderate number of erroneous tree centers are present in a dataset, ICP would certainly
provide a biased result.
6.4
Sensitivity Analysis
To characterize the parameter sensitivity of position estimates produced by the
localization algorithm, a sensitivity analysis in the form of a DOE was conducted. The
key parameters chosen were the number of stems per pose reported by the stem
identification and center estimation algorithm and the minimum stem radius. A single
dataset was used for the sensitivity analysis to serve as a canonical case for the thesis.
Path 1 data was used for the purposes of the DOE because it had the most number of
poses compared to the rest of the path data.
The DOE process is described as follows:
* Outer loop: Iterate over number of stems per pose (Range: 2-20, increments of 2)
* Inner loop: Iterate over minimum stem radius (Range: 0-0.1 m, 0.15, 0.2,
increments of 0.2)
e
At each iteration of the inner loop, invoke the localization algorithm based on the
established parameters and record the overall RMSE.
It is noted that all other parameters are kept the same as detailed in Table 8.
Figure 67 shows the DOE results with the number of stems per pose and minimum stem
radius forming the x and y axes, respectively, and estimated position error (RMSE) on the
z axis.
DOE Results (Number of Stems per File vs. Stem Radius vs. Estimated Position Error)
10
-
.
o
0
o
2
o
.....
0.2
2
0
Estiate
2145
Ste
RaiusNumber
of Stems per
File
Figure 67: 2 parameter DOE results versus estimated position error (RMSE) for path
Page 125 of 140
1 data (canonical case)
A Pareto front is clearly seen that minimizes the estimated position error for a
given range of values for the number of stems per pose and the minimum stem radius.
Figure 68 and Figure 69 show the relationship between estimated position error and each
of the DOE parameters: number of stems per pose and minimum stem radius.
DOE Results (Number of Stems per File vs. Stem Radius vs. Estimated Position Error)
12
10
0
8
...........
......
6
....... ..
*
..................
........
Qa
0
E
0
4
0
.o
F
:0
00
4
2
8
6
10
12
Number of Stems per File
16
14
18
20
Figure 68: Impact of number of detected stems per pose on RMSE for path I (canonical case)
DOE Results (Number of Stems per File vs. Stem Radius vs. Estimated Position Error)
- -.--.
-
12 --
10 --4 .. . .. .. . .. . .:
0
..........
...............
............
. 0..
.
- - .....
-.......
6-
...........
4-
...........
.
0
- -- - - - - - -
-
L
-
8-
..........
0.
2
00
0
0
II I
I
I
I
2002
0.18
0.16
0.14
I
I
I
0.12
0.1
0.08
Estimated Stem Radius
I
I
0.06
0.04
Figure 69: Impact of minimum stem radius on RMSE for path
1 (canonical
1
0.02
1
0
case)
Figure 68 shows that a minimum of 4 stems per pose is needed to attain position
errors of less than 5 m. The threshold of 5 m was specified in the mission operations
concept for this thesis. More stems per pose may be required to achieve higher accuracies
Page 126 of 140
as implied by the plot. This relationship agrees with the mechanics of the matching
process. More stems per pose translate to higher consensus in the correlation routine
between map M and S, and thus higher position accuracy can be expected. A higher
threshold is possible at the expense of discarding more stems that do not pass that
threshold. This becomes a common issue when considering sparse areas. Based on the
above, the algorithm has baselined a minimum of 4 stems per pose. Poses that contain
less than 4 stems are discarded.
Figure 69 shows that the overall position error does not change significantly as
the minimum radius is reduced from 0.08 m to 0.01 m. That means that the minimum
stem radius does not have a direct effect on the performance of the localization algorithm
as long as the minimum stem radius is kept below 0.08 m. It is therefore concluded that
the minimum stem radius should be set at 0 m i.e. not used.
6.5
Kalman Filter Application
The localization algorithm can be integrated to online filtering techniques such as
recursive least squares or Kalman Filter frameworks. The benefit of such integration is to
produce continuous and smoothed best estimates of the geoposition of the mobile robot.
The localization framework can be treated as a standalone geoposition sensor that
supplies location estimates at each pose of the mobile robot. Figure 70 shows the results
of integrating the localization algorithm to a Discrete Kalman Filter (DKF) framework.
--E-
320G X 10
GPS
1CP
- -KF
-
3.2904
-
3.2904
-
3.2904
-
3.2904
-
3.2904
-
3.2904
-
3.2904
-
3.2904
3.2235
3.8236
3.8237
3,8238
3.8239
3.824
Easting (m)
3.8241
3.8242
3.8243
3.8244
X 10 5
Figure 70: Unfiltered geoposition estimates by the localization algorithm for path 1 data (ICP in blue) as
compared to the filtered version using a DKF (KF in green) and the true position data (GPS in red)
Page 127 of 140
The standard DKF framework calculates error covariance matrices and updates
them as new input data is provided. The state space model is comprised of the
geoposition of the mobile robot along Easting and Northing. In order to estimate the
optimal geoposition of the mobile robot, the DKF uses the calculated error covariance
matrices to estimate the a priori state space model and propagate it. Data from the
subsequent pose are used to update the state space model post priori.
Figure 70 shows a canonical case of a DKF implementation using the localization
algorithm as input. Data from the "path 1" dataset was used for demonstration. The plot
shows the filtered geoposition estimates using the DKF framework along with the
unfiltered geoposition estimates from the localization framework and the true position. It
is clear that the DKF method produces a more stable output compared to the raw
unfiltered geoposition estimates by the localization algorithm. It is noted that the raw
unfiltered geoposition estimates were produced by the standard ICP algorithm and not the
robust version. The intent was to show the marked improvement of utilizing a DKF
approach compared to traditional methods, which would not have been easily seen if data
from the robust ICP algorithm were used instead.
In summary, compared to a raw position accuracy of 2.4 m with 1.8 m standard
deviation for the "path 1" dataset, the DKF approach achieved ~1 m standard deviation,
which is a significant improvement. However, the mean error position reported by the
tO the
state
space propagaLi11 CnU UdaLe
LJK
1frameVVork
VVts
12.
m. This
1s due
process of the DKF framework that produces a systematic linear offset between the raw
and DKF estimates. Therefore, there is a tradeoff that needs to be considered when a
DKF framework is applied. Although effective in smoothing the raw position estimates,
the DKF framework can introduce systematic offsets that degrade the overall accuracy.
This section serves to demonstrate the flexibility of the localization algorithm in
terms of integration to state space models and online filtering techniques. The method is
envisioned to be utilized standalone but could be integrated readily with other methods
and/or sensors to provide smoothed estimates for real-time navigation purposes.
Page 128 of 140
-Blank-
Page 129 of 140
Chapter 7
Thesis Summary, Conclusions and
Recommendations
7.1
Thesis Summary
Absolute localization of mobile robots without the use of GPS is an area currently
undergoing
intensive research. Although GPS is considered an effective and accurate
geolocation technology, its signals are easily attenuated by canopies in forest
environments, or dithered and spoofed by adversaries in areas of major conflicts. In this
context, a novel algorithm is presented by this thesis that provides accurate
geopositioning of mobile robots without relying on GPS. The algorithm utilizes vision
data acquired by sensors onboard the mobile robot and compares them to absolute maps
loaded into the robot a priori.
Chapter I gave a summary of the overarching goals of the thesis and the
fundamental concepts related to the problem of determining the absolute geoposition of a
vehicle. The top level goals of the thesis were then expanded in Chapter 2 to develop
candidate mission operations concepts for a mobile robot operated in two main modes,
teleoperation and autonomous. In particular, the use context of the localization scheme
was examined to identify likely scenarios and applications that would utilize vision based
localization. The chapter concluded by specifying a set of operations and functional
requirements that detail scenarios, operations aspects and performance parameters for
mobile robot localization in dense forest environments.
Based on the operations concepts and requirements, a top-level architecture of the
vision based localization framework was defined in Chapter 2. The architecture captured
the important functional building blocks required to perform localization of a mobile
robot. This includes defining the subsystems (modules) to be integrated as part of the
Page 130 of 140
overall framework. In addition, inputs and outputs at the top-level of the framework are
defined in order to clarify how the framework can be integrated to mobile robots and
their GN&C subsystems.
The automatic tree crown identification, delineation and extraction component is
discussed in Chapter 3. This module is responsible for estimating tree crown centers, and
hence tree stems, from aerial or orbital imagery. Various image processing algorithms are
implemented and explained, most notably, the watershed segmentation algorithm that
enables unambiguous delineation of tree crowns. The output from this module is a global
map comprised of detected tree stems (stem map S).
Chapter 4 discusses the methodology of generating local maps M at each pose of
the mobile robot. The algorithm's design borrows heavily from prior work by Matt
McDaniel of the Robotic Mobility Group at MIT. McDaniel's original algorithm was
customized and integrated into the vision based algorithm. The customization involved
adjusting parameters related to the breast height brh threshold as well as the search
windows for candidate main stem data. Local maps generated by this component are used
for subsequent matching with the global map S for localization purposes.
The last component of the vision based algorithm is the matching of locally
generated maps to global maps of the area explored by the robot. Chapter 5 discussed the
design of a robust ICP algorithm that performs matching and alignment of locally
generated maps of tree stems to global stem maps estimated from overhead imagery. The
main feature of the robust ICP algorithm developed in this thesis is its ability to handle
and discard outliers (false positive stems identified in local stein maps A/). The method
employs an exhaustive search approach that systematically eliminates outliers by
minimizing resultant position error between the best-known robot position and that
estimated by the algorithm. A canonical example with and without outliers is discussed
along with a comparison showing the superior performance of the robust ICP method
compared to standard ICP frameworks.
Several tests were undertaken using real-world data to validate and verify the
performance and functionality of the localization framework. Chapter 6 discusses the
methodology adopted to collect, process and validate performance using real-world data
fed to a simulated robot in a dense forest near Lake Mize in Florida, USA. Different test
regimens were used to assess performance. Data from the Lake Mize site consisted of
high-resolution LiDAR scans of the forest grounds coupled with open-source highresolution aerial imagery of the same area. To characterize accuracy, position estimates
from the vision based localization algorithm were compared to true measurements of the
simulated mobile robot. RMS errors of approximately 2 m were observed for all paths,
which agreed well with the accuracy of the aerial imagery used (2.1 m RMS).
Page 131 of 140
7.2
Contributions
This thesis develops and validates a novel approach to the geolocation of
terrestrial mobile robots operating outdoors in dense forest environments. The following
specific thesis contributions can be identified:
-
Developed a methodology and associated algorithm for the processing of vision
based overhead imagery and LiDAR data to extract tree crown and stem objects.
-
Developed methodology and associated algorithm for the geometric matching of
local stem maps observed by a ground LiDAR to overhead constellation stem
maps of the exploration area.
" Developed methodology and associated algorithm for the calculation of the
geoposition of a mobile robot based on matching local stem maps to overhead
stem constellation maps.
-
Produced and tested a software toolbox written in Matlab that allows mobile robot
geoposition estimation based on input steams of real world data (imagery and
LiDAR).
*
Validated and verified performance of the geolocation algorithm
experimental real world data of a forest around Lake Mize, FL, USA.
7.3
using
Limitations
The limitation of the localization method is its reliance on the best estimate of the
robot's geoposition in order to constrain the search space. This constraint is limiting
within the context of a kidnapped robot scenario where there are no prior estimates of the
geoposition of the mobile robot. In such a scenario, the vision based localization
algorithm fails because the search space cannot be specified. A global search would be
required in order to estimate geoposition. This is a complex and time-consuming
computation task that involves processing a large library of overhead imagery covering
the globe. In theory this is a feasible undertaking but the computational requirements and
processing times may yield significant penalties that render the utilization of this
algorithm prohibitive and impractical.
7.4
Recommendations for Future Work
Recommendations for future work are divided into two main aspects: short-term
enhancements and long-term improvements. Short-term enhancements include the
following:
Pagc 132 of 140
"
Utilize higher resolution overhead imagery of the exploration area. The thesis
project used an overhead image with 0.33 m (1 foot) resolution. The accuracy of
the delineation algorithm can be further improved by using higher resolution
imagery e.g. sub decimeter. Higher resolution imagery can lead to increased
crown classification accuracy because of the higher detail attained in delineating
the edges of crowns.
-
Obtain and utilize higher accuracy (RMSE) overhead imagery. In similar fashion
to the previous point, a higher accuracy image with sub-meter RMSE should
significantly improve the overall geolocalization accuracy of the algorithm.
Long-term improvements include the following:
Implement a more robust delineation and stem identification algorithms to reduce
or eliminate the percentage of false positive stems identified in the overhead
imagery and LiDAR data. This would involve utilizing 2D and 3D tree
morphology models (templates) that can be used for identifying trees from
underbrush.
Page 133 of 140
-Blank-
Page
134 of 140
References
[1]
C. Kee, B.W. Parkinson, P. Axelrad, "Wide Area Differential GPS", Navigation,
Vol. 38, No. 2, pp. 123-146, 1991.
[2]
B. Liu, M. Adams, J. Ibanez-Guzman, "Multi-aided Inertial Navigation for Ground
Vehicles in Outdoor Uneven Environments," IEE International Conference on
Robotics and Automation ICRA, pp.4703-4708, 2005.
[3]
P. Sigrist, P. Coppin, M. Hermy, "Impact of forest canopy on quality and accuracy
of GPS measurements", International Journal of Remote Sensing, 20(18), 35953610, 1999.
[4]
B. Barshan, and H.F. Durrant-Whyte. "Inertial navigation systems for mobile
robots.", IEEE Transactions on Robotics and Automation, pp. 328-342, 1995.
[5]
J.J. Leonard, H.F. Durrant-Whyte, "Simultaneous map building and localization for
an autonomous mobile robot," IEEE/RSJ International Workshop on Intelligent
Robots and Systems - Intelligence for Mechanical Systems, vol. 3, pp.1442-1447,
1991.
[6]
J. Guivant, F. Mason, E. Nebot, "Simultaneous Localization and Map Building
Using Natural Features and Absolute Information", Robotics and Automation
Systems 40, pp. 79-90, 2002.
[7]
D. Nistdr, 0. Naroditsky, J. Bergen, "Visual odometry for ground vehicle
applications", Journal of Field Robotics, 23(1), 3-20, 2006.
[8]
S. Se, D. Lowe, J. Little, "Global localization using distinctive visual features",
IEEE/RSJ International Conference on Intelligent Robots and Systems, Vol. 1, pp.
226-23 1, 2002.
[9]
J. A. Castellanos, J.D. Tardos,"Mobile robot localization and map building: A
multisensor fusion approach", Kluwer academic publishers, 2000.
[10] Y. Fuke, E. Krotkov, "Dead reckoning for a lunar rover on uneven terrain". IEEE
International Conference on Robotics and Automation, Vol. 1, pp. 411-416, 1996.
[11] R. Li, B.A. Archinal, R. E. Arvidson, "Spirit rover localization and topographic
mapping at the landing site of Gusev Crater, Mars", Journal of Geophysical
Page 135
of
140
Research, 11 1(E2), 2006.
[12] S. Panzieri, F. Pascucci, G. Ulivi,. "An outdoor navigation system using GPS and
inertial platform", IEEE/ASME Transactions on Mechatronics, 7(2), 134-142, 2002.
[13] A. Georgiev,P. K. Allen, "Localization methods for a mobile robot in urban
environments", IEEE Transactions on Robotics, 20(5), 851-864, 2004.
[14] B. Hofmann-Wellenhof,
System", Springer, 2000.
H. Lichtenegger, H, J. Collins, "Global
Positioning
[15] P. Misra, P. Enge, "Global Positioning System: Signals, Measurements and
Performance Second Edition", Massachusetts: Ganga-Jamuna Press, 2006.
[16] B.W. Parkinson, J. J. Spilker,"Global Positioning System: part 1 GPS fundamentals,
part 2 GPS performance and error effects; vol. 2, part 3 differential GPS and
integrity monitoring, part 4 integrated navigation systems, part 5 GPS navigation
applications, part 6 special applications (Vol. 2)", AIAA, 1996.
[17] P. Bolstad, A. Jenks, J. Berkin, K. Horne, H.W. Reading, "A comparison of
autonomous, WAAS, real-time, and post-processed global positioning systems
(GPS) accuracies in northern forests." Northern Journal of Applied Forestry 22, no.
1, 2005.
[18] K. Ohno, T. Tsubouchi, S. Yuta, "Outdoor map building based on odometry and
RTK-GPS positioning fusion", IEEE International Conference on In Robotics and
Automation ICRA'04, Vol. 1, pp. 684-690), 2004.
[19] P. Sigrist, P. Coppin, M. Hermy, "Impact of forest canopy on quality and accuracy
of GPS measurements". International Journal of Remote Sensing, 20(18), 35953610, 1999.
[20] M.S. Braasch, "GPS multipath model validation". IEEE Position Location and
Navigation Symposium, pp. 672-678, 1996.
[21] M. Mostafa, J. Hutton, B. Reid, "GPS/IMU products-the Applanix approach",
Photogrammetric Week, Vol. ], pp. 63-83, 2001.
&
[22] Y. Cheng, M.W. Maimone, L. Matthies, "Visual odometry on the Mars exploration
rovers-a tool to ensure accurate driving and science imaging", IEEE Robotics
Automation Magazine, 13(2), 54-62, 2006.
[23] H. Bay, T. Tuytelaars, L. Van Gool, "Surf: Speeded up robust features", Computer
Vision-ECCV 2006 (pp. 404-417). Springer Berlin Heidelberg, 2006.
[24] D.G. Lowe, "Object recognition from local scale-invariant features".
international conference on Computer vision, Vol. 2, pp. 11 50-1157, 1999.
Page 136 of 140
IEEE
[25] J.P. Grotzinger, J. Crisp, A.R. Vasavada, R.C. Anderson, C.J. Baker, R. Barry, R.C.
Wiens,
"Mars Science
Laboratory mission
and science
investigation",
Space
science reviews, 170(1-4), 5-56, 2012.
[26] A. Howard, "Real-time stereo visual odometry for autonomous ground vehicles".
IEEE/RSJ International Conference on Intelligent Robots and Systems IROS 200,
pp. 3946-3952, 2008.
[27] K. Nishino, K. Ikeuchi, "Robust Simultaneous Registration of Multiple Range
Images", 51h Asian Conference on Computer Vision, 2002.
[28] M. Song, F. Sun, K. lagnemma,
Forested
Landmark Extraction in Cluttered
International
IEEE
Environments",
"Natural
Conference
on
Robotics
and
Automation (ICRA), pp.4836-4843, 2012.
[29] J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. KammelS. Thrun,
"Towards fully autonomous driving: Systems and algorithms", IEEE Intelligent
Vehicles Symposium (IV), pp. 163-168, 2011.
[30] B. Planitz, A. Macder, J. Williams, "The correspondence framework for 3D surface
matching algorithms" Computer Vision and Image Understanding, 97(3), 347-383,
2005.
r 31]
A
NAian
NA
Bennniin
R.
s
"
A idu-mticli
rrespfnellnce fnr IF) mdingo
An extensive review", International Journal of Shape Modeling, 11(2), 253-291,
2005.
[32]
K. Mikolajczyk, C. Schmid, "A performance evaluation of local descriptors", IEEE
Transactions on Pattern Analysis and Machine Intelligence, 1615-1630, 2005.
[33] S. Se, H.K. Ng, P. Jasiobedzki, T.J.
localization
for
planetary
exploration
Moyung, "Vision based modeling and
rovers",
Proceedings
of
International
Astronautical Congress (pp. 434-440), 2004.
[34] Z. Zhang, "lterative point matching for registration of free-form curves and
surfaces", International Journal of Computer Vision, 13(2), 119-152, 1994.
[35] A. Hayashi, T. Dean, "Satellite-map position estimation for the Mars rover",
Proceedings of the NASA Conference on Space Telerobotics, Vol. 2, pp. 275-282,
1989.
[36] F. Stein, G. Medioni, "Map-based localization using the panoramic horizon", IEEE
Transactions on Robotics and Automation, 11(6), 892, 1995.
[37] A. Behar, J. Matthews, C. Raymond, E. Means, "The JPL PAUSE aerobot". The
Page 137 of 140
International Conference on System, Man and Cybernetics, Vol. 4, p. 3939-43,
2005.
[38] S. Ramalingam, S. Bouaziz, P. Sturm, M. Brand, "Skyline2gps: Localization in
urban canyons using omni-skylines", IEEE/RSJ International Conference on
Intelligent Robots and Systems (IROS), 2010, pp. 3816-3823,
0 10.
[39] A.L. Majdik, Y. Albers-Schoenberg, D. Scaramuzza, "MAV urban localization
from Google street view data", IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS) 2013, pp. 3979-3986, 2013.
[40] H. Kretzschmar, C. Stachniss, C. Plagemann, W. Burgard, "Estimating landmark
locations from geo-referenced photographs". IEEE/RSJ International Conference
Intelligent Robots and Systems IROS 2008, pp. 2902-2907, 2008.
[41] P.J. Carle, P. T. Furgale, T.D. Barfoot, "Long-range rover localization by matching
LIDAR scans to orbital elevation maps", Journal of Field Robotics, 27(3), 344-370,
2010.
localization and mapping"
[42] S. Thrun, J.J. Leonard, "Simultaneous
handbook of robotics, 871-889, 2008.
Springer
[43] H. Durrant-Whyte, T. Bailey, "Simultaneous localization and mapping: part I",
IEEE Robotics & Automation Magazine, 13(2), 99-110, 2006.
[44] R. Iser, A. Martens, F.M. Wahl, "Localization of mobile robots using incremental
local maps". IEEE International Conference on Robotics and Automation (ICRA)
2010, pp. 4873-4880, 2010.
[45] W.C. Wright, P.W. Liu, K.C. Slatton, R.L. Carter, H. Lee, "Predicting L-Band
Microwave Attenuation Through Forest Canopy Using Directional Structuring
Elements and Airborne LiDAR", Geoscience and Remote Sensing Symposium,
IGARSS, 2008, vol.3, pp.I 1 - 688,111 - 691, 2008.
[46] G.S. Biging, M. Dobbertin, "A comparison of distance-dependent competition
measures for height and basal area growth of individual conifer trees", Forest
Science, 38(3), 695-720, 1992.
[47] K. Mechelke, T. P. Kersten, M. Lindstaedt, "Comparative investigations into the
accuracy behaviour of the new generation of terrestrial laser scanning systems",
Proc. in the Optical, 3 19-327, 2007.
[48] M. Larsen, M. Rudemo, "Estimation
of tree positions from aerial photos",
Page 138 of 140
Proceedings of the 1997 Swedish Symposium on Image Analysis pp. 130-134,
1997.
[49] R. J. Pollock," The Automatic Recognition of Individual Trees in Aerial Images of
Forests Based on a Synthetic Tree Crown Image Model", PhD thesis, Computer
Science, The University of British Colombia, Vancouver, Canada, 1996.
[50] R. J. Woodham and M. H. Gray, "An analytic method for radiometric correction of
satellite multispectral scanner data", IEEE Transactions on Geoscience and Remote
Sensing, 25(3): 258-271, 1987.
[51] D. Marr, E. Hildreth,"Theory ol' edge detection", Proceedings of the Royal Society
of London, Series B. Biological Sciences, 207(1167), 187-217, 1980.
[52] U. Bacher, H. Mayer, "Automatic extraction of trees in urban areas from aerial
imagery", International Archives of Photogrammetry and Remote Sensing,
33(B3/I; PART 3), 5 1-57, 2000.
[53] M. Larsen, M. Rudemo, "Optimizing templates for finding trees in aerial
photographs", Pattern Recognition Letters, 19(12), 1153-1162, 1998.
[54] M. Larsen "Crown Modeling to Find Tree Ton Positions in Aerial Photographs".
Third International Airborne Remote Sensing Conference and Exhibition, 1997.
[55] L.Wang, P. Gong, GS. Biging, "Individual Tree-crown Delineation and Treetop
Detection in-Spatial Resolution Aerial Imagery", Photogrammetric Engineering and
Remote Sensing, Vol. 70, No. 3, pp. 351-357, 2004.
[56] M. Hugentobler, "Quantum GIS", Encyclopedia of GIS, pp. 935-939, Springer US,
2008.
[57] C. Poynton, "Digital Video and HD", Elsevier, 2003.
[58] A. Bleaut, L.J. Leon, "Watershed-based segmentation and region merging",
Computer Vision and Image Understanding, 77(3), 3 17-370, 2000.
[59] M. W. McDaniel, "Classification and Modeling of Forested Terrain Using LiDAR
Sensing", S.M. Thesis, MIT, 2010.
[60] M. McDaniel, T. Nishihata, C. Brooks, P. Salesses, K. lagnemma, "Terrain
Classification and Identification of Tree Stems Using Ground-Based LIDAR",
Journal of Field Robotics, Vol. 29, No. 6, pp. 891-910, 2012
[61] Y. Chen, G. Medioni, "Object Modeling by Registration of Multiple Range
Images", IEEE International Conference on Robotics and Automation, Vol.3,
pp.2724-2729, 1991.
Page 139
of 140
[62] H. Bay, T. Tuytelaars, L. Van Gool, "Surf: Speeded up robust features", Computer
Vision-ECCV, Springer Berlin Heidelberg, pp. 404-417,2006.
[63] J-F. Hamel, M-K. Langelier, M. Alger, P. lies, K. MacTavish, "Design and
Validation of an Absolute Localization System for the Lunar Analogue Rover
"ARTEMIS"". i-SAIRAS 2012, 2012.
[64] D. Chetverikov, D. Svirko, D.Stepanov, P. Krsek, "The Trimmed Iterative Closest
Point Algorithm", ICPR, 2002
[65] S. Granger, X. Pennec, "Multi-scale EM-ICP: A Fast and Robust Approach for
Surface Registration", 7 "t European Conference on Computer Vision, Part IV, pp.
418-432, 2002
[66] A.W. Fitzgibbon, "Robust Registration of 2D and 3D Point Sets", Elsevier Image
and Vision Computing 21, pp. 1145 - 1153, 2003
[67] J.M. Phillips, R. Liu, C. Tomasi, "Outlier Robust ICP For Minimizing Fractional
RMSD", 6 th International Conference on 3-D Digital Imaging and Modeling
(3DIM), 2007
[68] B. Jian, B.C. Vemuri, "Robust point set registration using gaussian mixture models"
IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(8), 16331645, 2011.
[69] P. BesI, N. McKay, "A Method for Registration of 3D Shapes", IEEE Transactions
on Pattern Analysis and Machine Intelligence, Vol. 14, No. 2, pp. 239-256, 1992.
[70] M.A. Fischler, R.C. Bolles, " Random Sample Consensus: A Paradigm for Model
Fitting with Applications to Image Analysis and Automated Cartography",
Communications of the ACM, vol.24, no.6, 1981.
Page 140 of' 140