3D Environmental Mapping and Imaging
for AUVSI RoboBoat
David Bumpus, Dan Kubik, & Juan Vazquez
Advisor: Dr. José Sánchez
Customer: Mr. Nick Schmidt
Department of Electrical
& Computer Engineering
November 19, 2015
Problem Background: Significance
• AUVSI – Association for Unmanned Vehicle System Int’l
 International RoboBoat Competition
 Bradley has attended since 2012
Fig. 1. Bradley 2013 RoboBoat [1]
2
Problem Background: Significance
• Boat undergoes challenges
Fig. 2. RoboBoat navigating through course [1]
Fig. 3. 2014 RoboBoat course [2]
3
Problem Background: Motivation
Fig. 4. Boat navigation [3]
Fig. 5. Boat struggling [3]
4
Problem Background: Objective
• Return:
 Distance measurements
 Camera image
 Image with distance information
 Location of nearest object
5
David Bumpus: Work Accomplished
• Research and develop registration method
• Simulate registration method in MATLAB
6
David Bumpus: Work Accomplished
7
Work Accomplished: Research Registration [4]
• Read papers on Lidar and image registration
• Feature Detection vs. Mutual Information
• Mutual information is computationally expensive
• Feature Detection is dependent on presence of sufficient
number and distribution of features
Lidar Point
Cloud
2D Image
Lidar
Intensity
Fig. 6. Mutual
Information Principle [4]
8
Work Accomplished: Research Registration [5]
• Scale Invariant Feature Transform (SIFT)
• Speeded-Up Robust Features (SURF)
• Principal Components Analysis SIFT (PCA-SIFT)
Table I- Performance Comparison of 3 Algorithms [5]
Algorithm Time
Scale
Rotation
Blur
Illumination Affine
SIFT
Common
Best
Best
Best
Common
Good
PCA-SIFT
Good
Common
Good
Common
Good
Good
SURF
Best
Good
Common
Good
Best
Good
9
David Bumpus: Work Accomplished
10
Work Accomplished: MATLAB Simulation
Fig. 7. MATLAB SIFT Algorithm Implementation [6]
11
Work Accomplished: MATLAB Simulation
• Vectors represent keypoint descriptors
• Each descriptor describes:
 Location of keypoint
 Magnitude of gradient
 Direction of gradient
Fig. 7. MATLAB SIFT Algorithm Implementation [6]
12
Work Accomplished: MATLAB Simulation
Fig. 8. Matched features for two very different images [6]
13
Work Accomplished: MATLAB Simulation
• Able to match features from very different environments
• Some error present, but attribute to relative object shifts
• Ball has moved in relation to chalkboard and camera
angle has changed
Fig. 8. Matched features for two very different images [6]
14
Work Accomplished: MATLAB Simulation
Fig. 9 Matched features for two similar images [6]
15
David Bumpus: Work Accomplished
16
David Bumpus: Future Work
• Implement registration on embedded device for single frame
 Delaunay Triangulation
 SIFT Feature Matching
 RANSAC Application
 Produce depth overlay
• Implement registration for live video
17
Juan Vazquez: Work Accomplished
Oct-15
Activity
Operating System Installation
Logitech C500 Interface Implementation
VLP-16 Interface Implementation
Nov-15
1 6 8 15 20 22 27 29 3 5 10 12 17 19
70%
• OS Selection & Installation
• Logitech C500 & Odroid-XU4 Interface
• VLP-16 & Odroid-XU4 Interface
18
Juan Vazquez: Work Accomplished
Oct-15
Activity
Operating System Installation
Logitech C500 Interface Implementation
VLP-16 Interface Implementation
Nov-15
1 6 8 15 20 22 27 29 3 5 10 12 17 19
70%
19
Work Accomplished: OS Installation
• Ubuntu Server
 Memory usage
 Software availability
 Configurability
Fig. 10. Ubuntu Server [7]
20
Work Accomplished: OS Installation
• Operating system booting
 MicroSD
 EMMC (Embedded MultiMediaCard)
• Establishing network connection
 Ethernet connection bridge
Fig. 11. MicroSD [8]
Fig. 12. EMMC [9]
21
Juan Vazquez: Work Accomplished
Oct-15
Activity
Operating System Installation
Logitech C500 Interface Implementation
VLP-16 Interface Implementation
Nov-15
1 6 8 15 20 22 27 29 3 5 10 12 17 19
70%
22
Work Accomplished: Camera Interface
• Logitech C500 Linux driver
• OpenCV
 Functions library
 Video capture
 Image registration
Fig. 13. OpenCV [10]
23
Work Accomplished: Camera Interface
• Image capture program
 Live video read
 Continuously capturing/overwriting
 Stores in PPM (Portable Pixmap Format)
Fig. 14. Logitech C500 [11]
24
Juan Vazquez: Work Accomplished
Oct-15
Activity
Operating System Installation
Logitech C500 Interface Implementation
VLP-16 Interface Implementation
Nov-15
1 6 8 15 20 22 27 29 3 5 10 12 17 19
70%
25
Work Accomplished: VLP-16 Interface
• Libpcap
 Library for packet capture
• Packet Detection
 UDP (User Datagram Protocol)
• Packet Storage
 Text file
Fig. 15. VLP-16 [12]
26
Juan Vazquez: Future Work
Activity
VLP-16 Interface Implementation
Test Timing And Transmission of Data
Odroid-XU4 Multicore Programming
N Dec-15 Jan-16
Feb-16
24 1 3 8 21 26 28 2 4 9 11 16 18 23 25
70%
• VLP-16 & Odroid-XU4 Interface
 Packet Formatting
 Accuracy Testing
27
Dan Kubik: Work Accomplished
80%
• Simulation of lidar (MATLAB)
• Implementation of lidar reading functionality (C++)
28
Dan Kubik: Work Accomplished
80%
29
Work Accomplished: Simulation of lidar
• VeloView Software
Fig. 17. VeloView view of box
Fig. 16. VeloView full view of room with box
30
Work Accomplished: Simulation of lidar
• Keys to successful simulation:
 Visualizing how data will be received
• Stream of bytes vs. full packet
 Understanding data structure
• Packets, blocks, data points
 Conversion from spherical to rectangular coordinates
31
Work Accomplished: Simulation of lidar
• MATLAB
Fig. 19. MATLAB
view of box
Fig. 18. MATLAB full view of room with box
32
Dan Kubik: Work Accomplished
80%
33
Work Accomplished: Lidar Implementation
• Goal: Implement for Odroid XU4
• C++ and Object Oriented Programming
 Creation of classes:
• Storage classes (DataPacket, DataBlock,
HalfDataBlock, sphDataPoint)
• Interface classes (MasterBlock, recDataPoint)
34
Dan Kubik: Future Work
80%
• Implementation of lidar reading functionality
 Installing code to Odroid XU4
 Testing
35
Conclusion
• David
 Registration algorithm development
• Juan
 OS installation & interfacing
• Dan
 Simulation & implementation of lidar
36
Conclusion
• Progress and Future work:
 On schedule
 Potential problems with registration
• Difficulty obtaining key points from lidar data
37
3D Environmental Mapping and Imaging
for AUVSI RoboBoat
David Bumpus, Dan Kubik, & Juan Vazquez
Advisor: Dr. José Sánchez
Customer: Mr. Nick Schmidt
Department of Electrical
& Computer Engineering
November 19, 2015
Additional Slides
39
Testing of VLP-16 Puck™
• Install Veloview Software
• Test acquisition of data to Veloview and verify distances
• Familiarize self with Veloview data manipulation
Fig. 20. Data acquisition to Veloview using VLP-16 Puck
40
Testing of Logitech C500 Webcam
• Install Logitech software
• Capture images and Video
Fig. 21. Color
image from
Logitech C500
41
Registration Flowchart
• Filter point cloud
• Interpolate point cloud
• Create depth maps
• Feature detection
• Outlier elimination
• Feature matching
Fig. 22.
Registration
method
flowchart [13]
42
EMMC vs. microSD Card
Fig. 23. Write/Read Speed Comparison [14]
43
UDP Detection
Fig. 23. VLP-16 UDP Detection [15]
• Current Packet Sniffer Detection
 TCP (Transmission Control Protocol)
 UDP (User Datagram Protocol)
 ICMP (Internet Control Message Protocol)
 IGMP (Internet Group Management Protocol)
 Others (Detects all Ethernet Frames)
44
Ubuntu Desktop vs. Server
Fig. 24. Ubuntu Desktop Requirements [16]
Fig. 25. Ubuntu Server Requirements [16]
45
Creating a Bootable MicroSD Card
• MiniTool Partition Wizard
 Partition creation and wiping
• 7zip
 Extraction of IMG.XZ file
• Win32 Disk Imager
 Writing the unpacked IMG file to Micro SD card
• Process can also used for flashing eMMC
46
OpenCV System Requirements [17]
• Operating System: Windows, Linux or Mac
• Memory (RAM): 256 MB
• Hard Disk Space: 50 MB
• Processor: 900 MHz
47
JPG, PNG, & PPM Comparisons [18] [19]
• JPG (Joint Photographic Experts Group)
 Supports 24-bit color RGB
 Compress image data by reducing sections of images to
blocks or pixels
 Compression is permanent
• PNG (Portable Network Graphics)
 Supports transparency, 8-bit color, and 24-bit color RGB
 Largest of three compared formats
• PPM (Portable Pixmap Utilities)
 Lowest common denominator color image file format
 Includes basic color
 Easy to format to process
48
Data Diagram: Velodyne User’s Manual
49
Storage Classes
• DataPacket: Contains 12 DataBlocks and a time stamp
• DataBlock: contains 2 HalfDataBlocks
• HalfDataBlock: contains 16 DataPoints and an azimuth
• DataPoint: contains one distance, reflectivity, and alt. ang.
50
Interface Class
• Contains many dataPoints (ability to change size)
• Ensures full 180° view and selected range are received
51
Link to Proposal
• http://ee.bradley.edu/projects/proj2016/lidar/Attachmen
ts/Project_Proposal.pdf
52
Resources
[1] http://www.bradley.edu/inthespotlight/story/?id=b46cf284-2bd9-4efb-917e-ba6ca565cf84
[2] http://learnlab.uta.edu/roboboat/the-auvsi-roboboat-competition/
[3] Bradley University RoboBoat 2013 Qualifying Run https://www.youtube.com/watch?v=V4pz1FiMXfA
[4] Parmehr, E. G., C. S. Fraser, C. Zhang, and J. Leach. "Automatic Registration of Optical Imagery with 3d Lidar
Data Using Local Combined Mutual Information." ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci. ISPRS
Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences II-5/W2 (2013): 229-34. Web. 17
Nov. 2015.
[5] Young, Jeffery, and Milan Simic. "LIDAR and Monocular Based Overhanging Obstacle Detection." Procedia
Computer Science 60 (2015): 1423-432. Web. 17 Nov. 2015.
[6] http://www.cs.ubc.ca/~lowe/keypoints/
[7] http://www.laybourn.org/index.php/sample-page/
[8] http://www.techwaredist.com/wordpress/index.php/740/
[9] http://www.hardkernel.com/main/products/prdt_info.php
[10] http://www.exuberantsolutions.com/opencv-training.htm
53
Resources
[11] http://www.engadget.com/products/logitech/webcam/c500/
[12] http://velodynelidar.com/vlp-16.html
[13] Lv, Fang, and Kan Ren. "Automatic Registration of Airborne LiDAR Point Cloud Data and Optical
Imagery Depth Map Based on Line and Points Features." Infrared Physics & Technology 71 (2015): 45763. Web.
[14] http://www.hardkernel.com/main/products/prdt_info.php
[15] http://velodynelidar.com/docs/manuals/VLP16%20User%20Manual%20and%20Programming%20Guide%2063-9243%20Rev%20A.pdf
[16] https://help.ubuntu.com/community/Installation/SystemRequirements
[17] http://getintopc.com/softwares/development/opencv-free-download/
[18] http://netpbm.sourceforge.net/doc/ppm.html
[19] http://www.howtogeek.com/howto/30941/whats-the-difference-between-jpg-png-and-gif/
54