Applications of Knowing
“Where am I”
Two Examples:
Prospect Eleven:
CoPilot|MinETA
Princeton University’s Autonomous
Vehicle Entry
2005 DARPA Grand Challenge
Real-Time
Dynamic
Minimum ETA Sat/Nav
Alain L. Kornhauser
Team Leader, Prospect Eleven
Professor, Operations Research & Financial Engineering
Princeton University
Co-founder & Board Chair, ALK Technologies, Inc.
Thursday, February 6, 2006
National Physical Laboratory
Teddignton, Middlesex, UK
Prospect Eleven:
The Making, Testing and Running of
Princeton’s Entry in the 2005 DARPA Grand Challenge
Alain L. Kornhauser
Team Leader, Prospect Eleven
Professor, Operations Research & Financial Engineering
Andrew Saxe’08
Gordon Franken’08
Scott Schiffres’06
Brendan Collins’08
Bryan Cattle’07
Rachel Blair’06
Josh Herbach’08
Anand Atreya’07
Kamil Choudhury’06
The DARPA Grand Challenge
Defense Advanced Research Projects Administration
• DARPA Grand Challenge
Created in response to a US Congressional and DoD mandate, it was a
field test intended to accelerate research and development in autonomous
ground vehicles that will help save lives on the battlefield. The Grand
Challenge brought together individuals and organizations from industry,
the R&D community, government, the armed services, academia,
students, backyard inventors, and automotive enthusiasts in the pursuit of
a technological challenge.
• The First Grand Challenge: Across the Mojave, March 2004
From Barstow, California to Primm, Nevada offered a $1 million
prize. From the qualifying round at the California Speedway, 15 finalists
emerged to attempt the Grand Challenge. However, the prize went
unclaimed as no vehicles were able to complete the difficult desert route.
• The 2005 Grand Challenge
October 8, 2005 in the desert near Primm. Prize increased to $2 million.
Jan 26, 2006
18 -month Life Cycle of Prospect Eleven
2004
2005
Jan
Jan
Automate Brakes
Feb
Feb
Install Computers
Mar
2004 Grand Challenge
Mar
Mount GPS
Apr
Announcement of 2005 GC
Apr
Work on Machine Vision
May
Decide to get involved
May
1st Site Visit
Jun
Hire 5 summer interns
Jun
Not top 40, Alternate Status
Jul
Literature survey
Jul
Aug
Literature survey
Aug 15 2nd Site Visit
Sep
Submit Application
Oct
Nov
Dec
Receive salvaged 05 Canyon
Automate Steering
Re-Do Stereo Vision
Sep 27
National Qualifying Event
Oct 8
2005 Grand Challenge
Fall
Break
Take care of unfinished business
Dec
Jan 26, 2006
Prospect Eleven & Competition
Jan 26, 2006
Objective
• Enrich the academic experience of the students
Constraints
• Very little budget
Guiding Principles
• Simplicity
Jan 26, 2006
Homemade
“Unlike the fancy “drive by wire” system employed by Stanford and VW, Princeton’s students built
a homemade set of gears to drive their pickup. I could see from the
electronics textbook they were using that they were learning as they went.”
http://www.pcmag.com/slideshow_viewer/0,1205,l=&s=1489&a=161569&po=2,00.asp
Jan 26, 2006
Teamwork
Team Leader
Alain Kornhauser*71P03
Organizational Systems
Rachel Blair’06
Real-time Decision System
Andrew Saxe’08
Computing Systems
Anand Atreya’07
Mechanical Systems
Gordon Franken’08
Electronic Systems
Bryan Cattle’07
Object Detection System
Brendan Collins’08
Control Systems
Scott Schiffres’06
Planning Systems
Josh Herbach’08
Jan 26, 2006
Mechanics
Design and Fabrication of:
• Vehicle Actuators
– Steering
– Brakes
– Transmission
• Sensor Housings
– Stereo Camera
– GPS Antennae
• System Protection
– Shock-absorbing computers
– Secure installation of components
Jan 26, 2006
Vehicle Actuators
•
Rotary Encoder
Steering
–
–
–
•
Brakes
–
–
–
–
•
Load Gears
cell
Steel
Motor from Bosch cordless drill.
Modified gears mounted directly to
steering wheel.
Rotary encoder used for position
feedback.
Bicycle brake cable used to depress
pedal.
Motor-driven linear actuator.
Load-cell used for tension feedback.
Also: pneumatic e-brake, for fail-safe
operation.
bike cable
motor
Transmission
–
–
brake pedal
Ability to shift from forward to reverse.
Not used due to several issues:
•
•
•
Slow actuation
Difficulty interfacing with vehicle
transmission
Not open up an Achilles heel.
Drill motor
Jan 26, 2006
Sensor Housings
• Stereo Camera
– Camera rated for “indoor
office use only” !!!
– Water & air tight enclosure
– Mounts for photographic
filters
• GPS Antennae
– 4 GPS receivers
– Need good location on top
of vehicle for best data.
– Wooden boom also
supported rotating beacon
Jan 26, 2006
System Protection
• Shock Absorption for
computers.
Shock-absorbing
feet
– Old way: Computers were
wrapped in foam.
– New way: Computers are
rack-mounted on shockisolating feet.
• Secure installation of
other components.
– Batteries, wires, power
supplies, air compressor…
– The “shake test”
Jan 26, 2006
Stereo Vision System
• Principle of Operation:
Difference between two
cameras gives depth
information
• Steps:
– Compute disparity image
– Find obstacles in each
column
– Approximate with
rectangles
– Filter in time domain
Jan 26, 2006
Challenges
– Quantization
– Noise
– Lighting condition
– Field of view
– Occlusion
– Range
Jan 26, 2006
The left camera sees:
Point Grey
Bumblebee
Focal Length:
4mm
Resolution:
640x480
Black & White
Jan 26, 2006
The right camera sees:
Point Grey
Bumblebee
Focal Length:
4mm
Resolution:
640x480
Black & White
Jan 26, 2006
Yielding a disparity map:
Intensity
indicates
distance: the
lighter, the
closer
White indicates
an invalidated
location.
Jan 26, 2006
Processing in each column:
Intensity
indicates
confidence that
an obstacle
exists at that
location. A
darker line
indicates a
higher
confidence.
Jan 26, 2006
Bounding with rectangles:
Darker
rectangles
indicate higher
confidence.
Jan 26, 2006
Data Representation
vs.
Reference:
http://www.darpa.mil/grandchallen
ge/TechPapers/Stanford.pdf
Jan 26, 2006
Tracking
Jan 26, 2006
Navigation
• Blind-Man’s Cane Approach:
– Find best instantaneous velocity vector (heading and
speed) within visible cone ahead.
?
Desired V
Jan 26, 2006
Step 1: Tube Projection
Jan 26, 2006
Gap Identification
Distance
Angle
Shoot the Gap
V
Jan 26, 2006
Examples of “Shoot the Gap”
Not without Problems
May Get Squeezed
Yippes!
Jan 26, 2006
‘Gotta’ Reverse
Interfacing to Existing Vehicle Data
Existing electronic information available on ’05 GMC Canyon:
• Engine status –
temp, RPM, diagnostic codes,...
• Transmission –
what gear are we in?
4WD on/off
• Car traction control –
are wheels slipping,
are we really moving?
• Wheel speed/odometry – how far have we moved?
how fast?
• Throttle is electronic
All monitored & used by Prospect Eleven
Jan 26, 2006
OBD-II Interface
10101110101...
ABS Wheel Encoders
Left
Right
Transmission position
1001 1101
1001 0101
“Data-acquisition card”
Counters
Throttle voltage
Digital inputs
DAC
Tension on the
brake pedal
101001...
ADC
101001...
100 lb
Jan 26, 2006
Other thoughts
Jan 26, 2006
Feedback Control Systems
Speed Control Module:
– Receives desired speed from Navigation
– Decides how to modulate throttle and brake
– Requirements
• Gracefully reach and maintain desired speed
• Smooth acceleration and braking
– To minimize skidding on loose sand wet grass
• Strict adherence to speed limit
– Minimize overshoot
Steering Control Module
–
–
–
–
Receives desired heading angle and speed from Navigation
Converts to desired steering wheel angle
Constrains wheel angle as function of speed to avoid roll-over.
Control Coefficients tuned to trade off response and overshoot
Jan 26, 2006
Speed Control -- Implementation
Percent Throttle
Desired Speed
Speed Control
Percent Brake Tension
Current Speed
Latest Speeds
• Proportional Integral Control
– Output = KPev + ∫KIevdt + KD (dev∕dt)
– Where evelocity = Vdesired – Vcurrent
• KP term deals with the bulk of error
• KI integrates up error to eliminate steady state error
• KD reduces overshoot and ringing by slowing response
Jan 26, 2006
Why P11 stopped after 9.4 miles
• We store a list all obstacles we are currently concerned about
– Each time a new obstacle is received, we add it to this list
– Each obstacle needs to have its relative position updated each time
we get new vehicle position information
• It gets these updates by subscribing to the RelativeFrameUpdated event
• When we have passed an obstacle by more than twenty feet,
we remove it from the list
– However, we were not unsubscribing the obstacle from the
RelativeFrameUpdated event
Thus, nine miles down the road, our computer was still processing a
bush it saw near the beginning of the course!
 C# is a powerful language, but it has to be used carefully
Jan 26, 2006
Timeline of Accomplishments of Princeton’s
“Prospect Eleven” DARPA Grand Challenge Team
May 2004 - Nov 2005
Segment
“Going Back” Oct 30Nov 2, ‘05
Description
After completing 10 miles in
GCE, Prospect Eleven
returns to the desert to
“complete” the 2005 and the
2004 Grand Challenge
courses
GPS tracks for 3 days
Images
Videos
GPS Tracks 0.3mb
Run Images 3mb
Cruisin’04 2.5mb
Nov 2, ‘05
Return to Beer Bottle Pass,
GPS Tracks 0.3mb
2004 DGC
Nov 1, ‘05
Oct 31, ‘05
GPS Tracks 0.1mb
Run Images 3mb
Cruisin’05 16mb
Gate’05 3mb,
Gate2 ’05 4mb
Changing “one line”, Oct 30, ‘05
Images 0.2mb
Fixing
Calibrating
DARPA Grand Challenge Event, 132 mile course in desert around Primm, NV; 23
entries; Prospect Eleven is #10 seed
Run Summary
Start_11mb
PassBy_29mb
Summary 17mb
National Qualifying Event held at California Speedway. 43 “bots” competing for
23 positions on 2.2 mile course
NQE Course
Sep 27-Oct 5, 05
CrashOutside_7mb
CrashInside_4mb
PrefectRun5_9mb
Run-up to NQE
Modifications and testing after invitation as one of three “Alternates” to the NQE
2005 GCE
Oct. 8, ‘05
NQE
2005 DGC,
Aug 17-Sep 26
2nd Site Visit
Aug 16, ‘05
1st Site Visit
May 3, ‘05
Automation of P11
Nov ’04-May ‘05
Application & Prep
May ‘04-Nov ’05
Pre-2nd_SiteVisit_6mb
2nd chance at demonstrating capabilities of P11 to DARPA officials at West
Windsor athletic fields after earning “Alternate status
Demonstration of Prospect Eleven’s capabilities and potential to DARPA officials,
mixed performance
Conversion of a GMC 05 Canyon to Prospect Eleven: Automation of brake,
throttle, steering; GPS; vision
Putting the team together, planning, organization, & literature search.
P11 Tech Paper
The Day Before
2005 Grand Challenge Event
DAPRPA Grand Challenge Event, October 07, 2005
The DARPA Grand Challenge Event Team 2005
Grand Challenge Event
L2R: Rachel Blair’06, Dan Chiou’05, Prof. Alain Kornhauser*71 P03, Ben Essenburg’05, Bryan Cattle’07, Josh Herbach’08. Jeff Jones*05,
Kamil Chihoudy’06, Scott Schiffres’06, Anand Atreya’07; Launch Team: Andrew Saxe’08, Brendan Collins’08, Gordon Franken’08
DAPRPA Grand Challenge Event, October 08, 2005
The Launch of Prospect Eleven
2005 Grand Challenge Event
DAPRPA Grand Challenge Event, October 08, 2005
Return of Prospect Eleven @ 8 Mile Mark
2005 Grand Challenge Event
DAPRPA Grand Challenge Event, October 08, 2005
Approaching Beer Bottle Pass 19:02
2005 Grand Challenge Course
DAPRPA Grand Challenge, Unfinished Business, October 31, 2005
Beer Bottle Pass 19:10 PST
2005 Grand Challenge Course
DAPRPA Grand Challenge, Unfinished Business, October 31, 2005
The Assent: 17:00
Rerun to Beer Bottle Pass
DAPRPA Grand Challenge, Unfinished Business, November 1, 2005
The Assent: 17:01
Rerun to Beer Bottle Pass
DAPRPA Grand Challenge, Unfinished Business, November 1, 2005
12:25
2004 Grand Challenge Course
DAPRPA Grand Challenge, Unfinished Business, November 2, 2005
14:26
2004 Grand Challenge Course
DAPRPA Grand Challenge, Unfinished Business, November 2, 2005
The Finish 18:44
2004 Grand Challenge Course
L2R: Prof. Alain Kornhauser*71 P03, Andrew Saxe’08, Bryan Cattle’07, Scott Schiffres’06
DAPRPA Grand Challenge, Unfinished Business, November 2, 2005
Prospect Eleven’s
Unfinished Business
Finish, ’05 Course
October 31-November 2, 2005
DARPA Grand Challenge ’04 & ’05 Courses
GPS Tracks and Timing Maps
DAPRPA Grand Challenge, Unfinished Business, October 31, 2005
14:16:03
2:23 remove rubble blocking I-15 underpass
GPS Tracks & Timing of Prospect Eleven’s
Autonomous run of the 05 GC Course 10/31/05
12:42:35
4:40 Pause
12:33:51
1:19 Pause
13:16:05
1:32 Rejoin course
13:18:53
1:10 Pause
15:13:31
1:30 Wait for traffic to clear on NV 161
11:29:52
42:00 Pull chase car out of mud
11:15:15
5:14 Pause at ranch, go thru gate
10:48:04
3:45 Open & pass thru gate
16:03:07
1:53 Pause
18:02:50
2:08 Pause
10:53:33
2:15 Pause
16:07:48
4:32 to take pictures
16:12:50
11:44 negotiate bulldozed area
10:37:37
5:54 Open & pass thru gate
18:20:53
38:50 prepare inside video
19:00:21 – 0:56 Adjust Camera
19:02:53 – 1:52 Wait for chase car
16:31:17
2:12 Survey another bulldozed area
8:20:55
5:20 Pictures at Primm return
Start 7:53:30 PST
Finish 19:35:49
Elapsed Time 11:43:49
Pause Time ~2:55:49
Autonomous Time 8:48:59
9:52:53
8:14 Open gate & pass through
8:44:48
1:10 Pause
8:31:42
Jan 26, 2006
8:53 Take down fence & pass through
Blue: 8:00 Outbound 11/2
Lake dried sufficiently,
no deviation required.
Red: 8:00 Outbound 10/31
Deviation to avoid “lake”
Yellow: 16:30 Return 10/31
Deviation to avoid “lake”
GPS Tracks & Timing of Prospect Eleven
Diverting Around Not-so-dry Lake
05 GC Course 10/31/05
Jan 26, 2006
Return to Beer Bottle Pass
GPS Tracks of return to
Beer Bottle Pass 11/01/05
DAPRPA Grand Challenge, Unfinished Business, November 1, 2005
Return to Beer Bottle Pass
2005 Grand Challenge Course, 1.9 miles
GPS Tracks & Timing of Prospect Eleven’s
Autonomous run up then back down
Beer Bottle Pass 11/02/05
Start Down: 17:10:17
End Up: 17:04:48
Duration: 0:10:21
End Down: 17:18:04
Duration: 0:7:47
Start Up: 16:54:27
DAPRPA Grand Challenge, Unfinished Business, November 1, 2005
GPS Tracks & Timing of Prospect Eleven’s
Autonomous run of the 04 GC Course 11/02/05
7:28:33
4:35 negotiate bulldozed area
8:39:13
11:50 take pictures
9:42:26
1:50 take pictures
8:34:01
4:30 take pictures
10:04:59
2:07 take pictures
7:08:36
Start ’04 GC Course
10:28:11
3:20 take pictures
7:55:26
33:08 Divert to get gas for P11
10:13:43
2:10 cross road
Jan 26, 2006
DAPRPA Grand Challenge, Unfinished
Business, November 2, 2005
GPS Tracks & Timing of Prospect Eleven’s
Autonomous run of the 04 GC Course 11/02/05
12:05:25
0:30 Xing CA 127
12:08:00
2:00 take pictures
10:57:13
6:30 take pictures
15:10:10
2:04 Inspect burial monument
12:38:03
2:17 take pictures
14:44:13
2:47 take pictures
12:19:37
1:50 take pictures
12:53:36
0:54 Pause
13:08:54
1:05:56 Fix steering encoder
14:33:15
2:45 take pictures
14:50:xx
4:00 Avoid Washout in 4 spots
Jan 26, 2006
DAPRPA Grand Challenge, Unfinished
Business, November 2, 2005
Finish, ’04 Course
15:19:00
4:32 take pictures
GPS Tracks & Timing of Prospect Eleven’s
Autonomous run of the 04 GC Course 11/02/05
15:38:01
1:37 take pictures
15:46:03
2:07 take pictures
16:12:07
0:50 Xing road
File photo
File photo
17:32:27
4:00 Inspecting Silt Filled underpasses < 4 ft. clearance
Start 7:08:36
Finish 18:44:19 – Arrival at gate of Slash X Cafe
Elapsed Time 11:37:43
Pause Time ~03:28:00
Autonomous Time 08:09:00
16:35:14
1:00 Xing Yermo Road
File photo
16:33:15
2:45 Letting traffic pass on Randall Drive
17:16:33
15:28 Replace Front Left Blowout
Daggett Pass
18:18 Top
18:35 Bottom
18:41:26
1:10 Pause let cars pass on Stoddar
Valley Rd
Jan 26, 2006
Jan 26, 2006
Accomplishments:
•
•
•
•
Invited to National Qualifying Event
Seeded 10th for Grand Challenge
Accomplished 10 miles of Autonomous Driving in GC
“Completed” the 2005 & 2004 Courses during Fall Break
Overview:
http://orfe.princeton.edu/~alaink/Prospect11/Movies/P11SummaryMovie.wmv
Wasn’t so easy, Outside:
http://orfe.princeton.edu/~alaink/Prospect11/Movies/NQE2CrashOutsideView.mov
Wasn’t so easy, Inside:
http://orfe.princeton.edu/~alaink/Prospect11/Movies/NQE2CrashInside.mov
Some Videos:
Essentially Perfect:
http://orfe.princeton.edu/~alaink/Prospect11/Movies/NQE5PerfectRun.mov
Launch at Grand Challenge:
http://orfe.princeton.edu/~alaink/Prospect11/Movies/GCE-Start.AVI
Doing Better than 2004’s Best:
http://orfe.princeton.edu/~alaink/Prospect11/Movies/GCE_PassBy8Miles.avi
Return to the Desert; 2005 Course:
http://orfe.princeton.edu/~alaink/Prospect11/Movies/PGC05_10h29mFastDryLake.AVI
http://orfe.princeton.edu/~alaink/Prospect11/Movies/PGC05_10h34mFastDryLake.AVI
Return to the Desert; Return to Beer
Bottle Pass:
http://orfe.princeton.edu/~alaink/Prospect11/Movies/PGCBB_Small.mov
Return to the Desert;
Jan 26, 20062004 Course:
Lessons Learned
• It is non-trivial to “Just Do It”
– You must respect Uncertainty (and plan for it)
– Harmonize Accuracy
– Time is your Friend
• (only know what you need to know when you need to know it)
– More is not necessarily Better
– Always assume your code has bugs
• Stereo Vision Does Work!
• Three (3) Regimes of Autonomous Control
– Under “7” mph
– Between 7- 25 mph
– Above 25 mph
Jan 26, 2006
CoPilot|MinETA:
Real-Time Dynamic Minimum ETA Sat/Nav
Operational Processes used by
Capital District Advanced Traveler Information System
(CD-ATIS)
by
Alain L. Kornhauser
Co-founder & Board Chair, ALK technologies, Inc.
Jan 26, 2006
Things Change!!
PROBLEM: How to get from A to B
•Many Paths
•Each with a Different Value to the Decision Maker
•Each Segment Changing with Uncertainty over Time
1
2
6
1
8
3
2
4
9
2
5
3
3
6
5
8
6
3
1
6
1
Jan 26, 2006
Link Travel Times
Historic,
& Forecast
OneActual
week-day
on oneDuring
link Day
Things change!
Jan 26, 2006
Real-Time Dynamic Minimum ETA Sat/Nav
“Advance” project
The late 90s
&
Illinois Universities
Transportation Research Consortium
Conducted its version of the abandoned “ADVANCE”
(Advanced Driver and Vehicle Advisory Navigation ConcEpt )project
•250 Volunteers using CoPilot|Live commuting to/from RPI
• CoPilot continuously shares real-time probe-based traffic
data
• CoPilot continuously seeks a minimum ETA route
Link
Jan 26, 2006
Project Objectives
• Create: real-time data collection from
vehicles and dissemination to vehicles of
congestion avoidance information which is used
to automatically reroute drivers onto the
fastest paths to their destinations
• Target locations: small to medium-sized urban
areas
• Aspects: operations, observability,
controllability, users, information transfer to
travelers
Jan 26, 2006
Experiment Details
3-month field test
Capital District (Albany), NY, USA
Journey-to-work
200 participants
80 Tech Park employees
120 HVCC staff & students
“Techy” travelers
Network:
Freeways & signalized arterials
Congested links
Path choices exist
Jan 26, 2006
Basic Operational Architecture
Two-way cellular data communications between
Customized
Live|Server
at ALK
Customized
CoPilot|Live
In vehicles
6
1
2, 4
3
7
5
Jan 26, 2006
8
Destinati
on
The In-Vehicle “Device”
CoPilot GPS unit:
Determines location
Sprint PCS Vision card and
battery pack: Communication
with the server
Pocket PC with 256 MB SD card:
Software platform and audio
communication with the driver
Jan 26, 2006
Other
Sat/Nav Platforms
Every Second
CoPilot|Live Determines “Where am I”,
Then…
If Momument, mj , is passed
Send mi , mj , ttk(mi, mj )= t(mi) - t(mi)
(52 bytes)
CoPilot|Live
Set i=j
“Where Am I”,
Then…
ALK Server
Updates:
TT(mi, mj )
Jan 26, 2006
Every “n” Minutes
CoPilot|Live …
Send… Current Location & Destination,
Last update time
(42 bytes)
ALK Server …
Determines Uk : set of TT(mi, mj ) within “bounding polygon”
of (Location;Destination)k that have changed more than
“y%” since last update.
ALK Server …
Send… New TT(mi, mj ) for every (i,j) in Uk
CoPilot|Live
Sends: “Where am I”,
Dest., Last update
Receives/Posts: updates
Computes: MinETA
Updates route, if better
(280 bytes/100arcs)
CoPilot|Live …
Updates TT(mi, mj ) in Uk , ETA on current route,
Finds new MinETA route, if MinETA
“substantially” better then… Adopt new route
Jan 26, 2006
ALK Server
Builds: set Uk
Sends: TT(mi, mj )
for every (i,j) in Uk
When Available
ALK Server …
Receives: Other congestion information from various
source, blends them in TT(mi, mj )
ALK Server
Updates:
TT(mi, mj )
Jan 26, 2006
What We Heard
I'm very impressed with the CoPilot
program thus far. The directions are
accurate and it adapts quickly to
route changes.
I find it interesting how willing I
am to listen to a machine tell me
which route to take
I like using it for when I have no
idea on how to get somewhere,
and it is good for my normal route
because it keeps me out of traffic
on route 4.
This thing is awesome. I was a little
skeptical at first but once i got the hang of
it I don’t know how I went along without
it. I think any student commuting to school
will benefit from this.
It is great, it took a while to
trust it telling me where to
go, but i like it because i cant
get lost! Thanks.
Jan 26, 2006
also Can
Watch Vehicles
1
2
Jan 26, 2006
3
North American Monument Network
• ~125,000 North American “Monuments”
• ~106 (mi, mj)
• 1st Commercial application in PC*Miler v19
(contains Median Travel Time by Time-of-Day for all NA)
(mi, mj) near Troy
(mi, mj) larger area
Jan 26, 2006
Thank you
Alain L. Kornhauser
alaink@alk.com
www.alk.eu.com
www.princeton.edu/~alaink/
Jan 26, 2006