Learning to Navigate Through
Crowded Environments
Peter Henry1, Christian Vollmer2, Brian Ferris1, Dieter Fox1
Tuesday, May 4, 2010
1University
of Washington, Seattle, USA
2Ilmenau University of Technology, Germany
The Goal
Enable robot navigation within crowded environments
Motivation
 Robots should move naturally and predictably within
crowded environments
 Move amongst people in a socially transparent way
 More efficient and safer motion
 Humans trade off various factors
 To move with the flow
 To avoid high density areas
 To walk on the left/right side
 To reach the goal
Challenge
 Humans naturally balance between various factors
 Relatively easy to list factors
 But they can’t specify how they are making the tradeoff
 Previous work typically uses heuristics and parameters are
hand-tuned
 Shortest path with collision avoidance [Burgard, et al., AI
1999]
 Track and follow a single person [Kirby, et al., HRI 2007]
 Follow people moving in same direction [Mueller, et al.,
CogSys 2008]
Contribution
 Learn how humans trade off various factors
 A framework for learning to navigate as humans do within
crowded environments
 Extension of Maximum Entropy Inverse Reinforcement
Learning [Ziebart, et al., AAAI 2008] to incorporate:
 Limited locally observable area
 Dynamic crowd flow features
Markov Decision Processes
 States
S0
S1
 Actions
 Rewards / Costs
 (Transition Probabilities)
S2
 (Discount Factor)
S3
Goal
Navigating in a Crowd as an MDP
 States si
 In crowd scenario: Grid cell + orientation
 Actions ai,j from si to sj
 In crowd scenario: Move to adjacent cell
 Cost = An unknown linear combination of action features
 Cost weights to be learned: θ
 Path: τ
 Features: fτ
cost( |  )   ·f 
  ·f
ai , j 
ai , j
Inverse Reinforcement Learning
 Inverse Reinforcement Learning (IRL):
 Given: The MDP structure and a set of example paths
 Find: The reward function resulting in the same behavior
 (Also called “Inverse Optimal Control”)
 Has been previously applied with success
 Lane changing [Abbeel ICML 2004]
 Parking lot navigation [Abbeel IROS 2008]
 Driving route choice and prediction [Ziebart AAAI 2008]
 Pedestrian route prediction [Ziebart IROS 2009]
Maximum Entropy IRL

 ·f
e
Exponential distribution over paths: P( |  ) 
 ·f 
  e
 Learning:
 *  arg max  log P( |  )

 T
 Gradient: Match observed and expected feature counts
F  f   Dai , j fai , j
ai , j
Locally Observable Features
 It is unrealistic to assume the agent has global knowledge
of the crowd
 Contrast: Continuum Crowd Simulator explicitly finds a
global solution for the entire crowd
 We do assume knowledge of the map itself
 Training: Only provide flow features for small radius
around current position
 Assumes that these are the features available to the “expert”
 A single demonstration path becomes many small
demonstrations of locally motivated paths
Locally Observable Dynamic
Features
 Crowd flow changes as the agent moves
 Locally observable dynamic feature training:
1.
2.
3.
4.
Update flow features within local horizon
Compute feature gradient within grid
Perform stochastic update of weights
Take the next step of the observed path
Locally Observable Dynamic IRL
 The path probability decomposes into many short paths
over the current features in the locally observable horizon
1 t  0hH  ·fat t h
P( |  ) 
e
Z( )
Decompose
over
timesteps
Local Horizon
Features for
actions
within
horizon at
time t
Locally Observable Dynamic
Gradient
 Uses current estimate of features at time t
 Computes gradient only within local horizon H
F  f 
t
t

ai , j H
Observed
features within H
t
ai , j
D f
t
ai , j
Expected
features for
actions within H
Map and Features
 Each grid cell encompasses 8 oriented states
 Allows for flow features relative to orientation
 Features
 Distance
 Crowd flow speed and direction
 Crowd density
 (many others possible…)
 Chosen as being reasonable to obtain from current sensors
Crowd Simulator
[Continuum Crowds, Treuille et al., SIGGRAPH 2006]
Simulator Environment
Experimental Setup
We used ROS [Willow Garage] to integrate the crowd simulator and IRL
learning and planner
1.
Extract individual crowd traces and observable features
2.
Learn feature weights with our IRL algorithm
3.
Use weights for a simulated robot in test scenarios

Planning is A* search
Re-planning occurs every grid cell with updated features
The robot is represented to the crowd simulator as just another
person for realistic reactions from the crowd


Quantitative Results
 Measure similarity to “human” path
 Shortest Path (baseline): Ignores crowd
 Learned Path: The path from our learned planner
 Mean / Maximum Difference: Over all path cells, difference
to closest “human” path cell
Shortest
Path
Learned
Path
Improvement
1.4
0.9
35%
Maximum Difference 3.3
2.3
30%
Mean Difference
(Difference is significant at p=0.05 level)
Mall Scenario (Video)
Lane Formation (Video)
Future Work
 Train on real crowd data
 Overhead video + tracking?
 Wearable sensors to mimic robot sensor input?
 Implement on actual robot
 Is the method effective for raw sensor data?
 Which are the most useful features?
 Pedestrian prediction
 Compare / incorporate other recent work [Ziebart IROS 2009]
Conclusion
 We have presented a framework for learning to imitate
human behavior from example traces
 We learn weights that produce paths matching observed
behavior from whatever features are made available
 Our inverse reinforcement learning algorithm handles
locally observable dynamic features
 Resulting paths are more similar to observed human paths