Robot-Assisted Urban Search and Rescue
at the WTC Disaster
Prof. Robin Murphy
Director, Research CRASAR
also Associate Prof., Computer Science & Eng.
University of South Florida
murphy@csee.usf.edu
John Blitch, Jenn Casper, Mark Micire, Brian Minten
www.crasar.org
NBC Today Show Sept 20, 2001
Feb 19, 2002
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
2
Outline
•
•
•
•
Why Robots?
What should robots do?
What did they do?
What is needed?
Feb 19, 2002
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
3
The First Known Robot-Assisted USAR
• CRASAR stood up by John Blitch Sept. 2 to transfer military
robots for USAR: deploy, train, evaluate, & research robots
for SAR
• At WTC, CRASAR robots and personnel worked with FDNY,
NYC, and FEMA teams such as INTF-1, PATF-1, VATF-1,
OHTF-1
Duration
Tuesday, Sept 11 – Oct. 2
Victims found
10+
CRASAR
Teams/people
Defense Advanced Projects Research Agency (1), US
Dept. of Defense (2), Foster-Miller (5), iRobot (8),
US Navy SPAWAR (3), University of South Florida (4)
Robots on site
17, 7 used on rubble pile
Field excursions
through 9/22…
11 (8 on rubble pile with depths of 20-45ft)
1 min set up, avg. 7 minutes per run
Feb 19, 2002
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
USF was only
USAR certified team
USF brought 6,
2 from NSF
USF team logged
most hours on
The Pit in 1st 2 weeks
(except Blitch)
4
photos courtesy of Justin Reuter
Feb 19, 2002
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
5
Why Use Robots?
• Things that humans can’t
do or can’t do safely
– “the human use of humans”
Norbert Wiener
– and that applies to dogs
• 135 rescuers died Mexico
City, 65 in confined spaces
• Not enough trained people
– 1 survivor, entombed: 10
rescuers, 4 hours
– 1 survivor, trapped/crushed:
10 rescuers, 10 hours
Feb 19, 2002
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
photos courtesy of Justin Reuter
6
ESF9 Tasks: Search, Assessment, Medical
Local Jurisdiction
FEMA Task Force
Organizational Chart
FEMA
DoD Liaison
Task Force Leader
Plan/Ops
Safety
Search Team
Rescue Team
Medical Team
Logistics Team
2 Canine
Search Spec.
2 Rescue Squads
12 Rescue Spec.
2 Medical
Specialists
Structure
Specialist
Tech Search
Rigging
Haz Mat
PIO
Comm. Spec.
Feb 19, 2002
photoRescue
courtesy of
Robot-Assisted Search and
atJustin
WTCReuter
Disaster (NSF)
Logs Spec.
7
Task Highlights
Search Team
Rescue Team
Medical Team
2 Canine
Search Spec.
Logistics Team
Structure
Specialist
needed
not used
Tech Search
Haz Mat
Rescue Phase
9-11 through 9-21
worked with FDNY, FEMA
found 6+ victims
(equivalent to FLTF-2)
Feb 19, 2002
Recovery Phase
9-23 through 10-02
NYC DDC engineers
found 3-5 victims
added HazMat sensors
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
8
Search Task Priorities
• Rescue first, recover later
– first responders (always take care of
your own)
– civilians in rubble pile
– civilians elsewhere
• Information for triage
– where are the survivors?
– where are the people likely to be?
– where are the survivors likely to
have survived?
– which pile of rubble do I remove
first?
• Typical resources
– dogs, search cams, acoustics,
sledgehammers
Feb 19, 2002
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
dogs:
injured by sharp metal,
smell only 0.3m due to
rain on 2nd day, lack
of circulating air
SearchCam:
camera on a pole
9
Robots Compared with SearchCams
• SearchCams ~$10K, Robots
~$12K
• SearchCams reach up to 5 meters
• Robots reach up to 30 meters,
averaged between 6 and 13 meters
• Robots can put light on the object,
prod it, look at it from different
angles
• Robots can go through more
twisting tunnels
• SearchCams and small robots take
<1.5 minutes to set up and insert
camera with zoom, lights couldn’t do this
helped to recognize remains of a body
Feb 19, 2002
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
10
History of Robot-Assisted USAR in USA
Feb 19, 2002
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
11
Timeline of Response
(Rescue and Recovery, Inspection not shown)
T
11
W
12
Th
13
F S Su M
14 15 16 17
1 S 1
T
18
W
19
4
U
Th F S
20 21 22
Inuktuns
DARPA
USF
Foster-Miller
Solem
rubble pile
(The Pit)
Foster-Miller
Talon
Packbot iRobot
UrBot
Murphy
Casper
Micire
Minten
Blitch
Mangolds
Haglund
Mouru
Frost
Moore
Ciholas
Everett
Moore Baruch
Ciholas/ Laird
Alibozek
Pratt
Schein
Smith
Levan
Hudson
Blitch
Foster-Miller
Feb 19, 2002
Frost Mangolds
Pratt Mouru
Norman
JPO
iRobot
Mangolds
Mouru
Frost
Pratt
Norman
SPAWAR
Murphy
Casper
Micire
Minten
SPAWAR
USF
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
12
collateral
buildings
Where The Robots Were in the Rubble Pile
9/12/01
9/12/01
FDNY
9/18/01
INTF-1
FDNY
0-live
9/16/01
0-live
1-dead
VATF-1
0-live
4-dead
0-live
9/13/01
FDNY
0-live
1-dead
Not shown:
PATF-1
OHTF-1
NYC
3
2
1
6
4
7
5
New York Times 9/23/01
Feb 19, 2002
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
13
Environment at Ground Zero
Personal Safety:
-Thick dust for days
- asbestos, glass
-Rubble largely stable, but
if fall, could be impaled
- 1 injury: fell out of chair
Weather:
-45-70 deg F
-rain 2 days, making rubble
slick, slippery
Feb 19, 2002
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
14
Contextual View: WTC 2
Feb 19, 2002
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
15
Confined Space using Tethered Bots
• 2 types (chemical & sewer, HVAC
inspection, TRL 9 but not for USAR)
– “polymorphic”: ~60 lbs, 7 hours
battery, fits in 1 backpack, $12K
• height of mouse to height of squirrel
– “fixed geometry”: ~70 lbs., 7 hours
battery, fits in 1 backpack, $10K
• Tethers: 100-300ft
• Terrain & Environment
– voids ~6-12 inch diameter
– dirt, rubble (but not mounds of
paper)
– inclines: depends
• up steep, go with heavier
microTracks
– water resistant
– not intrinsically safe, can melt tracks
Inuktun microVGTV,
Inuktun pipe crawler
video, 2 way audio
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
www.inuktun.com 16
Inuktun microTrac
video, 2 way audio
Feb 19, 2002
Needed Image Processing and Object
Recognition Technologies
WTC 2, 9/18/01
-tracks of previous robot run
-a watch
-3, possibly 4 victims (covered in dust, burned)
- head, arm socket?, nose, perhaps fingers
Feb 19, 2002
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
17
State of Available Information
•
what viewpoints have already been explored
– example: boot?
•
Feb 19, 2002
no one rewound tape far enough back to catch earlier
viewpoint which disambiguated the object!
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
18
Confined Space, Structural Assessment
using Wireless Bots
• 1 type (military hostage rescue, special
order, TRL 9 but not for USAR)
– backpackable by 2 people, 1 for bot, 1
for OCU & batteries, 12 hours-7 days
(standby)
– fast, can right itself with practice
– can add sensors, payloads
– ~$30K
• Wireless
– depends on material
• Terrain & Environment
–
–
–
–
voids: mansize
dirt, rubble,inclines: depends
water resistant
not intrinsically safe
FM Solem
video, striper, audio
www.foster-miller.com
Feb 19, 2002
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
19
Need Sensing of State of the Robot, World
WTC 4, 9/16/01
• Need state of robot
– pose, size relative to the
environment
– health
• some UIs display
• fault detection is hard
• diagnostics and recovery is slow
(replace or 35 minutes)
robot projected a grid to estimate
distance
• State of World
– topological vs. metric
– 3D mapping
– 3D interpretation
• video overlay
• structural reasoning
Feb 19, 2002
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
20
Search and Structural Assessment of
Collaterally Damaged Buildings
• 3 types (military hostage rescue, special
order research, TRL 4-6) ~$30-45K
– backpackable by 2 people, 1 for bot, 1 for
OCU & batteries, 12 hours (UrBot), 20min
(PackBot)
– fast, stairs, grasping
– self-righting or invertible
– can add sensors, payloads, software
• Wireless
• Terrain & Environment
–
–
–
–
buildings, large voids, 3 story drops
dirt, rubble,inclines, stairs
Packbot is water proof
not intrinsically safe
SPAWAR UrBot
video, audio
iRobot PackBot
Feb 19, 2002
FM Talon
video, audio, gripper
www.spawar.navy.mil, www.irobot.com
Robot-Assisted
Search
and Rescue
video, FLIR,
2 way
audioat WTC Disaster (NSF)
21
General Mobility & Hardware Problems
• Lost 1 Solem robot
– lost wireless comms, left in hole, wasn’t there later
• Damage
– detracked once (high heat)
– speared on rubble
– just wear and tear
• Tethers tangle
– only twice not immediately recoverable
• 7.75 “stuck assists” per drop (or once a minute)
– but tether handling is significant
• 9.25 “gravity assists” per drop
– still have tie a rope around the wireless ‘bots
• Wireless and dropouts
– can’t compress and do image processing
– too numerous/duration to count
– make it hard to do image proxy processing
• Fear of flipping and getting stuck
Feb 19, 2002
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
22
HRI Issues Overall
• Warning: camera occlusion 12.3% of a drop
– teleop doesn’t work well in those cases!
• Operator errors (Norman ‘91)
– Mistakes
• 2: wrong robot, had to remove and try another
– microTracks bulldozes in deep dirt, can’t climb
– microVGTV too light to get traction
• 10% of duration of Inuktun runs spent significant time adjusting lights
despite auto gain
– need image enhancement
– Slips
• 0.25 collisions per drop (oversteer)
• 8.9% of duration, robot wheel slip (high centered, wrong configuration)
• Human-Robot Interfaces scared end-users off
Feb 19, 2002
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
23
iRobot: PackBot (experimental)
iRobot PackBot
video, FLIR, 2 way audio
2 people, 35 kg
• game joystick plus laptop with video & audio
• robot state: battery, comms, orientation, camera, encoders
• scared off rescuers: too complicated, too long to train
Feb 19, 2002
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
24
Timeline of Shift w/ VATF-2 (7:00PM 9/18/01 – 7:00AM 9/19/01)
1.
2.
3.
4.
Drive to Ground
65min
Zero from Javits
Center
Parts of 9 member
~355min
group splits off
while the rest waits
2 operators, 2 robots and VATF-2
search void; MicroTrac failure
2 operator, 2 robots and VATF-2
search same void; VGTV failure
5.
2 members retrieve spare robot
from Javits Center
6.
2 members fix 2 robots using
parts from spare robot
7.
9 member group moves to new
location to wait
8.
Group returns to Javits Center
from Ground Zero
12:00AM 2
operators return with
robots after VATF-2
retreats
~6.5min
In route
~6.5min
Waiting
35min
Searching
~5min
~105min
~30min
7:00AM
5:00AM
3:00AM
1:00AM
11:00PM
9:00PM
• 13 minutes for entire shift (green)
• No drop >7 minutes
• <1.5 minute set up time or rescuers walk away
• 2002Found 3+ victims
Feb 19,
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
25
Other problems: Asynergism
• Lack of interoperability / “on the other robot”
– Image processing, intelligent assistance
– Software, sensors couldn’t be migrated between platforms
(“plug and perceive”)
– Only the “dumbest” robots were well-used
• Known capabilities “lost” or “back in the lab”
•
•
•
•
robots as wireless repeaters
self righting
self reacquistion of comms signal
general office navigation, obstacle avoidance
Demo, hardware focus, rather than Systems focus
Feb 19, 2002
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
26
Problems as Point of Departure for Theory
• USAR robots can be generic, adapted in the field for the
particular task, thought of a system
• Sensors and sensing need to be improved, miniaturized, $$
– State of robot: highly sensate robots, health monitoring
– State of world: 3D maps plus understanding structure
– State of available information
• Mobility and hardware need to be improved
– Polymorphic and elephant trunk sensors masts
– Teams of robots
• marsupialism for delivery, relay, proxy processing
• physically coupled: tether managers, in-line collaborative teleoperation
• distributed: hybrid tethers
• Human-robot Interaction needs to be improved
– Perceptual user interfaces not well-defined, mapping & human models
of situational awareness
– Training and access is an issue; over the internet
Feb 19, 2002
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
27
Robot-Assisted USAR as IT
CONNECTIVITY
HOSPITAL
MOBILITY
DECIDABILITY
UNIVERSITY
PERCEPTIBILITY
Forward Station
Section Chief
HOT ZONE
Search Teams
USABILITY
WARM ZONE
COLD ZONE
Base of Operations Command Post
Team Task Leader & Staging Area
personnel prep Incident Commander,
Structural Engineers,
Robot Specialists
CRASAR,
GOVT LABS
NEW-TECH
LABS
ADAPTABILITY
Feb 19, 2002
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
28
Summary of Performance
• Robots were successfully used at the World Trade
Center
– Success: quick response (luck)
– effective performance
• 12/11 equal to manual technical search for 24/10 days
• user acceptance
– Training in Dec. 02 for East Coast firefighters
– Training in Apr. 02 at FDIC in Indianapolis
– ~30 copies of data sets to fire rescue teams to date
• USF as a university provided expertise from field
research, managed data collection (and pushing data
dissemination)
Feb 19, 2002
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
29
However, We Could’ve Done More…
• Had more robots, sensors, and people than were utilized due to
credentialing, organizational issues
– Lost days 3,4
– Never saw a 24/7 deployment cycle
– Only 75% of the available Inuktuns were deployed at any given time,
despite effectiveness
– Buildings could have been surveyed, if necessary, stored on videotape
• Capabilities never exploited
– Robots could place tubing to transport air/water/meds to survivors
• Dr. Eric Rasmussen, 3rd Fleet Surgeon, sent medical equipment
– Added air quality monitors on larger robots after 9/25
– AI Software on larger robots not interoperable with smaller and newer
robots
Feb 19, 2002
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
30
Robots Can’t Replace the Real Heroes
more information and public video & stills: www.crasar.org
murphy@csee.usf.edu
Feb 19, 2002
Robot-Assisted Search and Rescue at WTC Disaster (NSF)
31