Flight Software Workshop
December 16 – 18 2014
Safe Testing of Autonomy in
Complex, Interactive Environments
(TACE)
David Scheidt david.scheidt@jhuapl.edu
Robert Lutz robert.lutz@jhuapl.edu
William D’Amico bill.damico@jhuapl.edu
Subodh Harmalkar subodh.harmalkar@jhuapl.edu
Outline
 Test & Evaluation Need and Challenges
 Background and Definitions
 Current Effort under TRMC’s T&E/S&T program
 Path to a Testing Capability
 Summary
From Tele to Auto to Autonomous Operations
Autonomous Operations
Operating
environmental
decisions
Measure
effect
Measure
effect
 APL has demonstrated the autonomous operation
of multiple and cooperative UXVs where the
operator is a “user-on-the-loop”
 There have been many demonstrations of APL’s
Mission Level Autonomy (MLA) under JHU/APL
internal research and at Camp Roberts (CA)
UXV
Sensor
 Unmanned systems usually require the operator
to be a “user-in-the-loop”
Control
Operational
objectives
 MLA has been shown to be “platform” and
“vendor” agnostic riding above proprietary
control/guidance algorithms
 MLA research led to insights on what would be
needed for rigorous test and evaluation
A measure of effectiveness of an
autonomous system is directly related to
“operational objectives” thus making the test
methodology and test results applicable to
both developmental and operational test
environments – true joint DT/OT is needed
Systems with Autonomous Behaviors Are Here
AACUS:
Autonomous
Aerial
Cargo/Utility
System
ACTUV: ANTI-SUBMARINE
WARFARE CONTINUOUS TRAIL
UNMANNED VESSEL
LDUUV: Large
Diameter UUV
AMAS: Autonomous Mobility Appliqué System
Autonomy Test & Evaluation Challenge
Autonomous systems respond to unpredictable change by devising a course of
action. Before we deploy such systems how can we be sure that autonomous
decisions that will be produced will always: (1) achieve objectives set by human
supervisors and (2) not produce unacceptable unintended consequences. Testing
autonomous systems is particularly challenging since we cannot possibly test all
interactions between the autonomous system and the natural world.
JHU/APL Proprietary
History of Autonomous Unmanned Systems at APL
4 UGVs @ APG
JHU/APL has been developing autonomous
unmanned vehicles since 1962.
OPISR – 4 UAVs, 1 UGV, 2 USV, 1 UUV,
3 UGS
Tactical Sensing 1 UAV
& Multiple UGS
@ Webster
Predator/SSN
Interoperability Demo
2 UAVs & 4 UGVs
1 UAV
1 UGS
@ APG
New Horizons
Unmanned Surface
Vehicles
Air-Exjam
(later EXDRONE and BMQ-147 Dragon)
1965
1970
@ APG
1990
2000
Autonomy S&T
Vision Program
2005
2010
2015
TACE
DADFS
3 UAVs, 2 UGS,
3 Mobile Users
@ Dugway & DOE Range
Agile UxV
Autonomy
1 UUV
TNT Experiments
6 UAVs, 3 UGS
Scheer “Beast”
DARPA UUV Program
Small Oceangoing USVs
NEAR burn failure
Since 2002 JHU/APL has conducted dozens of autonomous vehicle flight
programs and flown/launched hundreds of sorties. Currently JHU/APL has 24
active autonomy programs.
@ Camp Roberts
In Atlantic, Pacific &
Gulf of Mexico
Combined Autonomous Air/Ground Missions
2004 Aberdeen Test Center
OSD NII Swarming Unmanned Vehicle Experiment –
Cooperative Search, Patrol & Track (4 UGVs and 2 UAV)
Combinations of autonomous air and ground vehicles (an AACUS/AMAS
collaboration) are clearly possible and probably required for many military missions
Challenges for Autonomous System Test
• Test & Evaluation (T&E) Need
- Build an infrastructure for SAFE/LIVE testing of autonomous unmanned vehicles
(AUVs), especially for tactical UAVs that have significant airspace and platformrelated range safety issues
- Demonstrate live, virtual, and constructive (LVC) methods for SAFE/LIVE tests
- LVC methods should be Test and Training Network Architecture (TENA) compliant
- To our knowledge there are no AUV established T&E capabilities with these
features
• T&E Challenge
– Provide an infrastructure that supports safe testing when testing autonomous systems
operating over the horizon or in denied environments.
– Provide an infrastructure that supports safe testing when autonomous systems perform
unpredictable actions
– Provide an accurate, real-time, live-virtual-constructive environment that interacts with the
system under test’s autonomy in unpredictable ways.
– Reliable command and control (C2) of the systems under test (SUT) and robust 2-way data
links are needed to stimulate the SUT, to record behaviors, and to maintain SAFE/LIVE test
control
Safe Testing of Autonomy in Complex, Interactive
Environments (TACE) – Key Elements
 Capability

TACE is a 3-phase/36-month program (started in April 2013) where the open/general
architecture will be at TRL6 with initial capabilities for the testing (black and white box) of
autonomous unmanned vehicles (AUVs) of all types. TACE is “portable” but will rely
upon a “thin client” interface to the SUT and the inclusion on the SUT of a minimal “TACE
test applique.” There are 3 levels of control/override within TACE: SUT autonomy, TACE
automatic experiment controls, test range safety officer control
 Relevance to T&E Scenarios

The TACE architecture will use rigorous techniques to prevent unsafe AUV operations or
actions while stimulating, measuring, monitoring, and recording the AUV’s autonomous
response/performance in complex environments. TACE assurance algorithms provide
mathematically rigorous, on-board, real-time, safety guarantees, while TACE dual
simulation system provides a complex live-virtual constructive test environment.
 Cost Benefits

A critical element, as always, is selecting the critical tests. TACE is not a planning tool.
A recent award was made to JHU/APL under the Unmanned and Autonomous System
Test (UAST) program for an autonomy planning tool, Rapid Adversarial Planning Tool
(RAPT). The “cost” of conducting tests where the “Achilles heel” of the embedded
autonomous behavior is unknown would involve very high risk to the loss of the SUT and
the delay of system fielding. The combination of tools such as RAPT and TACE should
prevent unnecessary and expensive testing
TACE System Architecture
TACE Internal
Messages over
ZeroMQ TCP/IP
Socket
TACE Year 1-2
Sensor
Middleware
SUT Comms
TENA over
TCP/IP Socket
SUT
TACE External
Messages over
JUAS?
Stimulator
BIT
Middleware
TACE Year 3
TACE
GNC
Autopilot
Middleware
Non-TACE
Components
Watchdog
Autonomy
Middleware
SUT
Communication
Server
SUT
Ground Stations
SensorModel
Model
Sensor
Sensor
Model
Synthetic Force
Generator
SIMDIS
Test Data DB
Streaming Reliable
Communication
Model
Ground
Communication
Server
Blackbox
Monitor
Whitebox
Monitor
TENA
Datalogger
TACE
TACE TENA
TENA Execution
Execution
TENA to TENA Bridge
R/W
Test Config DB
TestExecutive
Executive
Test
TACEClient
Test
Client
Client
(User
Interface)
(UserInterface)
Interface)
(User
TACE Test
Executive
Watchdog
Manager
Server
Watchdog Manager
Client
(User Interface)
LVC
LVC
LVC
External
External TENA
TENA Execution
Execution
R/W
Test Result DB
R
R/W
Safety Constraint DB
LVC
LVC
LVC
TACE System Under Test (SUT )Payload
TACE’s payload for the Boeing ScanEagle is a repurposing of JHU/APL’s Autonomy
Toolkit (ATK) software, which has been used on more than a dozen types of UAVs
in Boeing, Lockheed, Aerovironment manufactured vehicles, and hardware
previously developed jointly under internal research funds by Boeing and JHU/APL
TACE SUT payload installed in a
Boeing ScanEagle payload bay
ATK payload with Persistent
Systems Wave Relay Wireless
Local Area Network Card
TACE Flight Tests at Aberdeen Test Center
Five Test Events with Multiple Sorties Were Executed during January/February 2014
APL Test Team on the tarmac at
Phillips Army Airfield (PAAF)
Aberdeen Test Center (ATC)
Hand launch of the Procerus
research AUV controlled by
JHU/APL’s Autonomy Tool Kit
(ATK)
Sample Results from TACE Testing at ATC
SUT
Moving NoGo
Virtual Target
Safe Loiter Point
Fixed NoGo Area
Constraint Violation - Range Safety Executive User Interface
Path to a Testing Capability – Phase 2 OV1
Restricted Airspace(s)
ScanEagle as the SUT at Yuma Test Center
Virtual JIMM Entities 1 to N
Watchdog Behaviors
3D Range Boundaries
Platform Safety Limits
Virtual Translated Air Traffic
Live Unicorn UAV
ScanEagle
SUT-System
Under Test
Tactical
EO/IR
Autonomous Behaviors
Search, Track, Avoid
Onboard
Test
Payload
Unicorn
UAV
Wave
Relay
2400MHz
Test Range
Topology
UHF PNT
Downlink
UHF
Safety
Link
Intruder
C2
Intruder
GS
Native
UHF
Link
Test
Director
Land Lines
Test
Ground
Station
Range
Safety
Defense Research & Engineering Network Connection
Live target
track to SUT
Notional Playbox at a Large Test Range
1.0 Mile
Geographic
Constraint
Boundaries
10
Miles
Flight Test
Operations
Base
7.5 Miles
2.5 Miles
1.0
Mile
2.0 Miles
1.0
Mile
Note: paths shown are not actual paths)
Proximity Violation (Virtual Entity)
Violation
Track
Established
Virtual
Aircraft
Live
Ground
Entity
Summary for Autonomy and T&E Perspectives
 TACE Phase 3 will provide some basic operational functions at TRL 6 –
this is an “initial architecture” but not a “range capability”
 To our knowledge, there are no “active” programs to extend beyond
TRL6 – highly understandable since there are no PORs with
autonomous behaviors
 At the recent NDIA Annual T&E conference – SHIFT LEFT is coming

RFPs will include CONOPS and initial TEMPs - will “autonomy” be a “shift left” –

Insights from Dr. Brown, Director TRMC - cited “cyber, big data, hypersonic, and
autonomy” as the T&E challenge areas for the near future
 The NATO Science and Technology Organization (STO) promotes and
... Validation and Verification of Autonomous Systems (SCI-274)
Acknowledgements
• JHU/APL TACE Team: Robert Chalmers, Robert Bamberger, Kristine Ramachandran,
Dean Kleissas, Brendan John, Subodh Harmalkar, William Van Besien, Michael Biggins
• TACE Subcontractors to JHU/APL :
– Trideum: Michael O’Connor
– Boeing Corporation: Gabriel Santander
– InSitu Inc.: Jerry McWithey
• Government Flight Test Support:
– Aberdeen Test Center: John Wiley
– Yuma Test Center: Mary Beth Weaver and Bob Vondell
• Unmanned and Autonomous Systems Test Test Technology Area
– Executive Agent: Vernon Panei
– Deputy Executive Agent: Stephanie Riddle
– Subject Matter Experts: William Hamel, Kirk Bonnevier